The Problem
When you create an Amazon EC2 instance, you are allowed to throw a small amount of data (64k ?) into instance metadata. This metadata can then be read back by the instance by querying a special URL. This feature has been used for server-classification and configuration of systems.

The problem is that everyone was using this metadata space in a different way. Some people were just throwing a single trigger word like “database” into this metadata space, and some people were storing large procedural configuration scripts that would be downloaded and executed by the instance. There has been much fragmentation onto how to use this metadata space for server configuration.

Cloud-init as a solution
Let’s say that you want an Ubuntu machine to boot up, upgrade all available packages, install Puppet, then check-in to your PuppetMaster:

#cloud-config
apt_upgrade: true
packages:
- puppet
puppet:
conf:
agent:
server: "puppetmaster.example.org"


I look forward to software like cloud-init becoming common and universally supported by more cloud providers. The concept is simple, easy to work with, and powerful.

Read more about cloud-init at the Ubuntu Docs Page for cloudinit

The Evil Template-Based Workflow Example

1. Configure your base system, install some good software
2. Take a Snapshot (somehow… re-bundle an AMI? Hit the snapshot button in VMWare?)
3. Shutdown your base system
4. Spin up instances based on this snapshot
5. Attempt to manage the chaos as configurations drift
6. Spin up your base system, apply various updates, and take a new snapshot
7. Spin up new instances based on new snapshot
8. Chaos!!

You can see how without a dynamic configuration management system, the template-based workflow is impossible to manage. Outdated templates and manual system configuration result in complete infrastructure chaos.

Configuration Management Systems (use them!)

Carl realized that the smarter alternative to the template-based workflow is to use a configuration management system, but there’s so many options! Carl wasn’t sure which one to use, how to begin using any of them, or how all of the pieces fit together. Carl told me that one of the most confusing things in his research was to understand where the responsibilities of the various management system begin and end.

A Nebulous Onion

I drew the following diagram for Carl to help him understand how all of the current tools available fit together into a bigger picture of Cloud Management. This workflow example is bias to Cloud Providers with advanced APIs, and Linux/UNIX operating systems- but why would you ever use anything else?

1. Creation of Servers: Some event occurs, and servers are created. Maybe this is an auto-scaling request from your load balancer, or maybe John in Marketing says that your website is going to be mentioned on Oprah later this afternoon. (woo!)
2. Servers are created programmatically: A bit of configuration metadata is stored somewhere in an API that will be referenced in step 3. e.g., these 10 new LARGE servers should be WEB servers in the PRODUCTION cluster.
3. The OS boots: The instances boot, and cloud_init (or some custom script you spent all afternoon writing) reads the metadata that you stored back in step 2, and basic configuration begins. Puppet is installed, and your new instances prepare for their first connection to your Puppet master for configuration.
4. Your Systems are 100% Configured: A good configuration management system will leave your systems in a fully-configured state. No manual intervention should be required.
5. Connection to External Data: This could be the topic of a larger blog post, but the final step in configuration is to connect your instances to external data sources. This could mean mounting an NFS share, connecting an EBS volume, or joining a Gluster or Casandra ring. This is really upto your imagination, and your application’s needs.

Summary:

There are many tools for Wrangling the Cloud, and I’ve only described one example workflow from a very abstract level. There is no one-size-fits-all solution, however that doesn’t mean that you should reinvent the wheel. We’ve built a lot of systems like this for our clients, contact us for help with yours.

Luckily, Debian has packaged a new version of Postgres that will use the libreadline6 library if it has been installed instead of the inferior (but more open-licensed) libedit. Unfortunately, the new Postgres package is only currently available in Debian Backports so you’ll have to enable the backports repo first.

Once you’ve installed Postgres8.4 > r114 from backports, all you have to do is install readline:
sudo apt-get update && sudo apt-get install libreadline6

In order to move the swap file from the OS drive to the data drive:

zfs create -V 4G tank/swap
zfs set dedup=off tank/swap
swap -a /dev/zvol/dsk/tank/swap
swap -d /dev/zvol/dsk/rpool/swap


And then to make this change persistent across reboots, also edit the swap path in your /etc/vfstab.

/dev/zvol/dsk/tank/swap - - swap - no -


Using a shared swap device is fine for development and small office servers, but you should consider using dedicated and mirrored swap if this server will be under any serious load.

First find two physical disks for your new mirrored swap

root@gyro:~# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c7t0d0
/pci@0,0/pci8086,3472@1f,2/disk@0,0
1. c7t1d0
/pci@0,0/pci8086,3472@1f,2/disk@1,0
2. c7t2d0
/pci@0,0/pci8086,3472@1f,2/disk@2,0
3. c7t3d0
/pci@0,0/pci8086,3472@1f,2/disk@3,0
4. c9t0d0
/pci@0,0/pci8086,25f7@2/pci8086,350c@0,3/pci11ab,11ab@2/disk@0,0
5. c9t1d0
/pci@0,0/pci8086,25f7@2/pci8086,350c@0,3/pci11ab,11ab@2/disk@1,0
6. c9t2d0
/pci@0,0/pci8086,25f7@2/pci8086,350c@0,3/pci11ab,11ab@2/disk@2,0
7. c9t3d0
/pci@0,0/pci8086,25f7@2/pci8086,350c@0,3/pci11ab,11ab@2/disk@3,0


Let’s pick the last two drives for our example

zpool create SWAP_ZPOOL mirror c9t2d0 c9t3d0
zfs set mountpoint=legacy SWAP_ZPOOL

Setting mountpoint=legacy will keep Solaris from mounting this drive automatically upon import. From here, follow the steps above to add this device to configure this device as swap and add to vfstab.

So how can Puppet help us with this task?

#!/usr/bin/puppet

$package_list = ['apache2','php5','screen']

package{$package_list:
ensure => 'latest',
provider => 'apt',
}

Then make sure to chown this file:

$ chmod +x packages.pp
$ ./packages.pp

Now, when you run this file, Puppet will use the built in ‘apt’ package provider to install the list of packages defined at the top of the file.

It’s actually much easier to write a complex script using a high-level language such as Puppet, rather than Bash, despite this example that requires more code to accomplish the same task using Puppet than Bash.

Using Puppet instead of Bash for basic system scripting gives you many advantages such as better debugging, better error handling, standard formatting, better readability, and is generally much easier to perform complicated tasks than Bash or perl.

It also is a great introduction if you’re just getting your feet wet in the Puppet world. Good luck, and check back soon for more Puppet tutorials.

Problem symptoms included Tomcat freezing while initializing, then after 10mns timing out with error messages saying that it was unable to contact the WSDL URL. Before we could connect to that URL, we had to tell Java’s SSL Libraries to allow unsafe renegotiation.

How to fix this:

Short Answer:
In /etc/default/tomcat6 (Debian/Ubuntu), add a new line to your ‘CATALINA_OPTS=’ section.

-Dsun.security.ssl.allowUnsafeRenegotiation=true


noatime
A very typical UNIX/Linux tweak is to turn off ‘atime’ on your system drive. As part of the POSIX, your OS must write ‘last accessed’ time to each time upon read. On SSD disks, the default ‘atime’ feature can cause pauses and high latency due to excessive small writes. Turning off ‘atime’ may affect HD clean-up tools that check for infrequently accessed files, and may also affect a few UNIX applications, such as older versions of Mutt.

In UNIX and Linux, the noatime setting is easily configured via /etc/fstab, but in OS X, a launchd file must be configured. The idea and implementation for this is from: http://blogs.nullvision.com/?p=275

Open your terminal (/Applications/Utilities/Terminal) and type:

sudo nano /Library/LaunchDaemons/com.nullvision.noatime.plist

And inside that new file, paste the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
        "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.nullvision.noatime</string>
        <key>ProgramArguments</key>
        <array>
            <string>mount</string>
            <string>-vuwo</string>
            <string>noatime</string>
            <string>/</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
    </dict>
</plist>

Hold CTRL and hit X to exit, make sure to save when asked. Then reboot your computer, and open the terminal again. We will now confirm that your system has ‘noatime’ enabled.

mount | grep ” / “

You should see something simular to:

/dev/disk0s2 on / (hfs, local, journaled, noatime)

Success! your operating system drive has been mounted with ‘noatime’.

Move your Swap
If you have a system with both an SSD as well as a standard magnetic HD, you may consider moving your swap file off the SSD and onto the HD. There are various arguments for making this change- by moving your swap to the slower/larger storage device, paging operations to disk will obviously be much slower, however if your system has ample RAM, the swap file should be mostly untouched anyhow. Your upgrade priorities should be reevaluated if you have an SSD in a system that has less than 4GB of RAM.

To move your swap to a new location, we will modify a built-in system launchd file, and create a wrapper script. Editing base files installed by the OS is scary, and dangerous, so make sure you have a backup before attempting this step. This step also requires that the Xcode developer tools have been installed, since the plist editor is not included in the base OS X install. (implementation modified from http://hintsforums.macworld.com/showthread.php?p=452409 )

First, we’re going to create the wrapper script:

mkdir -p /opt/local/bin/
sudo nano /opt/local/bin/dynamic_pager_init.sh

And now inside that file:

#!/bin/bash
#launch Apple’s dynamic_pager only when the swap volume is mounted

MOUNTPOINT=$1

#check to see if swap storage has been mounted yet
/sbin/mount | /usr/bin/grep -q $MOUNTPOINT

#if $MOUNTPOINT is mounted, then create swap file, otherwise exit1
if [ $? -ne 0 ];then
echo “Swap storage not mounted yet, waiting”;
exit 1
else
echo “Launching dynamic pager on volume Swap”;
/bin/mkdir -p $MOUNTPOINT/.vm
/sbin/dynamic_pager -F $MOUNTPOINT/.vm/swapfile;
fi

Hit CTRL-X and save the file, and then set execute permissions on the file:

sudo chmod +x /opt/local/bin/dynamic_pager_init.sh

Now that the wrapper script is in place, we’re going to tell the launchd config to use this script to create the swap file at boot. Launchd will continually rerun the script until your selected swap volume is mounted, and dynamic_pager runs successfully.

Open your terminal, (/Applications/Utilities/Terminal) and type in:

sudo open /Developer/Applications/Utilities/Property\ List\ Editor.app /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist

You should see the following plist settings:

In order to use the wrapper script instead of calling the dynamic_pager_init binary directly, make your plist look like this:

Make sure to change the value at “item 1″ to the name of your mounted HD. The HD on my system is mounted at “/Volumes/Data”. In order to find out the name of yours, check in the OS X “Disk Utility” or from the Terminal, type in “mount” and find the name of your HD in the list.

I then had to save this file to my desktop, and then copy the file in place via:

sudo cp ~/Desktop/com.apple.dynamic_pager.plist /System/Library/LaunchDaemons/

Now reboot, and test to make sure your swap has been successfully relocated by typing:

• Boot into Windows Safe Mode
• Navigate to c:\windows\system32\drivers\
• Rename applehfs.sys to applehfs.sys.BACKUP
• Rename applemnt.sys to applemnt.sys.BACKUP
• Reboot into normal Windows mode

Before you can rename these files, make sure that Windows is showing file extensions. You may notice after rebooting that Windows will no longer mount your HFS OS X volumes. This is an unfortunate side-effect to this fix, however MacDrive does a much better job than Apple’s driver at mounting HFS volumes.

UPDATE:

It turns out that the latest version of MacDrive now requires that the Apple HFS drivers are loaded. So, ether use an older version of MacDrive, or start holding your breath and wait for Apple to update their official (broken) HFS drivers.