I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

It gets you out of hardware. Out of substantial up-front costs, management and repair, depreciation, and end-of-life planning.

It gets you out of data centering. Out of power, cooling, and cabling overhead and management.

It might even get you out of this. With a virtualized infrastructure, you can get access to and administer your servers and network from almost anywhere. From your office, your home, the beach…

What else could you be getting out of with a virtual data center?

If your company still doesn’t have a blog, it can be a great way for your employees to connect and communicate with your market in a regular and timely manner. If you’re already blogging on LiveJournal or some other blog community, you might want to bring it in-house, to take greater control over your blog’s capabilities or user experience, or to consolidate staff blogs in a single location.

Managing a modern, complex web site can be made easier using a CMS, or content management system. A CMS can help you think about your website as separate design, structure, and content layers, and work with these layers separately for best effect. It can also help decentralize maintenance and updating, while enforcing consistency across the site. Some blog software can double as a basic CMS, but there are many systems capable of much heavier duty.

A photo gallery, either public or private, can provide a single, controlled repository of photos taken by your company of products, processes, and publicity events for reference or use in blogs, training, etc.

As your business grows, you may reach a point where email can’t keep up with your customer support needs. A support ticketing system offers a single, shared repository of customer issues and responses which can be used and referenced by everyone on your staff.

Wikis can be a great solution for collaborative knowledge management and documentation. An internal wiki could be used to document and maintain your company’s documentation on everything from products to processes and policies. Pages can be created in the wiki for ad hoc planning and documentation of individual projects.

Have more ideas on what else you could use your VDC for? Post them in the comments!

I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Building a cubicle, no matter what the instructions say, is not exactly trivial. Even after putting the walls together, trying to get them all even and lined up is a trick. Then putting up the desk supports and the desk surfaces themselves, which are usually massive slabs of particle board and plastic veneer. If they’re fastened with screws, it’s always from beneath, out of the light, where you’re certain to get wood dust in your face.

In stark contrast, using vCloud Director to build a virtual machine was easy. Login, select a hardware and OS template, optionally configure, and deploy. It was up and running in a few minutes, maybe a little longer if you have added software or network configuration requirements. It’s all done with a keyboard and mouse, and I’m pretty certain I’ve never gotten wood dust in my face from it.

In both cases, a little planning goes a long way. Making a map of your cubicle layout lets you know whether everything is going to fit, what you’ll need, and whether you have it. Updating your map when you change your mind is also important. Making a map of your systems and network is similarly illuminating. How much CPU, memory, and disk do you need? What systems need outside IP addresses?

Also, paying attention to the details pays off. It’s a lot easier to make the desks level if all the supports are attached at the same height. Double check your IP address assignments and DNS records.

Finally, building a virtual machine doesn’t involve any power tools. Well, let’s hope not.

A Kickstart file basically answers the questions that pop up in the installer as the installer goes removing the need for human interaction. If an question isn’t answered, the installer pops up with the proper dialog, takes user input, and continues. I can pick and choose what information I want to populate automatically and which information dialogs I want the customer to answer. In my auto install ISOs I prompt the customer for a username and password as I want the users to enter that information.

When I was tasked with making an auto installing ISO for our customers I was able to create one quickly by using a kickstart file.

The process of making a CD is a bit verbose, and better handled by some of the how-tos out there.

But I’ll take your through my Kickstart file.

First are some of basic information about the system. These are fairly self-explanatory.

platform=AMD64
#System language
lang en_US
#Language modules to install
langsupport en_US
#System keyboard
keyboard us
#System mouse
mouse none
#System timezone
timezone America/Chicago

I disable root, to reflect the Ubuntu default. You can enable it by removing the next line, and setting it with the second.

rootpw --disabled
#rootpw jpDhuZtql4of4rfq

I do not automatically add a user, but you can with the next line.

#user johndoe --fullname "John Doe" --password changeme

I don’t think this does much in an Ubuntu Server install but I put it in anyways.

#Use text mode install
text

We’re installing not upgrading.

#Install OS instead of upgrade
install

Use the CD-ROM.

#Use CDROM installation media
cdrom

Where are we going to put the bootloader?

#System bootloader configuration
bootloader --location=mbr

Get rid of any existing partitions.

#Partition clearing information
clearpart --all --initlabel

Partition the disks using Ubuntu defaults (512MB swap, etc) This allows the ISO to work on whatever size disk you want. Linux isn’t great about using swap anyways, so 512 is plenty.

#Disk partitioning information
part /boot --fstype ext3 --size=200 --ondisk=hda
part swap --recommended
part / --fstype ext4 --size 1 --grow

Passwd information. I know… MD5… You can use something more secure if you wish.

#System authorization infomation
auth  --useshadow  --enablemd5

We need DHCP for some of the following steps, as I have no idea what type of network this will be run on. You can specify other info here if you want.

#Network information
network --bootproto=dhcp --device=eth0

My customers hate having UFW on. I don’t think this actually works yet in Ubuntu, so I also do it in a later script.

#Firewall configuration
firewall --disabled

X-Windows on a Server? No thanks.

#Do not configure the X Window System
skipx

And finally, we want to reboot after installing. This isn’t actually done, as we’re going to run a post-install script.

#Reboot after installation
reboot

Add additional packages to install. I install the fewest here, as I update in a later script, so why install a bunch of stuff only to update it later?

%packages
@dns-server
@openssh-server
gcc
build-essential

Here comes a a post install script.

%post

Mount the CD again, as there’s data we want off of the CD.

echo Making CD Mountpoint
mkdir -p /mnt/cdrom
echo Mounting CD
mount -t iso9660 /dev/sr0 /mnt/cdrom

Copy over a script that I’ve written that does updates and additional installs when the virtual machine is first booted.

echo Copying Firstboot Script
cp /mnt/cdrom/firstboot /etc/init.d/
chmod +x /etc/init.d/firstboot

Updated the init structure to run the firstboot script on boot.

update-rc.d firstboot defaults
echo Adding new Crontab

Add a custom crontab with some randomized sleep values.

cp /mnt/cdrom/crontab-template /etc/crontab

A script that I wrote that edits resolv.conf to point to the local bind server

echo Copying resolvfix init script
cp /mnt/cdrom/resolvfix /etc/init.d/
chmod +x /etc/init.d/resolvfix
update-rc.d resolvfix start 99 2 3 4 5 .

An updated sources.list with a closer mirror.

echo Copying Apt Sources
cp /mnt/cdrom/geeks-org-sources.list /etc/apt/sources.list

A new dhclient with the local bind server seeded.

echo Copying dhclient.conf
cp /mnt/cdrom/dhclient.conf /etc/dhcp3/

A new named.conf.options with some useful defaults.

echo Copying named.conf.options
cp /mnt/cdrom/named.conf.options /etc/bind/

Moving over vmware-tools for installation upon first boot.

mkdir /vmware
cd /vmware
echo Extracting Tools
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz

Ejecting the CD.

echo Unmounting CD
umount /mnt/cdrom

Update the system.

echo Updating
apt-get update
apt-get -y dist-upgrade

And finally, reboot the system (sync for good luck ;) ).

echo Rebooting
sync
reboot

Now, as I mentioned before, there’s a firstboot script that I run that does quite a bit of work before the machine is finished. It does things like wipe out the SSH keys, install VMware Tools, remove and purge old kernels and install applications like MySQL, Apache, as required.

Well, that’s one of the tricks I have tucked up my sleeve, I hope it helps!

Then I stumbled on this link (conshell.net) which explains how to use dd and netcat to copy a disk over a network.

It was so simple, it verged on genius! But did it work?

The steps are easy:

1) Create a virtual machine with  a disk about the same size or larger than your source (not smaller)

Pick an arbitrary port, (9001 in this example) and set up your firewall or VSE to allow that port to the target machine.

2) Boot that new VM into a rescue environment or use a live cd.

3) Use the following commands:

On the VM: nc -l -p 9001 | dd of=/dev/sda

On your source machine: dd if=/dev/sda | nc 9001

4) Wait a long time… I averaged around 15Mbps from my test machine to my new VM, it ranged from 30Mbps down to 7Mbps. I’m sure that had more to do with my network than anything. Still, this can take a while.

5) Once the dd has completed (dd will dump summary information) reboot the machine back into the live-cd environment, check the partitions with e2fsck the partitions and re-size them. (I cheated and used gparted)

6) At this point you can either mount the filesystem and remove the udev rules (in /etc/udev/rules.d/) or boot into your VM and remove them via the console. Either way, you have to reboot after the udev rule are removed.

7) Reboot, and voilà!

The live cd I used was CDLinux. It’s a small Linux distribution that runs XFCE, and fits in an 80MB ISO. It also includes an SSH server, so you can set up an ssh tunnel, and use netcat against that rather than use an arbitrary port. It also has the VMware paravirtual scsi drivers.

Anyways, this worked. Wow did it work. I didn’t bother to zero out the remaining space on the disk, it took me about 2.5 hours to move 8GB worth of data but I was greeted with a familiar prompt in a new place as soon as I booted it up.

Now a couple of caveats. I did this on a running system, with no prep work. I would recommend trimming unnecessary data and shutting down as many services as you can. It’s best to do this when the machine is “down,” not doing anything beyond facilitating the copy. However, it does work on a live system. Still, if I were moving a production system, I would follow the advice in the linked article above.

But, it was my system, in a test environment, so I didn’t really care.

Still, isn’t it amazing what a couple of UNIX pipes can do?

Reboot your server from the VMware Converter CD. Once VMware Converter is running, click on “Import Machine”. Click through the first couple of panels. On the “Source Data” panel, select “Import all disks and maintain size”. On the “Destination Type” panel, set the destination type to “Other VMware Virtual Machine”. On the next panel, set the VM Name and the Location where you’re storing your disk image. Since Converter doesn’t support direct OVF export here, you’ll have to set the type of virtual machine to Workstation 6.x and change it later. Click through the rest of the panels, and Finish. VMware Converter will now export your server to a .vmx description file and .vmdk disk image(s). This may take a while.

After VMware Converter is done, you’ll need to use VMware’s OVF Tool to convert the .vmx file to a .ovf file.

ovftool.exe name.vmx name

Next, the tricky bit. You’ll need to edit the .ovf file, which is in XML format, and find and update the VirtualSystemType field to vmx-07, a reasonably current version. Once you’ve edited the .ovf file, you’ll also need to generate its SHA1 hash and update the .mf manifest file. Now, you should have something which can be imported into your vmForge VDC.

Login to your vmForge VDC (or any VMware vCloud Director system), and create a new catalog if you don’t already have one. In your catalog, click on the Upload button. Select your .ovf file, give it a name and description, and click on the Upload button. Again, this may take a while. Once its done, you’ll be able to use the template from your catalog to create a new virtual machine, a clone of your old physical server. The last step will be to power on the virtual server, login via the virtual console, and reconfigure networking.

my first test is 10,000 requests for a static HTML page (2866 bytes). this test was run against a single apache server in the pool and against each of software load balancers with two back-end servers, from one source and from three simultaneously.

the single apache server actually performed the best, easily beating any of the software load balancers in raw throughput. the test from three sources is effectively a mild denial-of-service attack, and none of the software load balancers could handle it. each of them failed and stopped accepting connections well before 10,000 requests were completed.

the second test is 20 requests for a PHP script which performs exactly 1 second of mathematics then returns results. in this case, almost all the load is on the back-end servers, and there was negligible difference in results between the four front-ends.

the third test is 400 requests for the PHP script, but issuing 20 concurrent requests from each source at a time. this generates significant load on the back-end servers, but is the first test where having multiple back-ends shows any improvement.

the fourth test is a monster, 400 requests for the PHP script, 100 concurrent requests from each source at a time.

these results suggest that a software load balancer might be an option for putting more capacity and resilience into a script-heavy website, but clearly shouldn’t be chosen for performance. the single apache server performed better than my small cluster in nearly every test, and much better in a few cases. the total meltdown of the software load balancers in difficult situations is of particular concern. the results for the cluster might improve with more back-end servers, but the software load balancer itself seems to be the bottleneck.

of the three software load balancers, the simpler Balance and Pen outperformed Crossroads in general. interestingly, Balance fared spectacularly poorly against heavy traffic in the first test, but very well against a different sort of heavy traffic in the fourth.