Changing your disk scheduler on a Linux virtual machine to increase performance.

First some background of what we do with storage at ipHouse in our VMware environments.

We really like NFS. Architecturally it’s simpler than block based storage; you just need a good local area network and a storage system that can export a file based protocol. There’s no need for specialized hardware or intelligent host bus adapters, just let the storage array handle the storage. Virtualization lends itself to file based storage quite well. VMDKs are just files after all. I kind of snickered when VMware first came out with their VAAI storage extensions. It seemed, to me, like they were enhancing block-level storage devices to do a lot of what NAS based storage already does.

While I was taking my VCP4 class my colleges, most of whom were from big companies, snickered when I mentioned that our storage was on a NAS. A “filer” for them was a place for document sharing and storage. There was “no way” it would ever be fast enough, or good enough to backend their virtualized infrastructure. I’ve seen that notion fade more and more as ZFS has opened the doors for storage startups; and the big players are fighting back with their own specialized NAS devices. There are some really cool ideas floating around: NAS devices that are scale-out, that are optimized for virtualization, and that can do in-line deduplication of data.

That being said…

I have learned that there are some OS level tweaks that can enhance performance on virtual machines. Most x86 operating systems seem to be optimized for single disks, or internal RAID setups. Understandable as that has traditionally been the bulk of their install base. This means that the OS can manage disk queuing better that the dumb RAID card, or the dumber hard drive. CFQ, the default disk scheduler as of kernel 2.6.18 does this. As I understand it CFQ breaks synchronous read/write requests into queues, and assigns timeslices to each queue, weighted by IO priority. The effect is that higher priority processes get longer queues which keeps IO requests from the same process close together. Great idea when the OS has direct access and is managing the storage. Not so great when the storage is handled remotely; the array on the other side is doing the scheduling. All of that optimization is ostensibly ignored. So for a virtual machine it’s better to switch to a simpler algorithm and let the storage array handle the write queuing.

From my reading (and testing) It’s better to switch to the noop scheduler. Noop simply shoves all requests into a first-in-first-out (FIFO) queue and can merge requests. It is simple, fast, and is great for flash storage (no mechanical latency) or for situations where optimization is handled by another device. Like a NAS! Perfect for virtualization!

I discovered this after getting a snippet of a shell script to try from Mike (who got it from a potential vendor that is a big storage geek). This wasn’t new information as Mike had mentioned this almost 18 months ago in passing but neither he nor myself ever tested it. After giving me the info, again, he suggested that I “test this out, and let me know if it works.”.

I’m still testing it, so caveat emptor, but I thought I’d share it with you.

***WARNING DO NOT DO THIS ON A VM WITH SNAPSHOTS***

#!/bin/sh

grep '' /sys/block/sd*/queue/scheduler
for d in /sys/block/sd*; do
echo noop > $d/queue/scheduler
done
grep '' /sys/block/sd*/queue/scheduler

This switches the scheduler from cfq to noop on all “SCSI” disks in the virtual machine.

He also added the following tweak to increase the read-ahead from 256 sectors to 1000 sectors, which caches more disk data for faster read times, after printing what the OS has mounted.

#!/bin/sh

mount
blockdev --getra /dev/sd?
blockdev --setra 10000 /dev/sd?
blockdev --getra /dev/sd?

Again, I’m still testing this on my personal stuff, but, qualitatively, things feel a lot faster. If anything, I haven’t crashed my Linux systems.

Anyways, I hope that helps!

I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

It gets you out of hardware. Out of substantial up-front costs, management and repair, depreciation, and end-of-life planning.

It gets you out of data centering. Out of power, cooling, and cabling overhead and management.

It might even get you out of this. With a virtualized infrastructure, you can get access to and administer your servers and network from almost anywhere. From your office, your home, the beach…

What else could you be getting out of with a virtual data center?

ZFS Version 28 adds some of the more important features of ZFS: Deduplication, triple parity RAIDZ3, and RAIDZ. This means that I can have full featured storage devices, running ZFS natively, via FreeBSD.

As a bonus I don’t have to learn Solaris.

You can run ZFS in Linux but you would either have to run via FUSE which is file system emulation in user-space, not in the kernel. Or download it and build it yourself. In my opinion, both of those options are idiotic. I’m not willing to jump through those kind of hoops just to run a filesystem in Linux. I’d rather have native, in kernel support for it. Until now, your choice was either run Solaris (or a fork of Solaris) or run an outdated version of ZFS via FreeBSD.

One project that should directly benefit of this: FreeNAS. FreeNAS is a customized installation of FreeBSD designed to operate as a NAS and iSCSI SAN. It has a pretty slick ajax/web interface as of version 8 but so far had missed out on key ZFS features.

One reason I want run up FreeBSD 9 and ZFS is to better learn ZFS troubleshooting and administration. FreeNAS aside, there are a lot of vendor supported storage devices that are coming into the market based on ZFS. I want to troubleshoot those devices on a lower level. Before this, I would have to install Solaris. This means that I would actually have to navigate to Oracle’s Website. No thank you.

In-line deduplication is on of my favorite impractical features of all time. It unfortunately, required gobs of memory (8 GB RAM for every 1 TB of storage, if memory serves) Hopefully, someone smart will figure out how to do it on flash, in a practical way, as rebuilding those tables after a power failure would suck. (see Mike’s post about Tegile – a company actually doing such in production today)

Obviously I don’t know a lot about ZFS yet which is why I’m glad I get to learn via FreeBSD.

If only I could convince ipHouse to give me a little more storage space on my personal VDC…hint hint!

If your company still doesn’t have a blog, it can be a great way for your employees to connect and communicate with your market in a regular and timely manner. If you’re already blogging on LiveJournal or some other blog community, you might want to bring it in-house, to take greater control over your blog’s capabilities or user experience, or to consolidate staff blogs in a single location.

Managing a modern, complex web site can be made easier using a CMS, or content management system. A CMS can help you think about your website as separate design, structure, and content layers, and work with these layers separately for best effect. It can also help decentralize maintenance and updating, while enforcing consistency across the site. Some blog software can double as a basic CMS, but there are many systems capable of much heavier duty.

A photo gallery, either public or private, can provide a single, controlled repository of photos taken by your company of products, processes, and publicity events for reference or use in blogs, training, etc.

As your business grows, you may reach a point where email can’t keep up with your customer support needs. A support ticketing system offers a single, shared repository of customer issues and responses which can be used and referenced by everyone on your staff.

Wikis can be a great solution for collaborative knowledge management and documentation. An internal wiki could be used to document and maintain your company’s documentation on everything from products to processes and policies. Pages can be created in the wiki for ad hoc planning and documentation of individual projects.

Have more ideas on what else you could use your VDC for? Post them in the comments!

I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Building a cubicle, no matter what the instructions say, is not exactly trivial. Even after putting the walls together, trying to get them all even and lined up is a trick. Then putting up the desk supports and the desk surfaces themselves, which are usually massive slabs of particle board and plastic veneer. If they’re fastened with screws, it’s always from beneath, out of the light, where you’re certain to get wood dust in your face.

In stark contrast, using vCloud Director to build a virtual machine was easy. Login, select a hardware and OS template, optionally configure, and deploy. It was up and running in a few minutes, maybe a little longer if you have added software or network configuration requirements. It’s all done with a keyboard and mouse, and I’m pretty certain I’ve never gotten wood dust in my face from it.

In both cases, a little planning goes a long way. Making a map of your cubicle layout lets you know whether everything is going to fit, what you’ll need, and whether you have it. Updating your map when you change your mind is also important. Making a map of your systems and network is similarly illuminating. How much CPU, memory, and disk do you need? What systems need outside IP addresses?

Also, paying attention to the details pays off. It’s a lot easier to make the desks level if all the supports are attached at the same height. Double check your IP address assignments and DNS records.

Finally, building a virtual machine doesn’t involve any power tools. Well, let’s hope not.

A Kickstart file basically answers the questions that pop up in the installer as the installer goes removing the need for human interaction. If an question isn’t answered, the installer pops up with the proper dialog, takes user input, and continues. I can pick and choose what information I want to populate automatically and which information dialogs I want the customer to answer. In my auto install ISOs I prompt the customer for a username and password as I want the users to enter that information.

When I was tasked with making an auto installing ISO for our customers I was able to create one quickly by using a kickstart file.

The process of making a CD is a bit verbose, and better handled by some of the how-tos out there.

But I’ll take your through my Kickstart file.

First are some of basic information about the system. These are fairly self-explanatory.

platform=AMD64
#System language
lang en_US
#Language modules to install
langsupport en_US
#System keyboard
keyboard us
#System mouse
mouse none
#System timezone
timezone America/Chicago

I disable root, to reflect the Ubuntu default. You can enable it by removing the next line, and setting it with the second.

rootpw --disabled
#rootpw jpDhuZtql4of4rfq

I do not automatically add a user, but you can with the next line.

#user johndoe --fullname "John Doe" --password changeme

I don’t think this does much in an Ubuntu Server install but I put it in anyways.

#Use text mode install
text

We’re installing not upgrading.

#Install OS instead of upgrade
install

Use the CD-ROM.

#Use CDROM installation media
cdrom

Where are we going to put the bootloader?

#System bootloader configuration
bootloader --location=mbr

Get rid of any existing partitions.

#Partition clearing information
clearpart --all --initlabel

Partition the disks using Ubuntu defaults (512MB swap, etc) This allows the ISO to work on whatever size disk you want. Linux isn’t great about using swap anyways, so 512 is plenty.

#Disk partitioning information
part /boot --fstype ext3 --size=200 --ondisk=hda
part swap --recommended
part / --fstype ext4 --size 1 --grow

Passwd information. I know… MD5… You can use something more secure if you wish.

#System authorization infomation
auth  --useshadow  --enablemd5

We need DHCP for some of the following steps, as I have no idea what type of network this will be run on. You can specify other info here if you want.

#Network information
network --bootproto=dhcp --device=eth0

My customers hate having UFW on. I don’t think this actually works yet in Ubuntu, so I also do it in a later script.

#Firewall configuration
firewall --disabled

X-Windows on a Server? No thanks.

#Do not configure the X Window System
skipx

And finally, we want to reboot after installing. This isn’t actually done, as we’re going to run a post-install script.

#Reboot after installation
reboot

Add additional packages to install. I install the fewest here, as I update in a later script, so why install a bunch of stuff only to update it later?

%packages
@dns-server
@openssh-server
gcc
build-essential

Here comes a a post install script.

%post

Mount the CD again, as there’s data we want off of the CD.

echo Making CD Mountpoint
mkdir -p /mnt/cdrom
echo Mounting CD
mount -t iso9660 /dev/sr0 /mnt/cdrom

Copy over a script that I’ve written that does updates and additional installs when the virtual machine is first booted.

echo Copying Firstboot Script
cp /mnt/cdrom/firstboot /etc/init.d/
chmod +x /etc/init.d/firstboot

Updated the init structure to run the firstboot script on boot.

update-rc.d firstboot defaults
echo Adding new Crontab

Add a custom crontab with some randomized sleep values.

cp /mnt/cdrom/crontab-template /etc/crontab

A script that I wrote that edits resolv.conf to point to the local bind server

echo Copying resolvfix init script
cp /mnt/cdrom/resolvfix /etc/init.d/
chmod +x /etc/init.d/resolvfix
update-rc.d resolvfix start 99 2 3 4 5 .

An updated sources.list with a closer mirror.

echo Copying Apt Sources
cp /mnt/cdrom/geeks-org-sources.list /etc/apt/sources.list

A new dhclient with the local bind server seeded.

echo Copying dhclient.conf
cp /mnt/cdrom/dhclient.conf /etc/dhcp3/

A new named.conf.options with some useful defaults.

echo Copying named.conf.options
cp /mnt/cdrom/named.conf.options /etc/bind/

Moving over vmware-tools for installation upon first boot.

mkdir /vmware
cd /vmware
echo Extracting Tools
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz

Ejecting the CD.

echo Unmounting CD
umount /mnt/cdrom

Update the system.

echo Updating
apt-get update
apt-get -y dist-upgrade

And finally, reboot the system (sync for good luck ;) ).

echo Rebooting
sync
reboot

Now, as I mentioned before, there’s a firstboot script that I run that does quite a bit of work before the machine is finished. It does things like wipe out the SSH keys, install VMware Tools, remove and purge old kernels and install applications like MySQL, Apache, as required.

Well, that’s one of the tricks I have tucked up my sleeve, I hope it helps!

I don’t think anyone would disagree that the documentation for CloudStack still needs work. But all the documentation in the world can’t help if you decide to skip important sections.

Some caveats notes while I was working with 2.2.12 and 2.2.13:

• vSphere 4.1 only, vSphere 5 is not compatible
• Regardless of vSphere licensing used, vSphere vDS is not supported
• vSphere DRS must be turned off so that CloudStack can do its own load balancing (it seems angry if things move)
• No mention of vSphere HA but I enabled HA because the point of using vSphere as the hypervisor was for automated high-availability
• Discussion of vSphere SDRS going on in the forums but so far this also seems like a toggle that needs to be disabled for now
• Secondary storage needs to be accessible by the Secondary Storage VM, vSphere hypervisors, and the CloudStack management system

At first I didn’t read the documentation all the way through – just the networking bits as that was the largest area of the document and initially my largest area of concern. The lack of vDS is a bummer but I’m going to guess that they’ll add this functionality in a later release. I believe vDS would make a few things easier on the users of CloudStack long-term and look forward to this addition.

Since I didn’t read the documentation completely I failed to notice that it states support for vSphere 4.1 and mentions nothing about vSphere 5. Even after I started digging further into the document I assumed that vSphere 5 would be supported. That’s a bad assumption and after a reinstall of the physical servers (and vCenter) I was able to get the initial configuration to pass. Strike one for me – I should know better than to skip portions of an install document.

Next was getting all the networking together with the trunking for VMs, storage network, interconnection between CloudStack and the physical servers. This wasn’t difficult but did need me to actually write everything down as there was a lot going on concurrently. See my last note above for the hint to my biggest hurdle I needed to overcome.

I have some worries about what happens when a physical server fails (and everything fails at some point). I don’t know if CloudStack will initially notice the failure before vSphere ends up bringing the VM back online on a different physical box. And once it is up I don’t know how CloudStack will respond or alert for such items. I bring this up because CloudStack has its own HA system in place but the configuration in the general settings area shows 1800 seconds to trigger (if I am even reading that right). vSphere HA is faster than that. I guess we’ll see what happens if/when it does occur.

And the biggest hurdle for me was the requirement that the secondary storage VM, CloudStack management system, and the hypervisors need to be on the same network for the secondary storage to work. This storage is for holding templates and ISO images for later deployment into a customers’ cloud environment. Once this items a pointed out to me then everything started to go together. This was an area of frustration as it wasn’t clear to me in the documentation that this is the case. The examples flow around an idea of one network for management, VM traffic, and storage. That is not how things are designed in my world as I separate out each of those types of traffic.

I found CloudStack simple to install and once I knew the different restrictions/notes above the process only took 10 minutes to complete.

If you want to read documentation in a browser, I found the cloud.mindtouch.us site to be very helpful. Of course CloudStack has PDF based documentation as well though multi-file searching is either great or abysmal depending on your workstation OS of choice so I opted for the web version to help narrow things down before going back to the PDF editions.

Our customer is still working out a few kinks and should be fully operational soon.