Changing your disk scheduler on a Linux virtual machine to increase performance.

First some background of what we do with storage at ipHouse in our VMware environments.

We really like NFS. Architecturally it’s simpler than block based storage; you just need a good local area network and a storage system that can export a file based protocol. There’s no need for specialized hardware or intelligent host bus adapters, just let the storage array handle the storage. Virtualization lends itself to file based storage quite well. VMDKs are just files after all. I kind of snickered when VMware first came out with their VAAI storage extensions. It seemed, to me, like they were enhancing block-level storage devices to do a lot of what NAS based storage already does.

While I was taking my VCP4 class my colleges, most of whom were from big companies, snickered when I mentioned that our storage was on a NAS. A “filer” for them was a place for document sharing and storage. There was “no way” it would ever be fast enough, or good enough to backend their virtualized infrastructure. I’ve seen that notion fade more and more as ZFS has opened the doors for storage startups; and the big players are fighting back with their own specialized NAS devices. There are some really cool ideas floating around: NAS devices that are scale-out, that are optimized for virtualization, and that can do in-line deduplication of data.

That being said…

I have learned that there are some OS level tweaks that can enhance performance on virtual machines. Most x86 operating systems seem to be optimized for single disks, or internal RAID setups. Understandable as that has traditionally been the bulk of their install base. This means that the OS can manage disk queuing better that the dumb RAID card, or the dumber hard drive. CFQ, the default disk scheduler as of kernel 2.6.18 does this. As I understand it CFQ breaks synchronous read/write requests into queues, and assigns timeslices to each queue, weighted by IO priority. The effect is that higher priority processes get longer queues which keeps IO requests from the same process close together. Great idea when the OS has direct access and is managing the storage. Not so great when the storage is handled remotely; the array on the other side is doing the scheduling. All of that optimization is ostensibly ignored. So for a virtual machine it’s better to switch to a simpler algorithm and let the storage array handle the write queuing.

From my reading (and testing) It’s better to switch to the noop scheduler. Noop simply shoves all requests into a first-in-first-out (FIFO) queue and can merge requests. It is simple, fast, and is great for flash storage (no mechanical latency) or for situations where optimization is handled by another device. Like a NAS! Perfect for virtualization!

I discovered this after getting a snippet of a shell script to try from Mike (who got it from a potential vendor that is a big storage geek). This wasn’t new information as Mike had mentioned this almost 18 months ago in passing but neither he nor myself ever tested it. After giving me the info, again, he suggested that I “test this out, and let me know if it works.”.

I’m still testing it, so caveat emptor, but I thought I’d share it with you.

***WARNING DO NOT DO THIS ON A VM WITH SNAPSHOTS***

#!/bin/sh

grep '' /sys/block/sd*/queue/scheduler
for d in /sys/block/sd*; do
echo noop > $d/queue/scheduler
done
grep '' /sys/block/sd*/queue/scheduler

This switches the scheduler from cfq to noop on all “SCSI” disks in the virtual machine.

He also added the following tweak to increase the read-ahead from 256 sectors to 1000 sectors, which caches more disk data for faster read times, after printing what the OS has mounted.

#!/bin/sh

mount
blockdev --getra /dev/sd?
blockdev --setra 10000 /dev/sd?
blockdev --getra /dev/sd?

Again, I’m still testing this on my personal stuff, but, qualitatively, things feel a lot faster. If anything, I haven’t crashed my Linux systems.

Anyways, I hope that helps!

MinneDemo happens two to three times a year and I think this has to be one of the most exciting MinneStar events (although, I have yet to attend MinneBBQ).

As you enter the area of presentations you feel a great surge of energy from the presenters and the crowd. There is an excellent mix of different age groups that attend. The first hour is for networking, eating, and of course – drinking. Once you have had your fill of food and conversation, you head into a medium-sized auditorium to take your seat, wait, and feel the anticipation build.

All of the presenters have a spark that ignites the entire audience sucking them into the presentation.

What are people presenting at MinneDemo?

Presenters show off real, working technology products created locally in Minnesota. One not only feels energy but a sense of pride in our great state!

I am just going to touch on two of the different presenters that I enjoyed wholeheartedly but before I do that I would like to congratulate Code 42 software, the company behind CrashPlan for landing a $52.5M growth capital investment round.

In case you want to catch the entire show there is video footage of the event and more great links on the video page.

Brahmageddon is an iOS game that is a lot like wack-a-mole but, with a spice of Hindu mythology. The company that makes the game, Bust Out Solutions, is based in Minneapolis, Minnesota. It was quite entertaining to watch the game being played and hearing the Hindu music in the background. The graphics are magical and I would love to meet the artist who created the Hindu demons. The game takes on the same effect that Angry Birds.What I mean by this is it is a fast paced game that is easy to understand with a pinch of challenge to keep the players enthralled. It is quite simple and an easy way to acquire entertainment on the go. I could easily see myself playing this game on the bus ride home from work. The audience for the game seems like it would cover all demographics. It was also encouraging that the creators still find the game quite fun to play.

Red Stamp’s presentation takes the quirky, stylish greeting cards that you would buy at say a Paper Source and puts it into a mobile correspondence. It was a pleasure to watch a company that moved with the times and did not just stay in the paper world. I think Red Stamp represents a beautiful evolution in the greeting card. Red Stamp is really catering to the crowd that just does not have time to stop by a store or order a card and have it shipped their way. I think this crowd of busy people is growing and Red Stamp is growing right along with them. It is refreshing to see a company that instead of saying “no we won’t do that” said “hey, we are up to the challenge and can make this happen”.

If you would like a kick in the butt to remember what dreams you used to have and what it is like to strive to make those dreams come true, you should check out MinneDemo.

And again, if you would like to see all the presentations, check out the video footage.

I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

The lesson I learned: Feature freeze is a good thing. Know when to stop fixing.

Now early in the project, I had a pretty good idea of what pieces needed to go together but I did not have a very good idea of how to get there. I put down a quick design and while I was doing that I started to see problems..

• Pieces did not fit together.
• Some things were missing.
• This was not going to work.

Time to start learning. I love learning.

So this is the good part. This is the fun part. This is where things go click. More on that in a moment.

I strive to make every day a boring day on my production servers. That doesn’t mean that they don’t do cool stuff. That doesn’t mean I don’t like my job or that I find it dull. I don’t. I just like to be beyond the point of being surprised, pleasantly or otherwise, when I am doing something for a client. That’s the goal.

Now technology moves too fast to be expert in everything. There will always be opportunities to learn something new, but my goal is to make the systems I run not scratch that itch for new learning. All that is to say that I have a craving for learning new stuff that is not and should not be filled by the day-to-day work I fit my learning projects around.

So there I am in the middle of doing a general something new. I have a general idea of how it goes, but there is a part missing or a process that I don’t know how to do. I know the next step, but not quite how to get there. Then I learn how, or I learn it is not going to work and I find a new way. My favorite learning is when I learn something new that brings two formerly unrelated pieces together in my mind and they fit together. That click is one of my favorite experiences in life.

After a while of this, things started to make a lot more sense. I had something that would mostly work. There were still some things to optimize, and some things to work out. More of the fun part. Here’s the problem: It gets addicting to learn stuff. If you’ve ever gone to Wikipedia and seen hours magically vanish, you know what I am talking about.

At some point though, it is time to stop fixing for a while. Freeze the design and commit to finish the thing, even though it is broken. Make version 1.0. There are a lot of things you can learn from seeing the finished system even with its flaws that you can’t see by looking at the parts.

I’m not saying leave it broken. Make version 2, and version 3, if you want but at some early point, freeze the specification, stop making changes, stop making fixes, go finish. Exercising the discipline to stop following new opportunities to improve my project, stop fixing, and stop learning (for a moment) meant the difference between having an imperfect but completed project and having a whole ton of good ideas.

So I stopped and did the drudge work. I finished. I made something imperfect. My reward? In addition to the problems I deliberately ignored, I see lots of little details that need fixing, which I would never have seen without finishing.

Off to version 2.0. I love learning.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Here’s a script that I use to backup my Maildir. Normally I write my own, but I was feeling lazy. So in the interest of not re-inventing the wheel, I decided to copy one. Unfortunately, I forgot where I got it. Normally I’m good about attributing things that I copy/utilize. So, if this is your script, thank you! If you want attribution, please let me know.

This is just a bash script. It creates a xz’d tar file in my /home/nick/Archives directory/

First, check for input.’

#----script to backup files

if [ $# -lt 1 ]
then
   echo Usage: backup.sh Directory
   exit
fi

Next, create the date stamp.

JJJ=`date '+%d%m%y'`

This is the main loop. It processes for each in the attribute list ($#)

while [ $# -gt 0 ]
do

Print the directory we’re currently working on, then set the DIR variable to that directory’s path.

   echo $1
   DIR=$1

Truncate the directory path of the DIR variable

   MODDIR=`basename $DIR`

Verify the directory exists before doing anything

   #----check file exists
   if [ ! -d $DIR ]
   then
      echo Error: File \'$MODDIR\' not found!!

Check to see if the destination folder exists. This is where I hard coded it to be /home/user/Archives. If it doesn’t, create the backup locally.

   else
      if [ -d ~/Archives ]
      then
        DESTTAR=~/Archives/$MODDIR.$JJJ.tar.xz
      else
        DESTTAR=$MODDIR.$JJJ.tar.xz
      fi

Check to see of the destination file exists. If it does, complain and exit.

    if [ -d $DESTTAR ]
        then
                echo Error: File \'$DESTTAR\' already exists!!
                exit 0

Tar the archive using the xz compression method (the “J” attribute in the tar command)

    fi
      tar -cJf $DESTTAR $DIR
   fi
   shift
done

And we’re done! At some point I’ll add an auto-pruning portion to the script.

I just stick the script in my personal bin folder, make it executable, edit my crontab and put an entry in it pointing to my Maildir, and I get nightlly backups of all my mail.

Building a cubicle, no matter what the instructions say, is not exactly trivial. Even after putting the walls together, trying to get them all even and lined up is a trick. Then putting up the desk supports and the desk surfaces themselves, which are usually massive slabs of particle board and plastic veneer. If they’re fastened with screws, it’s always from beneath, out of the light, where you’re certain to get wood dust in your face.

In stark contrast, using vCloud Director to build a virtual machine was easy. Login, select a hardware and OS template, optionally configure, and deploy. It was up and running in a few minutes, maybe a little longer if you have added software or network configuration requirements. It’s all done with a keyboard and mouse, and I’m pretty certain I’ve never gotten wood dust in my face from it.

In both cases, a little planning goes a long way. Making a map of your cubicle layout lets you know whether everything is going to fit, what you’ll need, and whether you have it. Updating your map when you change your mind is also important. Making a map of your systems and network is similarly illuminating. How much CPU, memory, and disk do you need? What systems need outside IP addresses?

Also, paying attention to the details pays off. It’s a lot easier to make the desks level if all the supports are attached at the same height. Double check your IP address assignments and DNS records.

Finally, building a virtual machine doesn’t involve any power tools. Well, let’s hope not.

A Kickstart file basically answers the questions that pop up in the installer as the installer goes removing the need for human interaction. If an question isn’t answered, the installer pops up with the proper dialog, takes user input, and continues. I can pick and choose what information I want to populate automatically and which information dialogs I want the customer to answer. In my auto install ISOs I prompt the customer for a username and password as I want the users to enter that information.

When I was tasked with making an auto installing ISO for our customers I was able to create one quickly by using a kickstart file.

The process of making a CD is a bit verbose, and better handled by some of the how-tos out there.

But I’ll take your through my Kickstart file.

First are some of basic information about the system. These are fairly self-explanatory.

platform=AMD64
#System language
lang en_US
#Language modules to install
langsupport en_US
#System keyboard
keyboard us
#System mouse
mouse none
#System timezone
timezone America/Chicago

I disable root, to reflect the Ubuntu default. You can enable it by removing the next line, and setting it with the second.

rootpw --disabled
#rootpw jpDhuZtql4of4rfq

I do not automatically add a user, but you can with the next line.

#user johndoe --fullname "John Doe" --password changeme

I don’t think this does much in an Ubuntu Server install but I put it in anyways.

#Use text mode install
text

We’re installing not upgrading.

#Install OS instead of upgrade
install

Use the CD-ROM.

#Use CDROM installation media
cdrom

Where are we going to put the bootloader?

#System bootloader configuration
bootloader --location=mbr

Get rid of any existing partitions.

#Partition clearing information
clearpart --all --initlabel

Partition the disks using Ubuntu defaults (512MB swap, etc) This allows the ISO to work on whatever size disk you want. Linux isn’t great about using swap anyways, so 512 is plenty.

#Disk partitioning information
part /boot --fstype ext3 --size=200 --ondisk=hda
part swap --recommended
part / --fstype ext4 --size 1 --grow

Passwd information. I know… MD5… You can use something more secure if you wish.

#System authorization infomation
auth  --useshadow  --enablemd5

We need DHCP for some of the following steps, as I have no idea what type of network this will be run on. You can specify other info here if you want.

#Network information
network --bootproto=dhcp --device=eth0

My customers hate having UFW on. I don’t think this actually works yet in Ubuntu, so I also do it in a later script.

#Firewall configuration
firewall --disabled

X-Windows on a Server? No thanks.

#Do not configure the X Window System
skipx

And finally, we want to reboot after installing. This isn’t actually done, as we’re going to run a post-install script.

#Reboot after installation
reboot

Add additional packages to install. I install the fewest here, as I update in a later script, so why install a bunch of stuff only to update it later?

%packages
@dns-server
@openssh-server
gcc
build-essential

Here comes a a post install script.

%post

Mount the CD again, as there’s data we want off of the CD.

echo Making CD Mountpoint
mkdir -p /mnt/cdrom
echo Mounting CD
mount -t iso9660 /dev/sr0 /mnt/cdrom

Copy over a script that I’ve written that does updates and additional installs when the virtual machine is first booted.

echo Copying Firstboot Script
cp /mnt/cdrom/firstboot /etc/init.d/
chmod +x /etc/init.d/firstboot

Updated the init structure to run the firstboot script on boot.

update-rc.d firstboot defaults
echo Adding new Crontab

Add a custom crontab with some randomized sleep values.

cp /mnt/cdrom/crontab-template /etc/crontab

A script that I wrote that edits resolv.conf to point to the local bind server

echo Copying resolvfix init script
cp /mnt/cdrom/resolvfix /etc/init.d/
chmod +x /etc/init.d/resolvfix
update-rc.d resolvfix start 99 2 3 4 5 .

An updated sources.list with a closer mirror.

echo Copying Apt Sources
cp /mnt/cdrom/geeks-org-sources.list /etc/apt/sources.list

A new dhclient with the local bind server seeded.

echo Copying dhclient.conf
cp /mnt/cdrom/dhclient.conf /etc/dhcp3/

A new named.conf.options with some useful defaults.

echo Copying named.conf.options
cp /mnt/cdrom/named.conf.options /etc/bind/

Moving over vmware-tools for installation upon first boot.

mkdir /vmware
cd /vmware
echo Extracting Tools
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz

Ejecting the CD.

echo Unmounting CD
umount /mnt/cdrom

Update the system.

echo Updating
apt-get update
apt-get -y dist-upgrade

And finally, reboot the system (sync for good luck ;) ).

echo Rebooting
sync
reboot

Now, as I mentioned before, there’s a firstboot script that I run that does quite a bit of work before the machine is finished. It does things like wipe out the SSH keys, install VMware Tools, remove and purge old kernels and install applications like MySQL, Apache, as required.

Well, that’s one of the tricks I have tucked up my sleeve, I hope it helps!

Then I stumbled on this link (conshell.net) which explains how to use dd and netcat to copy a disk over a network.

It was so simple, it verged on genius! But did it work?

The steps are easy:

1) Create a virtual machine with  a disk about the same size or larger than your source (not smaller)

Pick an arbitrary port, (9001 in this example) and set up your firewall or VSE to allow that port to the target machine.

2) Boot that new VM into a rescue environment or use a live cd.

3) Use the following commands:

On the VM: nc -l -p 9001 | dd of=/dev/sda

On your source machine: dd if=/dev/sda | nc 9001

4) Wait a long time… I averaged around 15Mbps from my test machine to my new VM, it ranged from 30Mbps down to 7Mbps. I’m sure that had more to do with my network than anything. Still, this can take a while.

5) Once the dd has completed (dd will dump summary information) reboot the machine back into the live-cd environment, check the partitions with e2fsck the partitions and re-size them. (I cheated and used gparted)

6) At this point you can either mount the filesystem and remove the udev rules (in /etc/udev/rules.d/) or boot into your VM and remove them via the console. Either way, you have to reboot after the udev rule are removed.

7) Reboot, and voilà!

The live cd I used was CDLinux. It’s a small Linux distribution that runs XFCE, and fits in an 80MB ISO. It also includes an SSH server, so you can set up an ssh tunnel, and use netcat against that rather than use an arbitrary port. It also has the VMware paravirtual scsi drivers.

Anyways, this worked. Wow did it work. I didn’t bother to zero out the remaining space on the disk, it took me about 2.5 hours to move 8GB worth of data but I was greeted with a familiar prompt in a new place as soon as I booted it up.

Now a couple of caveats. I did this on a running system, with no prep work. I would recommend trimming unnecessary data and shutting down as many services as you can. It’s best to do this when the machine is “down,” not doing anything beyond facilitating the copy. However, it does work on a live system. Still, if I were moving a production system, I would follow the advice in the linked article above.

But, it was my system, in a test environment, so I didn’t really care.

Still, isn’t it amazing what a couple of UNIX pipes can do?