Changing your disk scheduler on a Linux virtual machine to increase performance.

First some background of what we do with storage at ipHouse in our VMware environments.

We really like NFS. Architecturally it’s simpler than block based storage; you just need a good local area network and a storage system that can export a file based protocol. There’s no need for specialized hardware or intelligent host bus adapters, just let the storage array handle the storage. Virtualization lends itself to file based storage quite well. VMDKs are just files after all. I kind of snickered when VMware first came out with their VAAI storage extensions. It seemed, to me, like they were enhancing block-level storage devices to do a lot of what NAS based storage already does.

While I was taking my VCP4 class my colleges, most of whom were from big companies, snickered when I mentioned that our storage was on a NAS. A “filer” for them was a place for document sharing and storage. There was “no way” it would ever be fast enough, or good enough to backend their virtualized infrastructure. I’ve seen that notion fade more and more as ZFS has opened the doors for storage startups; and the big players are fighting back with their own specialized NAS devices. There are some really cool ideas floating around: NAS devices that are scale-out, that are optimized for virtualization, and that can do in-line deduplication of data.

That being said…

I have learned that there are some OS level tweaks that can enhance performance on virtual machines. Most x86 operating systems seem to be optimized for single disks, or internal RAID setups. Understandable as that has traditionally been the bulk of their install base. This means that the OS can manage disk queuing better that the dumb RAID card, or the dumber hard drive. CFQ, the default disk scheduler as of kernel 2.6.18 does this. As I understand it CFQ breaks synchronous read/write requests into queues, and assigns timeslices to each queue, weighted by IO priority. The effect is that higher priority processes get longer queues which keeps IO requests from the same process close together. Great idea when the OS has direct access and is managing the storage. Not so great when the storage is handled remotely; the array on the other side is doing the scheduling. All of that optimization is ostensibly ignored. So for a virtual machine it’s better to switch to a simpler algorithm and let the storage array handle the write queuing.

From my reading (and testing) It’s better to switch to the noop scheduler. Noop simply shoves all requests into a first-in-first-out (FIFO) queue and can merge requests. It is simple, fast, and is great for flash storage (no mechanical latency) or for situations where optimization is handled by another device. Like a NAS! Perfect for virtualization!

I discovered this after getting a snippet of a shell script to try from Mike (who got it from a potential vendor that is a big storage geek). This wasn’t new information as Mike had mentioned this almost 18 months ago in passing but neither he nor myself ever tested it. After giving me the info, again, he suggested that I “test this out, and let me know if it works.”.

I’m still testing it, so caveat emptor, but I thought I’d share it with you.

***WARNING DO NOT DO THIS ON A VM WITH SNAPSHOTS***

#!/bin/sh

grep '' /sys/block/sd*/queue/scheduler
for d in /sys/block/sd*; do
echo noop > $d/queue/scheduler
done
grep '' /sys/block/sd*/queue/scheduler

This switches the scheduler from cfq to noop on all “SCSI” disks in the virtual machine.

He also added the following tweak to increase the read-ahead from 256 sectors to 1000 sectors, which caches more disk data for faster read times, after printing what the OS has mounted.

#!/bin/sh

mount
blockdev --getra /dev/sd?
blockdev --setra 10000 /dev/sd?
blockdev --getra /dev/sd?

Again, I’m still testing this on my personal stuff, but, qualitatively, things feel a lot faster. If anything, I haven’t crashed my Linux systems.

Anyways, I hope that helps!

I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

ZFS Version 28 adds some of the more important features of ZFS: Deduplication, triple parity RAIDZ3, and RAIDZ. This means that I can have full featured storage devices, running ZFS natively, via FreeBSD.

As a bonus I don’t have to learn Solaris.

You can run ZFS in Linux but you would either have to run via FUSE which is file system emulation in user-space, not in the kernel. Or download it and build it yourself. In my opinion, both of those options are idiotic. I’m not willing to jump through those kind of hoops just to run a filesystem in Linux. I’d rather have native, in kernel support for it. Until now, your choice was either run Solaris (or a fork of Solaris) or run an outdated version of ZFS via FreeBSD.

One project that should directly benefit of this: FreeNAS. FreeNAS is a customized installation of FreeBSD designed to operate as a NAS and iSCSI SAN. It has a pretty slick ajax/web interface as of version 8 but so far had missed out on key ZFS features.

One reason I want run up FreeBSD 9 and ZFS is to better learn ZFS troubleshooting and administration. FreeNAS aside, there are a lot of vendor supported storage devices that are coming into the market based on ZFS. I want to troubleshoot those devices on a lower level. Before this, I would have to install Solaris. This means that I would actually have to navigate to Oracle’s Website. No thank you.

In-line deduplication is on of my favorite impractical features of all time. It unfortunately, required gobs of memory (8 GB RAM for every 1 TB of storage, if memory serves) Hopefully, someone smart will figure out how to do it on flash, in a practical way, as rebuilding those tables after a power failure would suck. (see Mike’s post about Tegile – a company actually doing such in production today)

Obviously I don’t know a lot about ZFS yet which is why I’m glad I get to learn via FreeBSD.

If only I could convince ipHouse to give me a little more storage space on my personal VDC…hint hint!

I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Here’s a script that I use to backup my Maildir. Normally I write my own, but I was feeling lazy. So in the interest of not re-inventing the wheel, I decided to copy one. Unfortunately, I forgot where I got it. Normally I’m good about attributing things that I copy/utilize. So, if this is your script, thank you! If you want attribution, please let me know.

This is just a bash script. It creates a xz’d tar file in my /home/nick/Archives directory/

First, check for input.’

#----script to backup files

if [ $# -lt 1 ]
then
   echo Usage: backup.sh Directory
   exit
fi

Next, create the date stamp.

JJJ=`date '+%d%m%y'`

This is the main loop. It processes for each in the attribute list ($#)

while [ $# -gt 0 ]
do

Print the directory we’re currently working on, then set the DIR variable to that directory’s path.

   echo $1
   DIR=$1

Truncate the directory path of the DIR variable

   MODDIR=`basename $DIR`

Verify the directory exists before doing anything

   #----check file exists
   if [ ! -d $DIR ]
   then
      echo Error: File \'$MODDIR\' not found!!

Check to see if the destination folder exists. This is where I hard coded it to be /home/user/Archives. If it doesn’t, create the backup locally.

   else
      if [ -d ~/Archives ]
      then
        DESTTAR=~/Archives/$MODDIR.$JJJ.tar.xz
      else
        DESTTAR=$MODDIR.$JJJ.tar.xz
      fi

Check to see of the destination file exists. If it does, complain and exit.

    if [ -d $DESTTAR ]
        then
                echo Error: File \'$DESTTAR\' already exists!!
                exit 0

Tar the archive using the xz compression method (the “J” attribute in the tar command)

    fi
      tar -cJf $DESTTAR $DIR
   fi
   shift
done

And we’re done! At some point I’ll add an auto-pruning portion to the script.

I just stick the script in my personal bin folder, make it executable, edit my crontab and put an entry in it pointing to my Maildir, and I get nightlly backups of all my mail.

A Kickstart file basically answers the questions that pop up in the installer as the installer goes removing the need for human interaction. If an question isn’t answered, the installer pops up with the proper dialog, takes user input, and continues. I can pick and choose what information I want to populate automatically and which information dialogs I want the customer to answer. In my auto install ISOs I prompt the customer for a username and password as I want the users to enter that information.

When I was tasked with making an auto installing ISO for our customers I was able to create one quickly by using a kickstart file.

The process of making a CD is a bit verbose, and better handled by some of the how-tos out there.

But I’ll take your through my Kickstart file.

First are some of basic information about the system. These are fairly self-explanatory.

platform=AMD64
#System language
lang en_US
#Language modules to install
langsupport en_US
#System keyboard
keyboard us
#System mouse
mouse none
#System timezone
timezone America/Chicago

I disable root, to reflect the Ubuntu default. You can enable it by removing the next line, and setting it with the second.

rootpw --disabled
#rootpw jpDhuZtql4of4rfq

I do not automatically add a user, but you can with the next line.

#user johndoe --fullname "John Doe" --password changeme

I don’t think this does much in an Ubuntu Server install but I put it in anyways.

#Use text mode install
text

We’re installing not upgrading.

#Install OS instead of upgrade
install

Use the CD-ROM.

#Use CDROM installation media
cdrom

Where are we going to put the bootloader?

#System bootloader configuration
bootloader --location=mbr

Get rid of any existing partitions.

#Partition clearing information
clearpart --all --initlabel

Partition the disks using Ubuntu defaults (512MB swap, etc) This allows the ISO to work on whatever size disk you want. Linux isn’t great about using swap anyways, so 512 is plenty.

#Disk partitioning information
part /boot --fstype ext3 --size=200 --ondisk=hda
part swap --recommended
part / --fstype ext4 --size 1 --grow

Passwd information. I know… MD5… You can use something more secure if you wish.

#System authorization infomation
auth  --useshadow  --enablemd5

We need DHCP for some of the following steps, as I have no idea what type of network this will be run on. You can specify other info here if you want.

#Network information
network --bootproto=dhcp --device=eth0

My customers hate having UFW on. I don’t think this actually works yet in Ubuntu, so I also do it in a later script.

#Firewall configuration
firewall --disabled

X-Windows on a Server? No thanks.

#Do not configure the X Window System
skipx

And finally, we want to reboot after installing. This isn’t actually done, as we’re going to run a post-install script.

#Reboot after installation
reboot

Add additional packages to install. I install the fewest here, as I update in a later script, so why install a bunch of stuff only to update it later?

%packages
@dns-server
@openssh-server
gcc
build-essential

Here comes a a post install script.

%post

Mount the CD again, as there’s data we want off of the CD.

echo Making CD Mountpoint
mkdir -p /mnt/cdrom
echo Mounting CD
mount -t iso9660 /dev/sr0 /mnt/cdrom

Copy over a script that I’ve written that does updates and additional installs when the virtual machine is first booted.

echo Copying Firstboot Script
cp /mnt/cdrom/firstboot /etc/init.d/
chmod +x /etc/init.d/firstboot

Updated the init structure to run the firstboot script on boot.

update-rc.d firstboot defaults
echo Adding new Crontab

Add a custom crontab with some randomized sleep values.

cp /mnt/cdrom/crontab-template /etc/crontab

A script that I wrote that edits resolv.conf to point to the local bind server

echo Copying resolvfix init script
cp /mnt/cdrom/resolvfix /etc/init.d/
chmod +x /etc/init.d/resolvfix
update-rc.d resolvfix start 99 2 3 4 5 .

An updated sources.list with a closer mirror.

echo Copying Apt Sources
cp /mnt/cdrom/geeks-org-sources.list /etc/apt/sources.list

A new dhclient with the local bind server seeded.

echo Copying dhclient.conf
cp /mnt/cdrom/dhclient.conf /etc/dhcp3/

A new named.conf.options with some useful defaults.

echo Copying named.conf.options
cp /mnt/cdrom/named.conf.options /etc/bind/

Moving over vmware-tools for installation upon first boot.

mkdir /vmware
cd /vmware
echo Extracting Tools
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz

Ejecting the CD.

echo Unmounting CD
umount /mnt/cdrom

Update the system.

echo Updating
apt-get update
apt-get -y dist-upgrade

And finally, reboot the system (sync for good luck ;) ).

echo Rebooting
sync
reboot

Now, as I mentioned before, there’s a firstboot script that I run that does quite a bit of work before the machine is finished. It does things like wipe out the SSH keys, install VMware Tools, remove and purge old kernels and install applications like MySQL, Apache, as required.

Well, that’s one of the tricks I have tucked up my sleeve, I hope it helps!

Then I stumbled on this link (conshell.net) which explains how to use dd and netcat to copy a disk over a network.

It was so simple, it verged on genius! But did it work?

The steps are easy:

1) Create a virtual machine with  a disk about the same size or larger than your source (not smaller)

Pick an arbitrary port, (9001 in this example) and set up your firewall or VSE to allow that port to the target machine.

2) Boot that new VM into a rescue environment or use a live cd.

3) Use the following commands:

On the VM: nc -l -p 9001 | dd of=/dev/sda

On your source machine: dd if=/dev/sda | nc 9001

4) Wait a long time… I averaged around 15Mbps from my test machine to my new VM, it ranged from 30Mbps down to 7Mbps. I’m sure that had more to do with my network than anything. Still, this can take a while.

5) Once the dd has completed (dd will dump summary information) reboot the machine back into the live-cd environment, check the partitions with e2fsck the partitions and re-size them. (I cheated and used gparted)

6) At this point you can either mount the filesystem and remove the udev rules (in /etc/udev/rules.d/) or boot into your VM and remove them via the console. Either way, you have to reboot after the udev rule are removed.

7) Reboot, and voilà!

The live cd I used was CDLinux. It’s a small Linux distribution that runs XFCE, and fits in an 80MB ISO. It also includes an SSH server, so you can set up an ssh tunnel, and use netcat against that rather than use an arbitrary port. It also has the VMware paravirtual scsi drivers.

Anyways, this worked. Wow did it work. I didn’t bother to zero out the remaining space on the disk, it took me about 2.5 hours to move 8GB worth of data but I was greeted with a familiar prompt in a new place as soon as I booted it up.

Now a couple of caveats. I did this on a running system, with no prep work. I would recommend trimming unnecessary data and shutting down as many services as you can. It’s best to do this when the machine is “down,” not doing anything beyond facilitating the copy. However, it does work on a live system. Still, if I were moving a production system, I would follow the advice in the linked article above.

But, it was my system, in a test environment, so I didn’t really care.

Still, isn’t it amazing what a couple of UNIX pipes can do?

Thanks for reading my series on moving from my single all-in-one server and my small ESXi server to ipHouse’s vmForge VDC product. I previously discussed moving my websites to a virtual webcluster, and moving email to a virtual mailcluster. Now I just had to move three small servers, and install a third.

The first server I moved was a small experimental VM used for testing various network, web and other items. I like to have dedicated testing environment for every operating system that I professionally run. This server was responsible for my personal Teredo tunneling, and was the one I put my CGI testing on from awhile a go. I could have easily moved it, but I wanted see how the export/import from ESXi to vmForge worked. I stopped the machine on my ESXi server, downloaded it as a OVF and uploaded it, via my Windows machine, to my catalog. It imported it as a template. I then deployed the template and deleted the server. It worked flawlessly! All I had to do renumber the machine and I was done.

The next server was a little more complicated. It was originally a CounterStrike:Source server that I had converted into a Apache Tomcat JSP host. Because it already had a working Java setup, I added an OpenFire Jabber server, and a LogicMonitor agent to it. This gave me the ability to monitor my internal network from LogicMonitor, a monitoring solution that we’re looking into. The triple Java duties of this machine, unfortunately, put a big crunch on its RAM, so that took a lot of tweaking on the application level to get them to play nicer with each other.

The next server was a monitoring server that I had set up running Zabbix. I had previously gotten Nagios working on it, but it was too burdensome for me to maintain. I also liked having graphing and service level alerting as well as agent based checks, both active and passive. The biggest problem with Zabbix was getting it initially set up to send alerts, so it was nice to be able to import this machine, that had a working base, than to start from scratch. LogicMonitors does pretty much everything that Zabbix does, and better, but why not have two monitoring solutions? I also set up that machine to be a centralized logging server if I ever want to install a log analyzer like Splunk. I set it to copy the logs to a MySQL database, and to run php-logcon, but that didn’t scale past a few thousand entries.

Next was installing a FreeBSD server to act as a centralized tool, mail environment, and storage space for myself and my friends. I love FreeBSD, the only reason I set up my other servers as Linux boxes was pure laziness on my part, which I’ll pay for later in administration time. Also, they are mostly single purpose appliances, and it’s nice to have some of the Debian style scripting for web built-in. I try to stay fairly OS agnostic, but I do have preferences.

Since my shell server would have the most exposure to the internet, so I wanted a relatively secure system. Also, I would be spending most of my time in that server, so I decided to go with the OS I love. That would also bring things full circle, as my pfSense box and Shell server are both FreeBSD.

I decided on installing FreeBSD 8.2 stable. I sliced my disks like this:

/           512MB
swap        1GB (1x Memory)
/usr        5GB
/var        10GB (Modest space for DB and info)
/home       140GB (An egregious space for storing files)

I installed the OS and ports, and I switched from cvsup to csup awhile ago, and updated my ports-supfile and stable-supfiles to point to a local(ish) mirror, and checked out /usr/src and /usr/ports. I then updated my kernel config (Tip: compile without debugging if you want it to fit in 512MB ) reinstalled, and rebooted. Voila! A new FreeBSD system. I’ll probably go into doing a comprehensive FreeBSD install in a later post.

I installed Postfix and Dovecot2 for local mail, Apache 2 for user directories, and migrated my users information, passwords, and home directories from my old server. Everything went surprisingly smooth. I installed Mutt for myself, Alpine for one of my users, and a few other pieces of software, and I had a fully running shell server. I was going to run PowerDNS and PowerAdmin on one of my Linux boxes, but I decided to stick with BIND on the FreeBSD server, as it was more efficient for me to edit text files than use a web interface. Weird, I know. Now that my shell server was done, and everything was migrated, I could turn off my old FreeBSD box. I admit that I did feel a little bad as I typed halt into its shell for the last time. It served me well over the last four years.

Now my infrastructure migration was complete, running fully virtualized, lowering my power consumption, gaining redundancy, and boosting performance for the fraction of the cost of having physical infrastructure.

I Win!

Game Over.

Last week I discussed creating a web cluster to take over hosting websites from my decommissioned server. I wrote about setting up the layout, load balancing inside of the firewall and performance testing. This week is about setting up a load balanced mail cluster.

Now, last week I described mail as being finicky. It is. The biggest problem with email is that it was designed during a time when the Internet was a small place, where everyone basically trusted everyone else. The core protocol is very trusting and open. Unfortunately, this makes it very easy to exploit. The big problem being that people will want to send your users spam, and will want your server to send their spam to someone else. That means you need to layer security on top of any email solution to protect your server. Email consists of multiple services that each need to be protected. This makes each server rather complicated.

For the design of the mail cluster, I had the choice of going either very complex, with multiple servers performing one role, or very simple, one server performing multiple roles. The former is geared towards performance, and requires a couple of things that I don’t have: an internal load balancer, and a lot more RAM and CPU. The latter is something that I’ve done, and frankly, is hard to support and upgrade. I might write about my ideal cluster in another post, but for now, this is what I created.

My mail cluster consists of three servers. Much like my web cluster, there’s a NFS/Database server, and a couple of front-ends. Each front-end does multiple things: greylist incoming mail, scans incoming mail for spam content, scans all mail for viruses, handles POP/IMAP connections and, acts as a webmail client. The load balancer manages all connections coming from the external network, and balances the connections between the front-ends. The NFS/Database server acts as the mailstore and has the account information, passwords, and common configuration files.

Setting up the backend server was a lot like setting up my mail cluster’s NFS and databases. I would need a directory structure for storing mail using the Maildir format. I needed a database for Maia, another for PostfixAdmin/User information, and a third for SQLGrey. Easy as pie.

The first question for me was what SMTP server to use. I’m very familiar with Postfix, I like the way it handles queuing, and it’s configuration files are fairly straightforward. I’m also a big fan of Dovecot, as it does POP3, IMAP, SASL and can act as a delivery agent with sieve rules. For greylisting, I chose SQLGrey. Again, it’s something that I’m familiar with, it has a very low overhead (unlike Policyd) and it just works. For mail content scanning, I used an Amavisd based software called Maia Mailguard and ClamAV. Maia acts as a front-end to SpamAssassin, giving my users individual configuration settings, quarantine support and a few other things.

Setting up Postfix was very similar to my decommissioned server’s configuration. I enabled SASL, and followed, more or less the settings from the Dovecot wiki for an SASL setup.

I added a couple of RBLs that I find *very* effective. An RBL is a dynamic list that someone maintains of servers that you shouldn’t accept mail from. Many RBLs are too aggressive, or have draconian criteria for removal that should not be encouraged (SORBS and Barracuda, to name a couple.) The only one that I currently enabled is zen.spamhaus.org. This is a very effective RBL with a very very low rate of false-positives, and is managed fairly, and effectively. There are a couple more that I have run in the past, but most illicit mail is caught be Zen.

Next is a parameter that checks SQLGrey to see if the message should be bounced by greylisting. Greylisting is a technique that temporarily bounces messages from a new source. This prevents poorly implemented mailservers from sending mail because they usually don’t have a properly implemented queue, and just move on to the next target. Unfortunately, this can delay mail from legitimate servers, as there’s no telling how long it will take them to send re-queued mail. Also, some server that are not RFC compliant may react unpredictably to greylisting. However, it works as expected greater than 90% of the time, and cuts down on a lot of what gets filtered by Amavisd/Maia/SpamAssassin. That filtering is CPU expensive, so qreylisting allows me to handle a lot more mail than I could without it, with the resources I have available to me.

The next change I add is a “sleep 7″ smtpd_client_restrictions. This adds a delay before one of the “250 OK” responses, that ensures that an smtp server is interactive and RFC compliant, and not some flat script that is designed to blast mail. Again, this winnows the amount of incoming mail.

The next change is a content_filter parameter that pipes incoming mail to Amavisd. This shuffles mail to the content filters. As I stated before, Maia uses SpamAssassin for content filtering. This gives a set of static rules that are editable, and allows for Bayesian based filtering. Basically, Bayesian filtering is: Every message is broken into “tokens” consisting of the content between any two spaces. Each token is dynamically weighed, messages that are marked as good contain good tokens, messages considered spam contain bad tokens. The tokens in a message are added together, and if  the number of bad tokens is a certain level above the number of good tokens, the message is marked as spam, and the tokens are reassessed. Overtime, good tokens get *really* good, and bad tokens get *really* bad, but the bulk of a message will be made up of fairly neutral tokens. Maia, unfortunately, needs a lot of Perl packages. There’s a fairly useful script out there for installing Maia on Ubuntu (that I had to be slightly modified, as it was for an older Ubuntu version) You can Google it. I might be convinced to write an in depth post on how to install Maia in the future.

Virus scanning is a bit tricky, because you *must* set up Amavisd to use a ClamAV socket. Otherwise, as a backup, Amavisd will run a clamscan process on every message, and your server will be brought to its proverbial knees (fork, exec again ;)) My frontends can normally handle a few hundred messages at a time will see an exponential decrease (down to maybe a dozen at a time) when the clamscan is used.

Going back to the Postfix setup, the next thing to configure is the database settings. The database contains mail account user information. I use PostfixAdmin as a front-end for managing my users, and it has a fairly comprehensive set of setup instructions. I use a read-only user for Dovecot and a separate one for Postfix. I also put in proxy_read_maps to share the database lookups between Postfix threads.

After getting Postfix squared away, I moved to setting up Dovecot. Now, a lot winds up going through Dovecot, it handles authentication, incoming mail, managesieve, and delivery.

I set up authentication to first refer to a passwd-file. This is used to proxy IMAP/POP3/Sieve connections to another server that I have set up for users that want to have a shell environment. Yes, Dovecot can act as a proxy for the services it supports. This allows me to filter mail inside my frontends, while giving them access to the mail features of the cluster. Unfortunately, I have to manually keep the file updated, but this is much easier than setting up LDAP, It would have to be LDAP because I need to add extra fields that are not in a traditional passwd file, so, as far as I know, I could not use NIS.

I then set up Dovecot to check my PostfixAdmin database. This is a simple SQL query, with a small customization. I added a recipient delimiter of a “+” so that my users could sign up for things using something bob+spamlist@example.com. This “to” is preserved in headers so you can filter against it, but would mess up the SQL query, so I have the query edit out the “+” and anything thereafter until you get to the “@” sign. My virtual users all sign in with their full email address. This is the way PostfixAdmin formats things, and it avoids name collision.

SASL is handled via a socket, Postfix queries the socket with the username and password, and Dovecot returns whether the account info is valid or not. This, again, allows for SMTP authentication.

Incoming mail is handled by POP3 and IMAP4. I enabled SSL and TLS, and added a self-signed certificate. This was relatively straightforward, I had to tweak the Mail Location though, as I added namespace support. This allowed me to set up a “Public” mail folder for each domain, so that they could share mail without replicating too much data.

On the Dovecot side, the mail deliver is fairly mindless. I had to add parameters to support quotas for virtual mail, and sieve filtering.

Once I had the front-ends and backend setup, I had to configure the firewall and load balancer. There are a lot more TCP ports to worry about. Instead of just 80(HTTP) and 443(HTTPS), you have POP3 (110) IMAP4 (143) POP3-SSL (995) IMAP4-SSL (993) SMTP (25) SMTP-SSL (465) SUBMISSION (587) sieve (2000)

Because of the way Relayd handles load balancing, each port required a virtual server and a pool of servers to attach to. Then, they had to be doubled, one for IPv4 and IPv6. This added up to a ton of configuration overhead.

Once I got all the pieces together, I had a fairly powerful little mailcluster with content scanning, and some ability to scale. It took awhile, but it was worth doing.

Now I just had a few other application/custom servers to set up.

Next week: The Other Stuff.