Debugging what is going wrong with a VPN setup is difficult. The IKE protocol is “chatty”, and negotiates back and forth between the two ends for several rounds. The GUI offers not much help, it is either  UP or Down. Most of the real debugging happens inside the CLI.

One problem in particular that has always bugged me is that you need access to the end machines involved to initiate traffic across the link. The network admin typically doesn’t have direct access on the computers on either side of the VPN in order to initiate that traffic. I’ll show you a method that can be used to initiate traffic from that network as well.

Here are some basic steps to troubleshoot VPNs for FortiGate.

In IKE/IPSec, there are two phases to establish the tunnel. Phase1 is the basic setup and getting the two ends talking. Then IKE takes  over in Phase2 to negotiate the shared key with periodic key rotation as well as dealing with NAT-T (NAT tunnelling), and all the other “higher-end” parameters.

The first trouble shooting step is to verify your parameters are all correct and matching.

For Phase1, is the end gateway dynamic or static? Fortigate to Fortigate can use both Main and Aggressive modes for dynamic connections, but many other brands can not. In general, if you are supporting a dynamic IP client end, you will have to use Aggressive mode Phase1, so make sure that mode is set for dynamic clients. If this a static config, you should use Main mode for Phase1, which is a bit more secure on the initial handshake.

For Phase2, are both sides setup to use PFS? Replay Detection? Dead-peer detection? While most VPN setups include a set of encryption and hash algorithms, you only need one that are the same. The reason for the set is to offer many choices. In practice, just pick one that your base client supports and go from there. Now-a-days, AES256/SHA1 is probably supported across the board, and that is all I ever use. You don’t have to match the set of them exactly, each side just needs a common one to talk.

After that all checks out, we need to see what IKE is doing that is failing.

So SSH or console into the CLI.

If this is debugging a VDOM
(like in this case), you may have to switch into the root VDOM if you
are the system admin of the firewall as opposed to a VDOM admin.

fgt300C-fw # config vdom
fgt300C-fw # edit root
current vf=root:0

fgt300C-fw (root) #

as the diag commands are only available in the individual VDOMs or from the root VDOM for the system admin.

To enable debug logging on the console (should be default) do

fgt300C-fw (root) # diagnose debug console

To enable debugging output

fgt300C-fw (root) # diagnose debug enable

Phase1 debugging isn’t too useful. IKE/Phase2 debugging is where the problem almost always is. Lets turn on full debugging logs there.

fgt300C-fw (root) # diagnose debug application ike -1

Now, the problem I’ve always run up against is getting the tunnel to trigger to open up with traffic running on the link. You either have to conference in somebody with access to help you, or use this nifty trick…

Open another SSH connection to the FW CLI.  (If this is a VDOM, you’ll have to ‘conf vdom; edit “vdom3″ to get into
the VDOM context where the network is you want to troubleshoot).

Set the ping source IP address to be in the inside network of the host you are trying to troubleshoot..

fgt300C-fw (vdom3) # execute ping-options source 172.30.3.254

And now, ping away from the CLI in order to bring up the tunnel interface

fgt300C-fw (vdom3) # execute ping 192.168.0.1

(assuming 192.168.0.1 is an existing host only reachable via the VPN tunnel, and the ping service is allowed through the tunnel).

fgt300C-fw (vdom3) # execute ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=46.9 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=47.3 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=45.5 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=66.3 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=45.7 ms

--- 192.168.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 45.5/50.3/66.3 ms

The trick here is that you are source as the network you are setting up, which should trigger the tunnel to come up if it isn’t up already, and you can see real live traffic. I don’t know how many times I’ve been stuck on a conference call waiting for whoever had access to do something to get around to doing the test I asked of them.

Back in the first debug window, you should see a whole bunch of IPSec and IKE messages fly past on the screen.

You have to learn to pick out the lines that are important, and zone in on them as everything is flying by. Learn to pause the display (or do a quick ‘diag debug dis’ to stop the output). Scrolling back and zeroing in on the one error out of 100 lines is going to be your key skill here.

If all is well, you should get something about the SA being established with the SPI value (not important).

ike 3:MyVPN_GW:18690:MyVPN:49143: added IPsec SA: SPIs=939fc892/b54d030

and of course, if it is configured for SNMP, something like

ike 3:MyVPN_GW:18690:MyVPN:49143: sending SNMP tunnel UP trap

is a nice confirmation that all is well with the VPN.

If you are seeing a lot of errors repeating with Phase1, and you see messages like

ike 3:MyVPN_GW:18698: sent IKE msg (P1_RETRANSMIT): ....

Most likely the problem is a mismatch preshare key for the VPN tunnel, as it isn’t passing out of P1 (which doesn’t have much to negotiate).

Also check again if this is dynamic client (generally requiring Aggressive mode) or a static connection that probably should be set to Main mode, but could be using Aggressive Mode.

If you don’t have a common encryption alg/hash, you should see some errors like..

ike 3:MyVPN_GW:18707: no SA proposal chosen

As it can’t find a matching SA between the two ends using the same encryption algorithm/hash combo to encrypt the tunnel. Fixup the encryption alg/hash and everything should go better.

The hardest problems to detect are different keylength timers (you’ll just have to review them on both sides to make sure your P1 and P2 keylife timers are identical on both sides). Problems that you encounter with different timers show up as a VPN that works for a while, but then stops work, and won’t come up unless you bounce both sides. With valid timers the same on both sides, the VPN should keep up and key rollovers happen automatically.

Also, DPD may not always negotiate. One side may have it on and let a VPN connection stay up for a certain time until the timer kicks off and closes the connection for the lack of keep-alive packets. Make sure both sides have it on, or both sides have it off.

There are a few other error conditions that may come up, but these are the more common errors.

The most important thing with the low level debugging like this is to learn to pick out the important error lines from all the rest of the junk flying by. It just takes practice. You may want to deliberately break an existing setup just to see what happens. But once you can zero in on that one error line out of a 100 that is important, it will be a lot easier to troubleshoot what problems may come at you.

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

It gets you out of hardware. Out of substantial up-front costs, management and repair, depreciation, and end-of-life planning.

It gets you out of data centering. Out of power, cooling, and cabling overhead and management.

It might even get you out of this. With a virtualized infrastructure, you can get access to and administer your servers and network from almost anywhere. From your office, your home, the beach…

What else could you be getting out of with a virtual data center?

I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Setting up a monitoring agent on your local network is easy. The server hosting the agent just needs a JRE (Java Runtime Environment) installed using version 1.6 or greater and must be able to make an outgoing SSL connection. To monitor Windows systems, you’ll need to install the agent on a Windows server.

Login to the LogicMonitor website, click on the “Settings” tab, then on “Agents” in the left navigation, then on the “Add” button. Click past the introduction, and indicate whether you’ll be installing the LogicMonitor agent on a Windows or Linux server. Download the agent installer, or copy the link and use wget to download the installer directly to your Linux system. Run the installer to install the agent on your server then return to your web browser and click “Next” to verify that its been installed correctly and is able to communicate with the LogicMonitor system.

To begin monitoring a host on your network, click on the “Hosts” tab, then on the “Add Hosts” button and select “New Host (wizard)”. Enter the host name or IP address. Note that if your monitoring agent and host are on a private internal network then this should be the IP address visible to your agent. Select your monitoring agent (if you have more than 1), and LogicMonitor will go ahead and verify that its able to gather information about the host.

NOTE: at this time, LogicMonitor does not support IPv6

I was in great need of Nick’s help because at first I thought I could create just any host name. So of course I chose the name barf. Well you can’t do that. You need to use a machine name that already exists. Nick said I should choose smtpgrey-2.iphouse.net or smtpgrey-1.iphouse.net (inbound SMTP border servers in use on our network).

Once I figured that out it was all smooth sailing!

Below is the Logic Monitor interface.

If you look to the left, my name is Gen.

So you say you would like to add a host. Click on the ‘add a host’ button on the left side. Then this screen pops up.

You have to pick a host name that already exists. Now you select an monitoring agent. I chose worldgen. You do need to choose one that will Windows based if it is a Windows machine so you can use WMI if your firewall allows such. Remember that.

Once you pick your agent, the wizard will check to see if every choice you made works out. Apparently, I did things correctly.

I decided that I do not want to add another host. I think I am good for now.

Okay, okay…I am sure I want to exit.

So now, this is what Logic Monitor really does. It monitors.

You can toggle around and see what has gone on with your CPU usage and gather load averages plus a whole slew of other statistics.

Now it is monitoring Disk. Watch it monitor.

DNS status seems to be neutral.

Okay this is the part that I think is the most interesting about Logic Monitor, the alerts. I know that no one wants to see a critical message. It sure is fun when you are testing new software though. I did not have any alerts so I moved on to show you someone elses alerts.

Here we go, this is critical as you can see from the bright orange and yellow colors.

There is a section under notes that you can click on and add in a note on what is going on or how you are going to fix it or if you did.

After all of this, I think Logic Monitor seems quite useful. It is nice that when an alert happens (because they do) a page can be sent to your cell phone.

Then I stumbled on this link (conshell.net) which explains how to use dd and netcat to copy a disk over a network.

It was so simple, it verged on genius! But did it work?

The steps are easy:

1) Create a virtual machine with  a disk about the same size or larger than your source (not smaller)

Pick an arbitrary port, (9001 in this example) and set up your firewall or VSE to allow that port to the target machine.

2) Boot that new VM into a rescue environment or use a live cd.

3) Use the following commands:

On the VM: nc -l -p 9001 | dd of=/dev/sda

On your source machine: dd if=/dev/sda | nc 9001

4) Wait a long time… I averaged around 15Mbps from my test machine to my new VM, it ranged from 30Mbps down to 7Mbps. I’m sure that had more to do with my network than anything. Still, this can take a while.

5) Once the dd has completed (dd will dump summary information) reboot the machine back into the live-cd environment, check the partitions with e2fsck the partitions and re-size them. (I cheated and used gparted)

6) At this point you can either mount the filesystem and remove the udev rules (in /etc/udev/rules.d/) or boot into your VM and remove them via the console. Either way, you have to reboot after the udev rule are removed.

7) Reboot, and voilà!

The live cd I used was CDLinux. It’s a small Linux distribution that runs XFCE, and fits in an 80MB ISO. It also includes an SSH server, so you can set up an ssh tunnel, and use netcat against that rather than use an arbitrary port. It also has the VMware paravirtual scsi drivers.

Anyways, this worked. Wow did it work. I didn’t bother to zero out the remaining space on the disk, it took me about 2.5 hours to move 8GB worth of data but I was greeted with a familiar prompt in a new place as soon as I booted it up.

Now a couple of caveats. I did this on a running system, with no prep work. I would recommend trimming unnecessary data and shutting down as many services as you can. It’s best to do this when the machine is “down,” not doing anything beyond facilitating the copy. However, it does work on a live system. Still, if I were moving a production system, I would follow the advice in the linked article above.

But, it was my system, in a test environment, so I didn’t really care.

Still, isn’t it amazing what a couple of UNIX pipes can do?

Reboot your server from the VMware Converter CD. Once VMware Converter is running, click on “Import Machine”. Click through the first couple of panels. On the “Source Data” panel, select “Import all disks and maintain size”. On the “Destination Type” panel, set the destination type to “Other VMware Virtual Machine”. On the next panel, set the VM Name and the Location where you’re storing your disk image. Since Converter doesn’t support direct OVF export here, you’ll have to set the type of virtual machine to Workstation 6.x and change it later. Click through the rest of the panels, and Finish. VMware Converter will now export your server to a .vmx description file and .vmdk disk image(s). This may take a while.

After VMware Converter is done, you’ll need to use VMware’s OVF Tool to convert the .vmx file to a .ovf file.

ovftool.exe name.vmx name

Next, the tricky bit. You’ll need to edit the .ovf file, which is in XML format, and find and update the VirtualSystemType field to vmx-07, a reasonably current version. Once you’ve edited the .ovf file, you’ll also need to generate its SHA1 hash and update the .mf manifest file. Now, you should have something which can be imported into your vmForge VDC.

Login to your vmForge VDC (or any VMware vCloud Director system), and create a new catalog if you don’t already have one. In your catalog, click on the Upload button. Select your .ovf file, give it a name and description, and click on the Upload button. Again, this may take a while. Once its done, you’ll be able to use the template from your catalog to create a new virtual machine, a clone of your old physical server. The last step will be to power on the virtual server, login via the virtual console, and reconfigure networking.

Thanks for reading my series on moving from my single all-in-one server and my small ESXi server to ipHouse’s vmForge VDC product. I previously discussed moving my websites to a virtual webcluster, and moving email to a virtual mailcluster. Now I just had to move three small servers, and install a third.

The first server I moved was a small experimental VM used for testing various network, web and other items. I like to have dedicated testing environment for every operating system that I professionally run. This server was responsible for my personal Teredo tunneling, and was the one I put my CGI testing on from awhile a go. I could have easily moved it, but I wanted see how the export/import from ESXi to vmForge worked. I stopped the machine on my ESXi server, downloaded it as a OVF and uploaded it, via my Windows machine, to my catalog. It imported it as a template. I then deployed the template and deleted the server. It worked flawlessly! All I had to do renumber the machine and I was done.

The next server was a little more complicated. It was originally a CounterStrike:Source server that I had converted into a Apache Tomcat JSP host. Because it already had a working Java setup, I added an OpenFire Jabber server, and a LogicMonitor agent to it. This gave me the ability to monitor my internal network from LogicMonitor, a monitoring solution that we’re looking into. The triple Java duties of this machine, unfortunately, put a big crunch on its RAM, so that took a lot of tweaking on the application level to get them to play nicer with each other.

The next server was a monitoring server that I had set up running Zabbix. I had previously gotten Nagios working on it, but it was too burdensome for me to maintain. I also liked having graphing and service level alerting as well as agent based checks, both active and passive. The biggest problem with Zabbix was getting it initially set up to send alerts, so it was nice to be able to import this machine, that had a working base, than to start from scratch. LogicMonitors does pretty much everything that Zabbix does, and better, but why not have two monitoring solutions? I also set up that machine to be a centralized logging server if I ever want to install a log analyzer like Splunk. I set it to copy the logs to a MySQL database, and to run php-logcon, but that didn’t scale past a few thousand entries.

Next was installing a FreeBSD server to act as a centralized tool, mail environment, and storage space for myself and my friends. I love FreeBSD, the only reason I set up my other servers as Linux boxes was pure laziness on my part, which I’ll pay for later in administration time. Also, they are mostly single purpose appliances, and it’s nice to have some of the Debian style scripting for web built-in. I try to stay fairly OS agnostic, but I do have preferences.

Since my shell server would have the most exposure to the internet, so I wanted a relatively secure system. Also, I would be spending most of my time in that server, so I decided to go with the OS I love. That would also bring things full circle, as my pfSense box and Shell server are both FreeBSD.

I decided on installing FreeBSD 8.2 stable. I sliced my disks like this:

/           512MB
swap        1GB (1x Memory)
/usr        5GB
/var        10GB (Modest space for DB and info)
/home       140GB (An egregious space for storing files)

I installed the OS and ports, and I switched from cvsup to csup awhile ago, and updated my ports-supfile and stable-supfiles to point to a local(ish) mirror, and checked out /usr/src and /usr/ports. I then updated my kernel config (Tip: compile without debugging if you want it to fit in 512MB ) reinstalled, and rebooted. Voila! A new FreeBSD system. I’ll probably go into doing a comprehensive FreeBSD install in a later post.

I installed Postfix and Dovecot2 for local mail, Apache 2 for user directories, and migrated my users information, passwords, and home directories from my old server. Everything went surprisingly smooth. I installed Mutt for myself, Alpine for one of my users, and a few other pieces of software, and I had a fully running shell server. I was going to run PowerDNS and PowerAdmin on one of my Linux boxes, but I decided to stick with BIND on the FreeBSD server, as it was more efficient for me to edit text files than use a web interface. Weird, I know. Now that my shell server was done, and everything was migrated, I could turn off my old FreeBSD box. I admit that I did feel a little bad as I typed halt into its shell for the last time. It served me well over the last four years.

Now my infrastructure migration was complete, running fully virtualized, lowering my power consumption, gaining redundancy, and boosting performance for the fraction of the cost of having physical infrastructure.

I Win!

Game Over.

Last week I discussed moving my personal infrastructure into an vmForge Virtual Data Center. I discussed setting up a pfSense firewall, and getting things ready for my various projects. The first one that I wanted to tackle was setting up a load balanced webcluster.

Backing up a bit, three years ago, I was a support tech, looking for ways to grow my knowledge base and expand my abilities. After a bit of haranguing from my boss, he suggested setting up both a web and mail cluster with the various bits of hardware lying around. The project was laid out very simply, with two machines acting as frontends, my old server acting as the share storage point, and another acting as a database server. Load balancing would be done via round-robin DNS resolution (ie multiple A records with the same name and different IPs) I mapped it out, and diagrammed, but I was not ready to implement it at the time. I wanted to know more about how Apache worked on a single server before tackling it on a cluster.

Fast forward a year and a half, and I decided to revisit the project, partly due to wanting to progress out of my support position, and partly due to the fact that I now understood Apache and MySQL a lot better. I had a VMware ESXi system, and had tried a few times to create a cluster with various amounts of success.

At the same time, I decided to try redoing how I set up PHP. I switched to a FCGID based Apache setup. Qualitatively, it felt faster in my testing, and it allowed me to tweak individual sites for various levels of support and resource utilization.

The final piece was the VDC itself. With a large NAS behind it, lots of spinning disks, and logical vApp networking, I could lay things out simply and easily the way I wanted infrastructure wise, without relying on public IP space, errant patch cables, or a stubborn old server.

The first thing I had to decide was what my layout would look like, and how big the VMs would be. I settled on two frontend machines running minimal disk for the operating system and potential disk based content caching, and a single backend machine that would be both the shared filesystem and database host. I considered installing MySQL slaves on the frontend, with the NFS/database server as the master, but none of my sites justified that amount of complexity.

The frontend machines were configured with 500 MiB of RAM and 8 GiB of disk space per. The NFS/database server was configured with 2 GiB of RAM and 128 GiB of storage.

After figuring out the layout, I had to pick what operating system to use. After some hemming and hawing, I decided on Ubuntu 10.04 LTS instead of my beloved FreeBSD. There were a couple of reasons for this. One, I like the Debian style Apache tools. They make scripting site management much much easier. Two, there were potentially a lot of Perl and Pear packages that would need to be installed, and I prefer to handle that via apt/dpkg rather than ports. Third, and formost, I had already created an auto installing ISO, and it made installing exactly what I wanted multiple times very easy.

After installing the NFS/database server, I added MySQL and the newest nfs-kernel-server for NFS4. Why NFS4? Why not?

I laid out my filesystem to export under /var (with sensible names like /var/apache-site-config /var/www, etc) and mounted them over to /export (with accessible names like www, conf.d, sites-available)

I then installed NIS and created the various users for SUEXEC and FTP. I had some trouble with idmapd, but I figured it out (RTFM, and a couple known issues about it not starting sometimes)

After getting the NFS/database server squared away, I set up Apache on the frontend machines, mounted the NFS directories in their spots, and fired it up. It worked! I then moved my personal blog over, and decided to do some tests. Like, serious load tests.

It was abysmal.

My blog made extensive use of memory caching (via mem_cache, not memcached) The combination of memory caching, FCGID and 500 MiB of RAM was too much for the frontends. I upped them to 1 GiB per, and things were perfect. Just a little more RAM made a huge difference! I did add memcached a little later, as the WordPress plugin “W3 Supercache”, and Drupal can use it, as well as other CMS/web applications.

Now, with a working model, I went back to pfSense, and set up my load balancer. pfSense uses something called “relayd” to load balance, and that’s basically all it does, it relays the TCP connection from a virtual server, to any number of actually IPs. There’s a setting elsewhere in pfSense to have it monitor states, and send any new requests from a source with an open state to the same IP, give you some persistence (or “stickiness” as they call it); a step above round robin DNS. Unfortunately, the virtual servers only answer for one port, so I had to added two: one for 80(HTTP) and one for 443(HTTPs) and they cannot cross IP protocols, so I had to add one for IPV4 and one for IPV6. So four total virtual servers, each with a separate pool of servers. This was verbose. The other major caveat is that the virtual servers only listen on the WAN interface. After some trial and error, the easiest thing to do was to set frontend 1 to also have the same IP on the LAN interface. That way, internally, machines have access to sites on the webcluster.

A representation of my Virtual Webcluster The dotted lines represent the RFC 1918 IP addresses. (click to enlarge)

After getting that setup, I did some more load testing, to see how the load balancing worked.

It was abysmal.

I didn’t have enough frontends for the amount of load that I wanted to support. However, I didn’t have enough addresses to add more machines.

I did some experimentation, and discovered that relayd would relay connections to a RFC1918 address as well as a world routable one. So, I cloned my second frontend and made two more, with 172.16.0.0 addresses. It worked flawlessly, go go virtualization. IPV6 addresses, of course, were not a problem.

After adding the two frontends, things performed better, but still, the load metrics were spiky and performance was not quite where I wanted it.

I looked into things, and discovered that my FCGID threads spawned by Apache were then forking when they got too busy. After a certain amount of spontaneous forks, the frontend tapped out until other threads died. This brought the performance of the cluster way down.

Remembering Mike’s mantra fork/exec is expensive, and bad, and should be avoided, I did some digging, and I found out that the default Apache in Ubuntu, MPM-Prefork, was designed to contain all spawned processes in order to sandbox non-threadsafe modules. This allows for extra stability and security at the expense of performance. So I switched the cluster to Apache-MPM-Worker and, voila! Smooth, predictable performance. It was much, much better.

I then started porting my users’ sites over, along with their databases. This involved quite a bit of config file parsing, and database dumping. mysqldump –database(s) came in handy here, as did setting up preview aliases via wildcard A records. It took about a week to move everything.

I then set up some scripts, sudo, and ssh private keys so that I could run scripts on the NFS/database server that would propagate commands to the frontends, and set up a couple of cron jobs for basic maintenance. I set up vsftpd for my users to upload and manage their content as well, and changed DNS for each of their sites.

Now I had a reasonably fast webcluster running. It seems to work well… I’m sure our webmaster will make fun of me, err, have something to say about my design and implementation choices though.

Now it was time to tackle one of my least favorite things. Email. Finicky, finicky, email.

Next week: The Mail Cluster.