Thanks for reading my series on moving from my single all-in-one server and my small ESXi server to ipHouse’s vmForge VDC product. I previously discussed moving my websites to a virtual webcluster, and moving email to a virtual mailcluster. Now I just had to move three small servers, and install a third.

The first server I moved was a small experimental VM used for testing various network, web and other items. I like to have dedicated testing environment for every operating system that I professionally run. This server was responsible for my personal Teredo tunneling, and was the one I put my CGI testing on from awhile a go. I could have easily moved it, but I wanted see how the export/import from ESXi to vmForge worked. I stopped the machine on my ESXi server, downloaded it as a OVF and uploaded it, via my Windows machine, to my catalog. It imported it as a template. I then deployed the template and deleted the server. It worked flawlessly! All I had to do renumber the machine and I was done.

The next server was a little more complicated. It was originally a CounterStrike:Source server that I had converted into a Apache Tomcat JSP host. Because it already had a working Java setup, I added an OpenFire Jabber server, and a LogicMonitor agent to it. This gave me the ability to monitor my internal network from LogicMonitor, a monitoring solution that we’re looking into. The triple Java duties of this machine, unfortunately, put a big crunch on its RAM, so that took a lot of tweaking on the application level to get them to play nicer with each other.

The next server was a monitoring server that I had set up running Zabbix. I had previously gotten Nagios working on it, but it was too burdensome for me to maintain. I also liked having graphing and service level alerting as well as agent based checks, both active and passive. The biggest problem with Zabbix was getting it initially set up to send alerts, so it was nice to be able to import this machine, that had a working base, than to start from scratch. LogicMonitors does pretty much everything that Zabbix does, and better, but why not have two monitoring solutions? I also set up that machine to be a centralized logging server if I ever want to install a log analyzer like Splunk. I set it to copy the logs to a MySQL database, and to run php-logcon, but that didn’t scale past a few thousand entries.

Next was installing a FreeBSD server to act as a centralized tool, mail environment, and storage space for myself and my friends. I love FreeBSD, the only reason I set up my other servers as Linux boxes was pure laziness on my part, which I’ll pay for later in administration time. Also, they are mostly single purpose appliances, and it’s nice to have some of the Debian style scripting for web built-in. I try to stay fairly OS agnostic, but I do have preferences.

Since my shell server would have the most exposure to the internet, so I wanted a relatively secure system. Also, I would be spending most of my time in that server, so I decided to go with the OS I love. That would also bring things full circle, as my pfSense box and Shell server are both FreeBSD.

I decided on installing FreeBSD 8.2 stable. I sliced my disks like this:

/           512MB
swap        1GB (1x Memory)
/usr        5GB
/var        10GB (Modest space for DB and info)
/home       140GB (An egregious space for storing files)

I installed the OS and ports, and I switched from cvsup to csup awhile ago, and updated my ports-supfile and stable-supfiles to point to a local(ish) mirror, and checked out /usr/src and /usr/ports. I then updated my kernel config (Tip: compile without debugging if you want it to fit in 512MB ) reinstalled, and rebooted. Voila! A new FreeBSD system. I’ll probably go into doing a comprehensive FreeBSD install in a later post.

I installed Postfix and Dovecot2 for local mail, Apache 2 for user directories, and migrated my users information, passwords, and home directories from my old server. Everything went surprisingly smooth. I installed Mutt for myself, Alpine for one of my users, and a few other pieces of software, and I had a fully running shell server. I was going to run PowerDNS and PowerAdmin on one of my Linux boxes, but I decided to stick with BIND on the FreeBSD server, as it was more efficient for me to edit text files than use a web interface. Weird, I know. Now that my shell server was done, and everything was migrated, I could turn off my old FreeBSD box. I admit that I did feel a little bad as I typed halt into its shell for the last time. It served me well over the last four years.

Now my infrastructure migration was complete, running fully virtualized, lowering my power consumption, gaining redundancy, and boosting performance for the fraction of the cost of having physical infrastructure.

I Win!

Game Over.

Last week I discussed moving my personal infrastructure into an vmForge Virtual Data Center. I discussed setting up a pfSense firewall, and getting things ready for my various projects. The first one that I wanted to tackle was setting up a load balanced webcluster.

Backing up a bit, three years ago, I was a support tech, looking for ways to grow my knowledge base and expand my abilities. After a bit of haranguing from my boss, he suggested setting up both a web and mail cluster with the various bits of hardware lying around. The project was laid out very simply, with two machines acting as frontends, my old server acting as the share storage point, and another acting as a database server. Load balancing would be done via round-robin DNS resolution (ie multiple A records with the same name and different IPs) I mapped it out, and diagrammed, but I was not ready to implement it at the time. I wanted to know more about how Apache worked on a single server before tackling it on a cluster.

Fast forward a year and a half, and I decided to revisit the project, partly due to wanting to progress out of my support position, and partly due to the fact that I now understood Apache and MySQL a lot better. I had a VMware ESXi system, and had tried a few times to create a cluster with various amounts of success.

At the same time, I decided to try redoing how I set up PHP. I switched to a FCGID based Apache setup. Qualitatively, it felt faster in my testing, and it allowed me to tweak individual sites for various levels of support and resource utilization.

The final piece was the VDC itself. With a large NAS behind it, lots of spinning disks, and logical vApp networking, I could lay things out simply and easily the way I wanted infrastructure wise, without relying on public IP space, errant patch cables, or a stubborn old server.

The first thing I had to decide was what my layout would look like, and how big the VMs would be. I settled on two frontend machines running minimal disk for the operating system and potential disk based content caching, and a single backend machine that would be both the shared filesystem and database host. I considered installing MySQL slaves on the frontend, with the NFS/database server as the master, but none of my sites justified that amount of complexity.

The frontend machines were configured with 500 MiB of RAM and 8 GiB of disk space per. The NFS/database server was configured with 2 GiB of RAM and 128 GiB of storage.

After figuring out the layout, I had to pick what operating system to use. After some hemming and hawing, I decided on Ubuntu 10.04 LTS instead of my beloved FreeBSD. There were a couple of reasons for this. One, I like the Debian style Apache tools. They make scripting site management much much easier. Two, there were potentially a lot of Perl and Pear packages that would need to be installed, and I prefer to handle that via apt/dpkg rather than ports. Third, and formost, I had already created an auto installing ISO, and it made installing exactly what I wanted multiple times very easy.

After installing the NFS/database server, I added MySQL and the newest nfs-kernel-server for NFS4. Why NFS4? Why not?

I laid out my filesystem to export under /var (with sensible names like /var/apache-site-config /var/www, etc) and mounted them over to /export (with accessible names like www, conf.d, sites-available)

I then installed NIS and created the various users for SUEXEC and FTP. I had some trouble with idmapd, but I figured it out (RTFM, and a couple known issues about it not starting sometimes)

After getting the NFS/database server squared away, I set up Apache on the frontend machines, mounted the NFS directories in their spots, and fired it up. It worked! I then moved my personal blog over, and decided to do some tests. Like, serious load tests.

It was abysmal.

My blog made extensive use of memory caching (via mem_cache, not memcached) The combination of memory caching, FCGID and 500 MiB of RAM was too much for the frontends. I upped them to 1 GiB per, and things were perfect. Just a little more RAM made a huge difference! I did add memcached a little later, as the WordPress plugin “W3 Supercache”, and Drupal can use it, as well as other CMS/web applications.

Now, with a working model, I went back to pfSense, and set up my load balancer. pfSense uses something called “relayd” to load balance, and that’s basically all it does, it relays the TCP connection from a virtual server, to any number of actually IPs. There’s a setting elsewhere in pfSense to have it monitor states, and send any new requests from a source with an open state to the same IP, give you some persistence (or “stickiness” as they call it); a step above round robin DNS. Unfortunately, the virtual servers only answer for one port, so I had to added two: one for 80(HTTP) and one for 443(HTTPs) and they cannot cross IP protocols, so I had to add one for IPV4 and one for IPV6. So four total virtual servers, each with a separate pool of servers. This was verbose. The other major caveat is that the virtual servers only listen on the WAN interface. After some trial and error, the easiest thing to do was to set frontend 1 to also have the same IP on the LAN interface. That way, internally, machines have access to sites on the webcluster.

A representation of my Virtual Webcluster The dotted lines represent the RFC 1918 IP addresses. (click to enlarge)

After getting that setup, I did some more load testing, to see how the load balancing worked.

It was abysmal.

I didn’t have enough frontends for the amount of load that I wanted to support. However, I didn’t have enough addresses to add more machines.

I did some experimentation, and discovered that relayd would relay connections to a RFC1918 address as well as a world routable one. So, I cloned my second frontend and made two more, with 172.16.0.0 addresses. It worked flawlessly, go go virtualization. IPV6 addresses, of course, were not a problem.

After adding the two frontends, things performed better, but still, the load metrics were spiky and performance was not quite where I wanted it.

I looked into things, and discovered that my FCGID threads spawned by Apache were then forking when they got too busy. After a certain amount of spontaneous forks, the frontend tapped out until other threads died. This brought the performance of the cluster way down.

Remembering Mike’s mantra fork/exec is expensive, and bad, and should be avoided, I did some digging, and I found out that the default Apache in Ubuntu, MPM-Prefork, was designed to contain all spawned processes in order to sandbox non-threadsafe modules. This allows for extra stability and security at the expense of performance. So I switched the cluster to Apache-MPM-Worker and, voila! Smooth, predictable performance. It was much, much better.

I then started porting my users’ sites over, along with their databases. This involved quite a bit of config file parsing, and database dumping. mysqldump –database(s) came in handy here, as did setting up preview aliases via wildcard A records. It took about a week to move everything.

I then set up some scripts, sudo, and ssh private keys so that I could run scripts on the NFS/database server that would propagate commands to the frontends, and set up a couple of cron jobs for basic maintenance. I set up vsftpd for my users to upload and manage their content as well, and changed DNS for each of their sites.

Now I had a reasonably fast webcluster running. It seems to work well… I’m sure our webmaster will make fun of me, err, have something to say about my design and implementation choices though.

Now it was time to tackle one of my least favorite things. Email. Finicky, finicky, email.

Next week: The Mail Cluster.

I had a problem. All of my personal infrastructure was on an aging server, cobbled together from various parts that were laying around. I had already replaced the motherboard once, and I was not looking forward to doing more maintenance. The system had 5 320gb SATA disks in a RAID 5 setup. Not very fast, and it could only survive one disk failure.

Software-wise the machine had long exceeded what it was designed to do. It was originally designed as a game server, with some web and email. I had added several other services to it as I learned and played. Data was spilling out of its assigned slices. Symlinks were used strategically but it was still a mess.

It was time to replace the server, and ipHouse’s new vmForge VDC was the ideal place to do so. I bartered with my boss by offering to get rid of my power hungry hardware in exchange for a modest virtual data center. He accepted. I was allowed to provision a VDC with 16 GiB of RAM, 8GHz of CPU, and 500 GiB of storage and a /28 worth of IP addresses.

I also had another server, with ESXi running on it, with a few VMs used for monitoring and FCGID based websites, and another FreeBSD server that had a few test items on it that I wasn’t using much. It was time to reduce and consolidate my infrastructure.

The first thing to decide was the network layout. Unfortunately, a Virtual Domain (VDOM) off of one of our Fortinet Firewalls was out of the question as they are too valuable to comp to an employee. So I was left with a vShield Edge device or running my own firewall as a VM.

Running my own VM won out as I wanted to do some rudimentary load balancing (something beyond round-robin DNS) and the vShield couldn’t do it (this feature isn’t exposed through VMware’s vCloud Director in version 1.5). I had previously futzed with pfSense, and was comfortable with using it. Unfortunately, this choice would cut into my available resources.

In order to facilitate my firewall, I asked our network guy to route my /28 to a /30. I also asked him to route me an IPv6 /64. I then set up my VDC (I provision them as part of my job) with two networks, an external one with the /30 on it, and an internal one with a bogus RFC 1918 network, set to DHCP so that it wouldn’t assign those addresses.

After the VDC was set up I booted my Windows XP VM in VMware Fusion, and uploaded the pfSense ISO to my VDC’s private catalog. Then I built a networking vApp, added a VM, “inserted” the ISO, booted it up, and installed pfSense via the console. Very easy, very quick.

After installing pfSense I was disappointed to find out that the current 2.0 release barely supports IPv6. I’m a big IPv6 fan, so I bit the bullet, and moved up to 2.1beta code as this would allow me to run IPv6 natively in all (well, most of) the services.

I installed additional few packages: Open-VM-Tools, OpenVPN Client Export, pfBlocker, snort, and mailreport. Using Snort as and IDS required some configuration and white-lists to avoid blocking my own networks, the rest was very straightforward.

After installing and configuring pfSense, I shut down the VM, and added it, as a template, to my private catalog. That way, if in the future goofed it up, I could quickly deploy a replacement. This took a bite out of my storage limit but the VM is very small and is well worth the price.

Now that I had the firewall straightened out I could figure out what I wanted to do.

I decided that I wanted to clusterize my web and mail services, have a shell server for myself and my friends who still wanted access to a UNIX(like) server, and have a couple of servers for miscellaneous applications. Lastly, I wanted one set up as a rSyslog/monitoring server.

I had a lot of work to do.

Next week: “The Webcluster”

What is IPv6 you ask? Well, that topic won’t be discussed in this posting as it has been discussed all over the Internet already. You can test your readiness by going to http://test-ipv6.com/ and checking your results.

This post is about what services are running IPv6 dual-stack on the ipHouse network today.

Many of our services have been operating in a dual-stack IPv4/IPv6 configuration since the middle of last year but since most connectivity is still only IPv4, most people would never see nor notice the IPv6 network capabilities we have already built into our network. Our network itself has been IPv4/IPv6 dual stack for what seems like forever with quite a few customers connected with native IPv6 connectivity via DSL, T1, metro-ethernet, and colocation.

Services at ipHouse that are IPv6 capable and enabled:

Web-hosting on our UNIX cluster is fully IPv6 capable and enabled, though only ipHouse sites currently have IPv6 addresses. If you are a web-hosting customer of ipHouse and would like to have your site enabled for IPv6, please email webmaster@iphouse.net with your request so we can make the necessary configuration changes in our load balancers..

Email inbound (greylisting servers), outbound (SMTP, POP, IMAP) are all IPv6 capable and enabled.

ipMom and webmail.iphouse.com are fully IPv6 capable and enabled.

DNS servers (caching resolvers and authoritative) are IPv6 capable and enabled.

T1, DS3, metro-ethernet, and colocation connectivity is all fully IPv6 capable and enabled (if you would like to request IPv6 connectivity, please send email to noc@iphouse.net with your request so we can work with you).

DSL service is IPv6 capable and enabled, though customer premise equipment gear is very lacking in IPv6 capabilities. See our previous blog post regarding this issue.

ipshell.com services are IPv6 capable and enabled.

MICE exchange connectivity is IPv6 capable and enabled – we are a full routing peer.

This blog site is IPv6 capable and enabled.

Our conference rooms and guest WIFI networks are IPv6 capable and enabled.

noc.iphouse.com is IPv6 capable and enabled.

Services at ipHouse that are IPv6 capable but not enabled, many of these will be enabled soon and this post will be updated to reflect the changes:

news.iphouse.com is IPv6 capable but not enabled.

lists.iphouse.net is IPv6 capable but not enabled.

stats.iphouse.com is IPv6 capable but not enabled.

Dial-up services are IPv6 capable though I don’t think we will be enabling it unless by customer request.

Microsoft IIS hosted websites are IPv6 capable but not enabled.

Services at ipHouse that are not IPv6 capable:

The Mailfoundry anti-spam systems, neither the web interface nor the SMTP services.

Our offsite DNS authoritative servers are not IPv6 capable at this time

I’m sure I missed a few things and I’ll update this as I find them (or remember, as the case may be).

The first thing to understand is the Registrar. A registrar is a company that is accredited, and allowed to work with ICANN ( The Internet Corporation for Names and Numbers). ICANN actually maintains domain names and their information. You pay a registrar for your domain registration, and they, in turn, pay ICANN and provide them with the domain’s information. There are other parties and services involved, but in the interest of keeping it simple, that’s how it works.

Once the registrar has the domain name registered and reserved, actual Domain Name Servers need to be assigned to it. DNS is the service that turns the domain name, say, example.com, into its associated IP address. Humans can remember words far better than numbers. And computers deal with numbers.  This is why we have DNS.

It also tells servers where to send email and what IP addresses various services may use. Each service normally has an A (or address) record. For example, an ftp server, ftp.example.com, may be on one IP address while the website, let’s say www.example.com, may be on another. Mail routing is controlled through the MX (mail exchange) record, which must point to a host name, like mail.example.com instead of an IP address. Canonical Name records and TXT record are used for more specialized purposes.

Usually, a registrar defaults to using their in-house nameservers. You can use these servers, putting in the information from your hosting company, usually by using a web based interface. However, we feel it is a better idea to switch the nameserver to your hosting company’s. That way, if your hosting company makes a change then they can update your DNS automatically alleviating your need to worry about these technical details.

DNS can be tricky and it is absolutely critical that your information is correct. One small mistake can cause you to not receive email, or not be able to view your own website. Many outages are associated with DNS problems and can easily be avoided by making sure the right information is in the right place.

Now, there are PTR (or rDNS) records that often are confusing. Most DNS queries are like looking up a name in a phone book directory, but you look up a server name to get the IP address (instead of a phone number). There are a few instances where you want to check the IP address and see who it belongs to, kind of like a reverse directory search of a phone number to see who’s calling. This is called a reverse DNS check.

This most often come into play when mail servers want to check the legitimacy of the SMTP servers that are sending them messages. If they query the IP address and get mail.example.com (or some other, similar Fully Qualified Domain Name) they let it pass. If they get something like 192-168-123-45-adsl-dynamic-customer.isp.net, they may reject or quarantine messages. These records are maintained by the ISP who provides the IP address, so if you run a server out of colocation space or off your Internet connection, you’ll want to contact the ISP to update that information.

Well, this was a very basic summary, hopefully it helps!