ipHouse Blog » Hosting

Speeding up CSS

Doug Rau — Tue, 31 Jan 2012 16:14:16 +0000

As web pages become richer and more complex, they can also become burdened with numerous CSS files, background images, icons, and scripts which put additional load on your network and degrade the user experience of your website. Fortunately, there are a few things you can do to improve things.

First, its become common practice for many pages to have at least three stylesheets; one for all media, one for screen-specific instructions, and another for print. Other stylesheets might be imported for specific pages or resources, such as a lightbox library or HTML form suite. Instead of putting each stylesheet existing in a separate file which must be loaded and parsed separately, consider consolidating them as much as possible by using @media directives within the CSS file. For example,

// - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - // combined.css // these styles apply everywhere, all the time
body { color: black; }
h1 { font-size: 150%; font-weight: bold; }
// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -
// these styles only apply on-screen
@media screen {
body { background: url(‘background.jpg’); }
div.nav a:hover { font-weight: bold; }
}
// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -
// these styles only apply in-print
@media print {
div.nav { display: none; }
p { text-align: justify; }
}
// – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -

Placing all your stylesheets in the section of your web page is best practice, and also improves apparent performance since the web browser can load and parse the style information before it begins rendering the page body. Associating stylesheets via tags instead of @import is better, since not all browsers handle @import directives promptly.

Also consider using Expires headers to instruct web clients to cache CSS files for a long time. In apache, this is easily achieved through the mod_expires instructions,

ExpiresActive On ExpiresByType text/css "access plus 1 year"

If you need to retain the ability to change your CSS files more frequently, add a version number to the filename and update it as necessary. For example, combined-1.0.css

The Value and Cost of Persistent Data

Nick Gasper — Fri, 27 Jan 2012 18:33:27 +0000

I’ve been cleaning out my house recently. There’s a lot of crud that’s just been lying around, collected through years. My wife describes me as a level 2 hoarder; she says that I would be a shoe-in for that A&E show. Going through many, many boxes that I’ve collected in the basement, I pick through each cord and think “I might need that.” I won’t need it though, so with a small mental push, I put it in the trash bag. Persistent data is a lot like that. A lot of companies have, either through policy or inertia, tons of useless information sitting on disks, or tapes, or CDs, that may be useful one day, but probably will not ever be.

I look at many cloud providers and I see the opposite. Their services were designed for expedience instead of permanence. They make it hard and, at times, very expensive to actually keep data around. Usually you have to attach a “disk” (or “volume”) to any machine that has data you want to keep and you have to pay for that privilege. You also better have backups because you have no idea about the underlying storage or data retention policies.

Any data that you absolutely need could mean you’re paying two or three times what you’d expect in order to keep it.

To my hoarder eyes the cloud is one big data furnace. It’s a dangerous place for your information to stay.

Enterprise data storage is expensive. I’ve often joked that virtualization is a scheme to sell storage arrays. It’s a tricky game of performance, space, and redundancy. Disks fail, flash is expensive, you never have enough RAM or CPU. There are dozens of types of arrays for hundreds of applications, retention policies, regulations; it’s a mess! When you have a service that has hundreds of thousands of customers then it may make sense that you discourage persistent data. You want people to consume your resources, pay their bill, and move on. Expedience instead of permanence. I’ve often been asked: Why online storage is so expensive when hard drives are so cheap? Well, this is why.

We built the ipHouse vmForge product with the idea that a virtual data center (VDC) replaces co-located infrastructure. The storage is persistent from the get-go. Is it any wonder that Mike has been loath to call it a ‘cloud service’?

This means that there are severe implications for any storage array that we put in place. We have to make sure that anything we put in place not only performs well but also goes the distance. It’s still a very good idea to do backups, though they probably will not be nearly as large, as most customers just need to back up a few key files or the database dumps that happen regularly. (you are backing up your database, right?)

Well, that’s my opinion anyways. Now I’m going to go back home and work on my basement.

Clone-tastic!

Nick Gasper — Fri, 20 Jan 2012 21:51:17 +0000

There are many things about virtualization is the ability to clone virtual machines. It’s really cool! Unfortunately, after you work with virtualization for a while you start to take it for granted. I can’t tell you how many times I roll out a new physical machine and sigh because I can’t simply clone it. Well, I can but that’s a discussion for another day.
Virtual machines are a set of files that are interpreted by a hypervisor. Since they are just files they can then be copied and/or edited. That’s all cloning is, the system is just copying the VMDKs (the “hard drive” files) and editing the VMX file (the config file to change things like the MAC address of a NIC and the virtual machine’s name).

You can even do it by hand if you have access to the backend storage. Mike once one-upped me by piping the VMX through sed. That’s cheating but all’s fair I guess. Cheater.

The vmForge VDC allows you to clone vApps and the individual machines contained therein. It automatically edits the config, can handle numbering the machine, and makes everything nice and easy. This is a killer feature in my book.

A lot of cloud providers are instance based. You select the operating system, push it out, and rely on automated services to configure them for you. Most of the time, you don’t get persistent storage. If you do, it’s usually a volume you attach to the instance and has nothing to do with its operating system. By using a vmForge VDC you can do the opposite. You can create a machine, configure it how you like, and then clone it. Configure once, and be done. Then you can keep a copy of it in your catalog for later deployments. Each clone is exactly that: a complete copy of your original system.

You may think that’s really cool! But wait, there’s more! (sorry, couldn’t resist)

When you build virtual machines in your VDC you are building them in vApps. A vApp is a logical container that holds virtual machines, internal networks, and can do things like set boot/shutdown order and power-down semantics.

When creating a vApp you also have the option to “fence” it. Fencing isolates the layer-2 networks within the vApp from any outside network. This means you can have internally consistent ip addressing inside the vApp. You can then “template” the vApp by moving it to your catalog and deploy it over and over and over again. That means that your preconfigured, multi-server application can be redeployed with a few mouse clicks!

Ultimately, cloning is about saving time. You get to use conventional tools to set up and multiple machines quickly and easily. You don’t have to learn any arcane scripting language, nor trust and maintain a complicated configuration service like Chef or Puppet. You just set up servers, push them out, and start to use them.

So, clone away!

Building things: cubicle analogy time

Doug Rau — Wed, 04 Jan 2012 22:17:34 +0000

This week, I’ve been building (actually, rebuilding) cubicle desks in our new office space and comparisons were naturally drawn to building virtual machines a few weeks ago.

Building a cubicle, no matter what the instructions say, is not exactly trivial. Even after putting the walls together, trying to get them all even and lined up is a trick. Then putting up the desk supports and the desk surfaces themselves, which are usually massive slabs of particle board and plastic veneer. If they’re fastened with screws, it’s always from beneath, out of the light, where you’re certain to get wood dust in your face.

In stark contrast, using vCloud Director to build a virtual machine was easy. Login, select a hardware and OS template, optionally configure, and deploy. It was up and running in a few minutes, maybe a little longer if you have added software or network configuration requirements. It’s all done with a keyboard and mouse, and I’m pretty certain I’ve never gotten wood dust in my face from it.

In both cases, a little planning goes a long way. Making a map of your cubicle layout lets you know whether everything is going to fit, what you’ll need, and whether you have it. Updating your map when you change your mind is also important. Making a map of your systems and network is similarly illuminating. How much CPU, memory, and disk do you need? What systems need outside IP addresses?

Also, paying attention to the details pays off. It’s a lot easier to make the desks level if all the supports are attached at the same height. Double check your IP address assignments and DNS records.

Finally, building a virtual machine doesn’t involve any power tools. Well, let’s hope not.

Kickstart your Linux install

Nick Gasper — Fri, 30 Dec 2011 19:54:20 +0000

I’ll admit it, I’m not a huge fan of Red Hat Enterprise Linux. I’ll administer it, I’ve worked with it. It’s a good distribution. I just have a bad taste for RPM based distributions based on my first forays into Linux back in my Mandrake days. I also first started to professionally work with Linux during the last couple of years of RHEL 5, when things were getting long in the tooth. Red Hat’s release schedule also conflicts with what most of my users want and expect; it’s far more suited to an corporate environment where having the latest features is not nearly as important as having consistent software versions. That being said, Red Hat has some fantastic tools; Anaconda and Kickstart being my favorite. So I was overjoyed when I discovered Ubuntu had support for Kickstart files! The Ubuntu installer can take Debian style preseed directives but in my opinion is overly complicated.

A Kickstart file basically answers the questions that pop up in the installer as the installer goes removing the need for human interaction. If an question isn’t answered, the installer pops up with the proper dialog, takes user input, and continues. I can pick and choose what information I want to populate automatically and which information dialogs I want the customer to answer. In my auto install ISOs I prompt the customer for a username and password as I want the users to enter that information.

When I was tasked with making an auto installing ISO for our customers I was able to create one quickly by using a kickstart file.

The process of making a CD is a bit verbose, and better handled by some of the how-tos out there.

But I’ll take your through my Kickstart file.

First are some of basic information about the system. These are fairly self-explanatory.

platform=AMD64
#System language
lang en_US
#Language modules to install
langsupport en_US
#System keyboard
keyboard us
#System mouse
mouse none
#System timezone
timezone America/Chicago

I disable root, to reflect the Ubuntu default. You can enable it by removing the next line, and setting it with the second.

rootpw --disabled
#rootpw jpDhuZtql4of4rfq

I do not automatically add a user, but you can with the next line.

#user johndoe --fullname "John Doe" --password changeme

I don’t think this does much in an Ubuntu Server install but I put it in anyways.

#Use text mode install
text

We’re installing not upgrading.

#Install OS instead of upgrade
install

Use the CD-ROM.

#Use CDROM installation media
cdrom

Where are we going to put the bootloader?

#System bootloader configuration
bootloader --location=mbr

Get rid of any existing partitions.

#Partition clearing information
clearpart --all --initlabel

Partition the disks using Ubuntu defaults (512MB swap, etc) This allows the ISO to work on whatever size disk you want. Linux isn’t great about using swap anyways, so 512 is plenty.

#Disk partitioning information
part /boot --fstype ext3 --size=200 --ondisk=hda
part swap --recommended
part / --fstype ext4 --size 1 --grow

Passwd information. I know… MD5… You can use something more secure if you wish.

#System authorization infomation
auth  --useshadow  --enablemd5

We need DHCP for some of the following steps, as I have no idea what type of network this will be run on. You can specify other info here if you want.

#Network information
network --bootproto=dhcp --device=eth0

My customers hate having UFW on. I don’t think this actually works yet in Ubuntu, so I also do it in a later script.

#Firewall configuration
firewall --disabled

X-Windows on a Server? No thanks.

#Do not configure the X Window System
skipx

And finally, we want to reboot after installing. This isn’t actually done, as we’re going to run a post-install script.

#Reboot after installation
reboot

Add additional packages to install. I install the fewest here, as I update in a later script, so why install a bunch of stuff only to update it later?

%packages
@dns-server
@openssh-server
gcc
build-essential

Here comes a a post install script.

%post

Mount the CD again, as there’s data we want off of the CD.

echo Making CD Mountpoint
mkdir -p /mnt/cdrom
echo Mounting CD
mount -t iso9660 /dev/sr0 /mnt/cdrom

Copy over a script that I’ve written that does updates and additional installs when the virtual machine is first booted.

echo Copying Firstboot Script
cp /mnt/cdrom/firstboot /etc/init.d/
chmod +x /etc/init.d/firstboot

Updated the init structure to run the firstboot script on boot.

update-rc.d firstboot defaults
echo Adding new Crontab

Add a custom crontab with some randomized sleep values.

cp /mnt/cdrom/crontab-template /etc/crontab

A script that I wrote that edits resolv.conf to point to the local bind server

echo Copying resolvfix init script
cp /mnt/cdrom/resolvfix /etc/init.d/
chmod +x /etc/init.d/resolvfix
update-rc.d resolvfix start 99 2 3 4 5 .

An updated sources.list with a closer mirror.

echo Copying Apt Sources
cp /mnt/cdrom/geeks-org-sources.list /etc/apt/sources.list

A new dhclient with the local bind server seeded.

echo Copying dhclient.conf
cp /mnt/cdrom/dhclient.conf /etc/dhcp3/

A new named.conf.options with some useful defaults.

echo Copying named.conf.options
cp /mnt/cdrom/named.conf.options /etc/bind/

Moving over vmware-tools for installation upon first boot.

mkdir /vmware
cd /vmware
echo Extracting Tools
tar zxf /mnt/cdrom/VMwareTools-*.tar.gz

Ejecting the CD.

echo Unmounting CD
umount /mnt/cdrom

Update the system.

echo Updating
apt-get update
apt-get -y dist-upgrade

And finally, reboot the system (sync for good luck ;) ).

echo Rebooting
sync
reboot

Now, as I mentioned before, there’s a firstboot script that I run that does quite a bit of work before the machine is finished. It does things like wipe out the SSH keys, install VMware Tools, remove and purge old kernels and install applications like MySQL, Apache, as required.

Well, that’s one of the tricks I have tucked up my sleeve, I hope it helps!

Setting up a LogicMonitor Agent

Doug Rau — Fri, 30 Dec 2011 19:38:21 +0000

LogicMonitor is a really cool server and network monitoring and measurement system which we’ve been working with. It uses a lightweight monitoring agent installed on your local network which collects data from your systems and passes it over SSL to an external aggregator. It’s capable of auto-discovery and is mostly self-configuring though you can adjust many of the metrics. After many years of working with patchwork monitoring and alert systems we’re pretty excited about it. Call us if you’re interested.

Setting up a monitoring agent on your local network is easy. The server hosting the agent just needs a JRE (Java Runtime Environment) installed using version 1.6 or greater and must be able to make an outgoing SSL connection. To monitor Windows systems, you’ll need to install the agent on a Windows server.

Login to the LogicMonitor website, click on the “Settings” tab, then on “Agents” in the left navigation, then on the “Add” button. Click past the introduction, and indicate whether you’ll be installing the LogicMonitor agent on a Windows or Linux server. Download the agent installer, or copy the link and use wget to download the installer directly to your Linux system. Run the installer to install the agent on your server then return to your web browser and click “Next” to verify that its been installed correctly and is able to communicate with the LogicMonitor system.

To begin monitoring a host on your network, click on the “Hosts” tab, then on the “Add Hosts” button and select “New Host (wizard)”. Enter the host name or IP address. Note that if your monitoring agent and host are on a private internal network then this should be the IP address visible to your agent. Select your monitoring agent (if you have more than 1), and LogicMonitor will go ahead and verify that its able to gather information about the host.

NOTE: at this time, LogicMonitor does not support IPv6

CloudStack + vSphere: the marriage

Mike Horwath — Thu, 15 Dec 2011 17:40:24 +0000

This was for a customer of ours that wanted to move from a pure VMware vSphere environment to a CloudStack managed environment with vSphere as the hypervisor.

I don’t think anyone would disagree that the documentation for CloudStack still needs work. But all the documentation in the world can’t help if you decide to skip important sections.

Some caveats notes while I was working with 2.2.12 and 2.2.13:

vSphere 4.1 only, vSphere 5 is not compatible
Regardless of vSphere licensing used, vSphere vDS is not supported
vSphere DRS must be turned off so that CloudStack can do its own load balancing (it seems angry if things move)
No mention of vSphere HA but I enabled HA because the point of using vSphere as the hypervisor was for automated high-availability
Discussion of vSphere SDRS going on in the forums but so far this also seems like a toggle that needs to be disabled for now
Secondary storage needs to be accessible by the Secondary Storage VM, vSphere hypervisors, and the CloudStack management system

At first I didn’t read the documentation all the way through – just the networking bits as that was the largest area of the document and initially my largest area of concern. The lack of vDS is a bummer but I’m going to guess that they’ll add this functionality in a later release. I believe vDS would make a few things easier on the users of CloudStack long-term and look forward to this addition.

Since I didn’t read the documentation completely I failed to notice that it states support for vSphere 4.1 and mentions nothing about vSphere 5. Even after I started digging further into the document I assumed that vSphere 5 would be supported. That’s a bad assumption and after a reinstall of the physical servers (and vCenter) I was able to get the initial configuration to pass. Strike one for me – I should know better than to skip portions of an install document.

Next was getting all the networking together with the trunking for VMs, storage network, interconnection between CloudStack and the physical servers. This wasn’t difficult but did need me to actually write everything down as there was a lot going on concurrently. See my last note above for the hint to my biggest hurdle I needed to overcome.

I have some worries about what happens when a physical server fails (and everything fails at some point). I don’t know if CloudStack will initially notice the failure before vSphere ends up bringing the VM back online on a different physical box. And once it is up I don’t know how CloudStack will respond or alert for such items. I bring this up because CloudStack has its own HA system in place but the configuration in the general settings area shows 1800 seconds to trigger (if I am even reading that right). vSphere HA is faster than that. I guess we’ll see what happens if/when it does occur.

And the biggest hurdle for me was the requirement that the secondary storage VM, CloudStack management system, and the hypervisors need to be on the same network for the secondary storage to work. This storage is for holding templates and ISO images for later deployment into a customers’ cloud environment. Once this items a pointed out to me then everything started to go together. This was an area of frustration as it wasn’t clear to me in the documentation that this is the case. The examples flow around an idea of one network for management, VM traffic, and storage. That is not how things are designed in my world as I separate out each of those types of traffic.

I found CloudStack simple to install and once I knew the different restrictions/notes above the process only took 10 minutes to complete.

If you want to read documentation in a browser, I found the cloud.mindtouch.us site to be very helpful. Of course CloudStack has PDF based documentation as well though multi-file searching is either great or abysmal depending on your workstation OS of choice so I opted for the web version to help narrow things down before going back to the PDF editions.

Our customer is still working out a few kinks and should be fully operational soon.

Logic Monitor through screen shots

Genevieve Ruebel — Mon, 12 Dec 2011 17:40:53 +0000

This is the part where I talk about LogicMonitor as a part of our ipHouse internal learning enrichment task. I am going to do this through screen shots because it works for me and I hope you learn a little along the way.

I was in great need of Nick’s help because at first I thought I could create just any host name. So of course I chose the name barf. Well you can’t do that. You need to use a machine name that already exists. Nick said I should choose smtpgrey-2.iphouse.net or smtpgrey-1.iphouse.net (inbound SMTP border servers in use on our network).

Once I figured that out it was all smooth sailing!

Below is the Logic Monitor interface.

If you look to the left, my name is Gen.

So you say you would like to add a host. Click on the ‘add a host’ button on the left side. Then this screen pops up.

You have to pick a host name that already exists. Now you select an monitoring agent. I chose worldgen. You do need to choose one that will Windows based if it is a Windows machine so you can use WMI if your firewall allows such. Remember that.

Once you pick your agent, the wizard will check to see if every choice you made works out. Apparently, I did things correctly.

I decided that I do not want to add another host. I think I am good for now.

Okay, okay…I am sure I want to exit.

So now, this is what Logic Monitor really does. It monitors.

You can toggle around and see what has gone on with your CPU usage and gather load averages plus a whole slew of other statistics.

Now it is monitoring Disk. Watch it monitor.

DNS status seems to be neutral.

Okay this is the part that I think is the most interesting about Logic Monitor, the alerts. I know that no one wants to see a critical message. It sure is fun when you are testing new software though. I did not have any alerts so I moved on to show you someone elses alerts.

Here we go, this is critical as you can see from the bright orange and yellow colors.

There is a section under notes that you can click on and add in a note on what is going on or how you are going to fix it or if you did.

After all of this, I think Logic Monitor seems quite useful. It is nice that when an alert happens (because they do) a page can be sent to your cell phone.

SysAdmin Golf: The Hard Way

Doug Rau — Fri, 09 Dec 2011 19:04:01 +0000

This is the hard way to p2v a Linux server into a vmForge VDC. You’ll need the VMware Converter bootable CD, and somewhere to store your disk image. If you have a Windows server and shared disk on your local network, that may be faster than a USB hard drive, which is what I used. This is a cold clone procedure, which means that your server is down while its being copied. And it may be down for a while, depending on how much disk your server has, how quickly it can be cloned to local storage, and finally how quickly it can be uploaded to your VDC.

Reboot your server from the VMware Converter CD. Once VMware Converter is running, click on “Import Machine”. Click through the first couple of panels. On the “Source Data” panel, select “Import all disks and maintain size”. On the “Destination Type” panel, set the destination type to “Other VMware Virtual Machine”. On the next panel, set the VM Name and the Location where you’re storing your disk image. Since Converter doesn’t support direct OVF export here, you’ll have to set the type of virtual machine to Workstation 6.x and change it later. Click through the rest of the panels, and Finish. VMware Converter will now export your server to a .vmx description file and .vmdk disk image(s). This may take a while.

After VMware Converter is done, you’ll need to use VMware’s OVF Tool to convert the .vmx file to a .ovf file.

ovftool.exe name.vmx name

Next, the tricky bit. You’ll need to edit the .ovf file, which is in XML format, and find and update the VirtualSystemType field to vmx-07, a reasonably current version. Once you’ve edited the .ovf file, you’ll also need to generate its SHA1 hash and update the .mf manifest file. Now, you should have something which can be imported into your vmForge VDC.

Login to your vmForge VDC (or any VMware vCloud Director system), and create a new catalog if you don’t already have one. In your catalog, click on the Upload button. Select your .ovf file, give it a name and description, and click on the Upload button. Again, this may take a while. Once its done, you’ll be able to use the template from your catalog to create a new virtual machine, a clone of your old physical server. The last step will be to power on the virtual server, login via the virtual console, and reconfigure networking.

comparing software load balancers

Doug Rau — Thu, 01 Dec 2011 19:42:19 +0000

now that i have three different software load balancers installed (Balance, Crossroads, and Pen), i want to evaluate their relative performance. benchmarking a single web server isn’t difficult using tools like ab, but trying to benchmark a load balanced cluster is somewhat different. since most load balancers support stickiness, all the requests from a single source will be directed to a single back-end server. thus, i’ll need to run the benchmarker from several different sources simultaneously, or i’m really just testing one server with something in the way. fortunately, i have three machines on different IP addresses sitting idle.

my first test is 10,000 requests for a static HTML page (2866 bytes). this test was run against a single apache server in the pool and against each of software load balancers with two back-end servers, from one source and from three simultaneously.

handler	single source	three sources
apache only	21.659 seconds	33.822
balance	106.794	failed
crossroads	37.729	failed
pen	39.112	failed

the single apache server actually performed the best, easily beating any of the software load balancers in raw throughput. the test from three sources is effectively a mild denial-of-service attack, and none of the software load balancers could handle it. each of them failed and stopped accepting connections well before 10,000 requests were completed.

the second test is 20 requests for a PHP script which performs exactly 1 second of mathematics then returns results. in this case, almost all the load is on the back-end servers, and there was negligible difference in results between the four front-ends.

the third test is 400 requests for the PHP script, but issuing 20 concurrent requests from each source at a time. this generates significant load on the back-end servers, but is the first test where having multiple back-ends shows any improvement.

handler	single source	three sources
apache only	23.493 seconds	31.097
balance	22.820	26.191
crossroads	34.199	40.355
pen	24.721	28.365

the fourth test is a monster, 400 requests for the PHP script, 100 concurrent requests from each source at a time.

handler	single source	three sources
apache only	14.912 seconds	22.604
balance	10.355	18.909
crossroads	failed	failed
pen	15.219	failed

these results suggest that a software load balancer might be an option for putting more capacity and resilience into a script-heavy website, but clearly shouldn’t be chosen for performance. the single apache server performed better than my small cluster in nearly every test, and much better in a few cases. the total meltdown of the software load balancers in difficult situations is of particular concern. the results for the cluster might improve with more back-end servers, but the software load balancer itself seems to be the bottleneck.

of the three software load balancers, the simpler Balance and Pen outperformed Crossroads in general. interestingly, Balance fared spectacularly poorly against heavy traffic in the first test, but very well against a different sort of heavy traffic in the fourth.