I had a problem. All of my personal infrastructure was on an aging server, cobbled together from various parts that were laying around. I had already replaced the motherboard once, and I was not looking forward to doing more maintenance. The system had 5 320gb SATA disks in a RAID 5 setup. Not very fast, and it could only survive one disk failure.

Software-wise the machine had long exceeded what it was designed to do. It was originally designed as a game server, with some web and email. I had added several other services to it as I learned and played. Data was spilling out of its assigned slices. Symlinks were used strategically but it was still a mess.

It was time to replace the server, and ipHouse’s new vmForge VDC was the ideal place to do so. I bartered with my boss by offering to get rid of my power hungry hardware in exchange for a modest virtual data center. He accepted. I was allowed to provision a VDC with 16 GiB of RAM, 8GHz of CPU, and 500 GiB of storage and a /28 worth of IP addresses.

I also had another server, with ESXi running on it, with a few VMs used for monitoring and FCGID based websites, and another FreeBSD server that had a few test items on it that I wasn’t using much. It was time to reduce and consolidate my infrastructure.

The first thing to decide was the network layout. Unfortunately, a Virtual Domain (VDOM) off of one of our Fortinet Firewalls was out of the question as they are too valuable to comp to an employee. So I was left with a vShield Edge device or running my own firewall as a VM.

Running my own VM won out as I wanted to do some rudimentary load balancing (something beyond round-robin DNS) and the vShield couldn’t do it (this feature isn’t exposed through VMware’s vCloud Director in version 1.5). I had previously futzed with pfSense, and was comfortable with using it. Unfortunately, this choice would cut into my available resources.

In order to facilitate my firewall, I asked our network guy to route my /28 to a /30. I also asked him to route me an IPv6 /64. I then set up my VDC (I provision them as part of my job) with two networks, an external one with the /30 on it, and an internal one with a bogus RFC 1918 network, set to DHCP so that it wouldn’t assign those addresses.

After the VDC was set up I booted my Windows XP VM in VMware Fusion, and uploaded the pfSense ISO to my VDC’s private catalog. Then I built a networking vApp, added a VM, “inserted” the ISO, booted it up, and installed pfSense via the console. Very easy, very quick.

After installing pfSense I was disappointed to find out that the current 2.0 release barely supports IPv6. I’m a big IPv6 fan, so I bit the bullet, and moved up to 2.1beta code as this would allow me to run IPv6 natively in all (well, most of) the services.

I installed additional few packages: Open-VM-Tools, OpenVPN Client Export, pfBlocker, snort, and mailreport. Using Snort as and IDS required some configuration and white-lists to avoid blocking my own networks, the rest was very straightforward.

After installing and configuring pfSense, I shut down the VM, and added it, as a template, to my private catalog. That way, if in the future goofed it up, I could quickly deploy a replacement. This took a bite out of my storage limit but the VM is very small and is well worth the price.

Now that I had the firewall straightened out I could figure out what I wanted to do.

I decided that I wanted to clusterize my web and mail services, have a shell server for myself and my friends who still wanted access to a UNIX(like) server, and have a couple of servers for miscellaneous applications. Lastly, I wanted one set up as a rSyslog/monitoring server.

I had a lot of work to do.

Next week: “The Webcluster”

What is IPv6 you ask? Well, that topic won’t be discussed in this posting as it has been discussed all over the Internet already. You can test your readiness by going to http://test-ipv6.com/ and checking your results.

This post is about what services are running IPv6 dual-stack on the ipHouse network today.

Many of our services have been operating in a dual-stack IPv4/IPv6 configuration since the middle of last year but since most connectivity is still only IPv4, most people would never see nor notice the IPv6 network capabilities we have already built into our network. Our network itself has been IPv4/IPv6 dual stack for what seems like forever with quite a few customers connected with native IPv6 connectivity via DSL, T1, metro-ethernet, and colocation.

Services at ipHouse that are IPv6 capable and enabled:

Web-hosting on our UNIX cluster is fully IPv6 capable and enabled, though only ipHouse sites currently have IPv6 addresses. If you are a web-hosting customer of ipHouse and would like to have your site enabled for IPv6, please email webmaster@iphouse.net with your request so we can make the necessary configuration changes in our load balancers..

Email inbound (greylisting servers), outbound (SMTP, POP, IMAP) are all IPv6 capable and enabled.

ipMom and webmail.iphouse.com are fully IPv6 capable and enabled.

DNS servers (caching resolvers and authoritative) are IPv6 capable and enabled.

T1, DS3, metro-ethernet, and colocation connectivity is all fully IPv6 capable and enabled (if you would like to request IPv6 connectivity, please send email to noc@iphouse.net with your request so we can work with you).

DSL service is IPv6 capable and enabled, though customer premise equipment gear is very lacking in IPv6 capabilities. See our previous blog post regarding this issue.

ipshell.com services are IPv6 capable and enabled.

MICE exchange connectivity is IPv6 capable and enabled – we are a full routing peer.

This blog site is IPv6 capable and enabled.

Our conference rooms and guest WIFI networks are IPv6 capable and enabled.

noc.iphouse.com is IPv6 capable and enabled.

Services at ipHouse that are IPv6 capable but not enabled, many of these will be enabled soon and this post will be updated to reflect the changes:

news.iphouse.com is IPv6 capable but not enabled.

lists.iphouse.net is IPv6 capable but not enabled.

stats.iphouse.com is IPv6 capable but not enabled.

Dial-up services are IPv6 capable though I don’t think we will be enabling it unless by customer request.

Microsoft IIS hosted websites are IPv6 capable but not enabled.

Services at ipHouse that are not IPv6 capable:

The Mailfoundry anti-spam systems, neither the web interface nor the SMTP services.

Our offsite DNS authoritative servers are not IPv6 capable at this time

I’m sure I missed a few things and I’ll update this as I find them (or remember, as the case may be).

If you have a Dialup, DSL or similar account at ipHouse with a ‘primary account’ within an iphouse.com, bitstream.net, pro-ns.net or goldengate.net domain you can now self-order additional POP mailboxes, and access an overview of mailboxes related to your primary account.

Check it out by logging into ipMom using your primary account.  You will see a new link titled ‘Add Mailbox’ under the ‘Add Services’ section of the menu.  You will also see ‘Account Overview’ under the Billing section.  You will be able to see your existing mailboxes and will be able to add new ones.  If you are already using all mailboxes that come with your account and you want to add a new one, you can check off an agreement to be billed for the new mailbox and a new POP mailbox will be added.  The new mailbox will be on to the next bill that is sent out (all billing information will stay the same).

You will not be able to delete mailboxes through ipMom.  If you do need to delete a mailbox, please email support@iphouse.net or billing@iphouse.net and we will be glad to help you out.

Customers with their own domain names who purchase mailboxes in blocks of ten will continue to enjoy the same admin control they currently have over their mailbox blocks.

We hope our customers enjoy this feature and we are excited for new things to come…:)

This is the mail system at host smtpin-2.iphouse.net.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

<blogtest@iphouse.com>: permission denied. Command output: maildrop: maildir
    over quota.

Breaking this error down; the first part tells us what server issued the rejection. In this case it was the host “smtpin-2.iphouse.net” – the mail server responsible for receiving email for the email address blogtest@iphouse.com   The second part tells us who we were sending the email to, in this case it was sent to testblog@iphouse.com  The third part tells us the error type, as in a permanent or temporary error. A permanent error means that the email won’t be delivered. A temporary error means that the sending mail server ought to try again later.  In this case it says “Permission Denied” – that’s a permanent error. The fourth part of this error message tells us why the email was rejected, in this case it says “Command output: maildrop: maildir over quota”. What this means is that the service “maildrop” says that the Mail Directory (maildir) is over quota – the mailbox is full. What this means for us the sender, is that we shouldn’t bother trying to send that person anymore email until they clear out their overstuffed mailbox. There isn’t anything we can do about it other that calling that person up to tell them that their mailbox is full.

Here’s the next error bounceback:

Unable to deliver message to: <blogtest@iphouse.com>
Delivery failed for the following reason:
smtpin-2.iphouse.net[216.250.188.181] responded with failure: 550 5.7.1 black
listed URL host ***.********.net by .black.uribl.com

This has been a permanent failure.  No further delivery attempts will be made.

I did make one edit to this error message, I replaced the blacklisted URL with a bunch of asterisks (*). I don’t see any reason to give a spammy site any more traffic than it already has. OK, so let’s break this error down. The first part of this error message tells us who we were sending the email to, in this case blogtest@iphouse.com was the intended recipient. The second part tells us what server issued the rejection, in this case it was the server smtpin-2.iphouse.net – so it was the recipient’s mail server issuing the rejection, not the sender’s. The third part tells us what kind of rejection it was, and why it was rejected. In this case the error code was “550″ (I’ll go into error codes later), and that the email was rejected because it had a link to a website (the one I used ***** to hide), which happened to be on the blacklist “black.uribl.com”. The last part is just another note that this was a permanent error, in case you didn’t already know that that’s what a 550 is. What this bounce back message means for us the sender is that we had a link to a website in our email that a spam filter on the recipient’s mail server didn’t like. We’ve got a few options here. The simplest thing would be to send the person a new email, but this time not include the offending website link. If we think that the website in question should not be on this blacklist in the first place (mistakenly listed), then we can always go to uribl.com and request that the offending website be removed from their list. Not all blacklists let you do this, but plenty do – and as it so happens, uribl.com lets you request removals. Another option would be to contact the intended recipient to tell them that a spam filter of theirs is rejecting that link. The intended recipient could then follow up with their IT staff, or their ISP. I suppose the final option would be to try to “game the filter” by breaking up the link a bit. For example, let’s say that iphouse.com was blacklisted somehow. I could try sending the person a link to iphouse.com like this:  www (dot) iphouse (dot) com    A human reading this ought to be able to figure out your intent, whereas a spam filter might be tricked.

Here’s the next error bounce back:

<testblog@iphouse.com>: host smtpgrey-2.iphouse.net[216.250.190.161] said: 550
    5.1.1 <testblog@iphouse.com>: Recipient address rejected: User unknown (in
    reply to RCPT TO command)

The first part of this error message tells us who we were sending the email to, in this case testblog@iphouse.com was the intended recipient. The second part tells us what server issued the rejection, in this case it was the server smtpgrey-2.iphouse.net – so it was the recipient’s mail server issuing the rejection, not the sender’s. The third part tells us what kind of rejection it was, and why it was rejected. It’s another 550, so a permanent rejection. It was rejected because the User was Unknown. This means that the email address doesn’t exist. If you caught the mistake in the email address right off the bat, plus 10 points for you. I accidentally sent an email to “testblog@iphouse.com” instead of “blogtest@iphouse.com”. Oops. As a funny aside, I actually generated this bounce back message accidentally when I was aiming for another type of bounce back. It took me a moment to catch my mistake. I guess that’s what more coffee is for…..    Anyhoo, what this means for us the sender, is that we just need to send out a new email, but this time type in the correct address. If you’re absolutely positively 100% double plus certain that you didn’t make any typo in the email you sent out, go ahead and look for the typo anyhow. It’s really easy to miss certain things like a “,” instead of a “.”, or a letter out of place. If your double check doesn’t yield any typos, and you’re certain that the address is a valid one, you might want to contact the intended recipient to inform them that something is amiss regarding their email address.

At the opposite end of our anti-spam system is content filtering. We use a third party vendor for this, MailFoundry in the form of two appliances. An appliance is a machine that you plug in, and is suppose to work with minimal configuration.

Now the MailFoundry appliances are “black box” systems. We don’t know how they work exactly, but we’re pretty sure that one of the techniques they use is Bayesian spam filtering.

Bayesian spam filtering uses the concept of probability to evaluate each token in a message, assign a weight to each, give the overall message a rating based on this weight, and evaluate the message based on a preset threshold.

Ok, unless you’re up on your statistics or logic based calculus, or a computer nerd with Wikipedia handy, I know your eyes just glazed over. Rest assured, you are not alone.

Basically, what it boils down to is that every “token” is a series of characters separated by whitespace. During this discussion, most “tokens” are words. Certain tokens are negative, they tend to appear in spam messages. Others are positive, they tend to appear in good (or ham) messages. Each token has a value (or a “weight”). A Bayesian spam filtering system reads the message, adds up all of the negative and positive weights of the tokens which produces an overall probability rating. If the rating is too negative, it considers the message spam. If it’s positive, it doesn’t.

Now, how does the Bayesian filter know which tokens are bad or good? Well, you have to give it examples of each. If a message is spam, and it gets through the filter, you have to tell the filter that it’s spam. When a message is marked as spam, all of the tokens in the message have their ratings lowered in the filter’s database. Ideally, you’d also mark good messages as good, but most people don’t. Most Bayesian filtering schemes are configured to mark all delivered messages as good unless they are marked as bad. Over time, the good tokens get “gooder” and the bad tokens get “badder” and the system can determine what is spam and what is not.

Bayesian spam filtering works amazingly well on individual email accounts, as it will be able to determine an individual’s taste in what is spam and what is not. Unfortunately, it’s not as effective across hundreds or thousands of users, but it still helps. Your personal spam filter will usually outperform anything on our end, because you may have a different definition of what is spam than other users out there. On a large system like ours, tokens that would be marked as negative for you, are nullified by others marking them as positive. So, say mail sent from a list that you signed up for, but no longer want may be spam to you, but may not be to other people out there. That’s why it’s best to unsubscribe from those lists rather than try to get our system to recognize it as spam.

If you want to help feed our filters, feel free to send examples of any spam you receive via our system, as attachments, to spam at iphouse.com.

I hope that helps!

1) If I’ve sent and received email from my friend for years, it shouldn’t get flagged as spam.

2) If I have their email address in my address book, their email won’t get flagged as spam.

3) If I avoid using certain words, my email won’t get flagged as spam.

None of these things are true. To understand why this is a tricky question to answer, it’s helpful to know a bit about what ISP’s are doing to filter spam. Most ISP’s have their own “custom blend” of what they do to filter spam, but it more or less boils down to using a combination of one or more of the following: Blacklists, Greylisting,  enforcing RFC’s, and more traditional Content Filters.

Blacklists can be based on all kinds of things. They can be lists of IP addresses that have been reported as sources of spam, lists of mail servers that have been found to be capable of being used as open mail relays, lists of URL’s that have been “spamvertised”, or any number of other things. Not all blacklists are the same. Some are very aggressive in what they list, and some are very conservative. The aggressive lists might block a lot of spam, but they are also more likely to have “false positives” – as in they blocked something that the recipient really did want to receive. Whereas the conservative lists might not have many false positives, but they’re likely to let more spam through.

Greylisting is when a receiving mail server issues a temporary error, which causes the sending mail server to re-queue the email and send it once more. Being able to re-queue an email is something that any RFC compliant mail server ought to be able to do. Greylisting can drastically reduce spam sent through “spam zombies” – home computers compromised by viruses that send spam out directly from the PC instead of through a mail server capable of re-queuing email.

RFC’s are, in a nutshell, the basic minimum standards for anything Internet related. Enforcing RFC compliance for mail can cut down on mail sent out from compromised PC’s/servers, and cut down on spam sent out from “sketchy” mail servers.

And lastly, content filters are the more traditional form of analyzing the content of an email to determine the “spamyness” of the email. Each spam filter system has its own “custom blend” of techniques to identify spam. Some of these criteria include; spammy words/spelling (\/1agra), format of an email (lot’s of CAPITAL/BOLD/etc lettering), lists of “spamvertised” websites, know spammer addresses, etc. Some filters use a feedback system that allows end users to submit examples of spam to train the filter.

Because blacklists and content filters are dynamic in nature, it can be very difficult to determine what it was at that exact moment that caused a particular email to be tagged as spam.