At the opposite end of our anti-spam system is content filtering. We use a third party vendor for this, MailFoundry in the form of two appliances. An appliance is a machine that you plug in, and is suppose to work with minimal configuration.

Now the MailFoundry appliances are “black box” systems. We don’t know how they work exactly, but we’re pretty sure that one of the techniques they use is Bayesian spam filtering.

Bayesian spam filtering uses the concept of probability to evaluate each token in a message, assign a weight to each, give the overall message a rating based on this weight, and evaluate the message based on a preset threshold.

Ok, unless you’re up on your statistics or logic based calculus, or a computer nerd with Wikipedia handy, I know your eyes just glazed over. Rest assured, you are not alone.

Basically, what it boils down to is that every “token” is a series of characters separated by whitespace. During this discussion, most “tokens” are words. Certain tokens are negative, they tend to appear in spam messages. Others are positive, they tend to appear in good (or ham) messages. Each token has a value (or a “weight”). A Bayesian spam filtering system reads the message, adds up all of the negative and positive weights of the tokens which produces an overall probability rating. If the rating is too negative, it considers the message spam. If it’s positive, it doesn’t.

Now, how does the Bayesian filter know which tokens are bad or good? Well, you have to give it examples of each. If a message is spam, and it gets through the filter, you have to tell the filter that it’s spam. When a message is marked as spam, all of the tokens in the message have their ratings lowered in the filter’s database. Ideally, you’d also mark good messages as good, but most people don’t. Most Bayesian filtering schemes are configured to mark all delivered messages as good unless they are marked as bad. Over time, the good tokens get “gooder” and the bad tokens get “badder” and the system can determine what is spam and what is not.

Bayesian spam filtering works amazingly well on individual email accounts, as it will be able to determine an individual’s taste in what is spam and what is not. Unfortunately, it’s not as effective across hundreds or thousands of users, but it still helps. Your personal spam filter will usually outperform anything on our end, because you may have a different definition of what is spam than other users out there. On a large system like ours, tokens that would be marked as negative for you, are nullified by others marking them as positive. So, say mail sent from a list that you signed up for, but no longer want may be spam to you, but may not be to other people out there. That’s why it’s best to unsubscribe from those lists rather than try to get our system to recognize it as spam.

If you want to help feed our filters, feel free to send examples of any spam you receive via our system, as attachments, to spam at iphouse.com.

I hope that helps!

1) If I’ve sent and received email from my friend for years, it shouldn’t get flagged as spam.

2) If I have their email address in my address book, their email won’t get flagged as spam.

3) If I avoid using certain words, my email won’t get flagged as spam.

None of these things are true. To understand why this is a tricky question to answer, it’s helpful to know a bit about what ISP’s are doing to filter spam. Most ISP’s have their own “custom blend” of what they do to filter spam, but it more or less boils down to using a combination of one or more of the following: Blacklists, Greylisting,  enforcing RFC’s, and more traditional Content Filters.

Blacklists can be based on all kinds of things. They can be lists of IP addresses that have been reported as sources of spam, lists of mail servers that have been found to be capable of being used as open mail relays, lists of URL’s that have been “spamvertised”, or any number of other things. Not all blacklists are the same. Some are very aggressive in what they list, and some are very conservative. The aggressive lists might block a lot of spam, but they are also more likely to have “false positives” – as in they blocked something that the recipient really did want to receive. Whereas the conservative lists might not have many false positives, but they’re likely to let more spam through.

Greylisting is when a receiving mail server issues a temporary error, which causes the sending mail server to re-queue the email and send it once more. Being able to re-queue an email is something that any RFC compliant mail server ought to be able to do. Greylisting can drastically reduce spam sent through “spam zombies” – home computers compromised by viruses that send spam out directly from the PC instead of through a mail server capable of re-queuing email.

RFC’s are, in a nutshell, the basic minimum standards for anything Internet related. Enforcing RFC compliance for mail can cut down on mail sent out from compromised PC’s/servers, and cut down on spam sent out from “sketchy” mail servers.

And lastly, content filters are the more traditional form of analyzing the content of an email to determine the “spamyness” of the email. Each spam filter system has its own “custom blend” of techniques to identify spam. Some of these criteria include; spammy words/spelling (\/1agra), format of an email (lot’s of CAPITAL/BOLD/etc lettering), lists of “spamvertised” websites, know spammer addresses, etc. Some filters use a feedback system that allows end users to submit examples of spam to train the filter.

Because blacklists and content filters are dynamic in nature, it can be very difficult to determine what it was at that exact moment that caused a particular email to be tagged as spam.

It’s also our first line of defense against spam.

Greylisting is a very simple technique. It is a daemon attached to database that keeps track of who externally sent mail to whom internally, including from what IP address. When a new sender/recipient/IP-address (or triplet as it is called) combination pops up, it bounces the transaction with a temporary, 450/451 response code. This is per the RFC and any properly implemented SMTP server should adhere to it, re-queue the message, and send it again later. If the server sends it before a specified “too early” window (in my case on my personal server, 2 mins, but that’s fairly aggressive) it’s temp-failed (tech term for try again later) again. If the message comes back after this “too early” window, but before a 24 hour expiration window, the message is passed through, and an entry is made in the database allowing that triplet to send mail unhindered for a few days (depending on configuration). If enough messages come from the same ip address and domain pass Greylisting, that whole domain can be automatically white-listed through the check.

The goal of greylisting is not to penalize legitimate mail servers but only to stop non-compliant botnets from getting through.

Greylisting is very effective because it keeps non-compliant SMTP servers from sending mail to our (or even your) servers. Most virus infected computers that send or relay spam won’t re-queue messages, or will re-queue them for only the briefest amount of time. Why? Their goal is to blast as much email/virus payload as possible, and any slowdown or long retry time is very counterintuitive to this goal.

Problems with greylisting are legitimate, by mis-configured SMTP servers either not re-queuing the messages because they are set to treat 400 series bounces as 500 series (permanent) bounces. Or they re-queue the messages, but report to the original sender that the message bounced.

Yahoo implements a more esoteric set up, where they have 4 servers listed in the MX record, and at any time, any of them will bounce messages. This is another way to test for non RFC compliant servers, as a server is supposed to try all of the MX entries in turn, by weight value. Most virus infected computers won’t do that. At least that is what it looks like from the outside.

Because some of our users may have problems with receiving mail, our web-based interface, ipMom, gives you the option to disable greylisting. If you log into ipMom with your email address and password, you’ll notice a “Greylist” option . Set it to off, and greylisting is no longer affecting your mail. Keep in mind that this does let more spam into the system, although our other anti-spam protections may still catch them.

I hope that helps!

Phishing is spam that attempts to extract personal information from the recipient. Here are some quick points about Phishing:

1. Email asks for your password: ipHouse will never ask for your password via email. This is a common policy with many companies so feel free to make it your own policy: Never send a password via email even if you think you know the recipient.

2. Strange reply-to address: The reply-to email address is not an official email address. ipHouse employees and internal addresses are all @iphouse.net. This latest round had the reply-to as an email address in Brazil (.br) or a yahoo.com address. A general rule for anyone is to always check a provider’s website for valid contact information. When going to their website type in the address yourself or use an existing valid bookmark. Do not click a link in an email even if it looks valid is it may be a “masked” URL whose destination is a different address.

3. Credit card fraud. While this email was looking for passwords, many Phishing scams ask for credit card numbers. And for decades there have been phone-based credit card Phishing scams. ipHouse will never ask for your credit card number via email nor ever via a call we initiate. Feel free to make it your own policy with everyone – never send a credit card number via email and never give your credit card number out to someone unless you initiate the call.

4. Spam filters don’t catch everything. While our multiple levels of Antispam catch most Phishing expeditions, some can get through. This one was harder to catch as it didn’t have any off-site hyperlinks and had enough words that it looked valid to the filters. We don’t publish for spammers how we adjust but trust me that we do adjust. Of course we do want to see what might get through. For example, yesterday alone ipHouse blocked 1,463,418 spam, Phishing, and viruses. We pride ourselves on an extremely low “false positive” rate. If a spam or Phishing message does get through, please forward it with full headers to spam@ipHouse.net. If you have an individual question or concern, our Support team can help.

5. Learn more! Here are some links to several sites’ take on Phishing:

• Blogs about Phishing: PhishingScam
• Popular OS: Apple, Microsoft
• Popular Guides (always with a grain of salt please): WikiPedia , About
• Trade/Industry groups: APWG, National Cyber Security Alliance, AARP
• Government: Stop-Think-Click

- Eric

ipHouse engineers have been waging a battle against spam since the 1990s. As the Internet has grown and changed during the last 15+ years, the amount of spam has increased exponentially.

The problem of online viruses has also grown. Effective spam and virus filtering is now an essential component of any mail server.

The dilemma is always how to balance false positives with reliable protection.

Mike Horwath, ipHouse senior admin, recently wrote about his frustration trying to find the perfect anti-spam solution.

ipHouse has developed a fairly comprehensive anti-spam solution for mailboxes hosted on our mail cluster. Not only are all incoming emails scanned for spam and viruses by our MailFoundry system, there are also a wide range of filters and blacklists that can be applied on a per mailbox basis. These customizable filters help customers get only the mail they want, because one person’s spam is another person’s legitimate email.

Through ipMom, ipHouse customers can decided exactly how many hoops they want their mail to jump through on its way to their inbox. Options include turning greylisting on or off, enabling various sender checks, and applying both DNS (IP address) blacklists and RHS (domain name) blacklists to incoming emails.

Greylisting is one of the most simple and most effective ways to stop spam.  If greylisting is turned on, new incoming email is deferred from the sending mail server. Mail servers that are configured correctly will just re-send the message and on the second send, it is successfully delivered to your inbox. This simple technique works because many spammers use mail servers that are not correctly configured.

Sender Checks check both the sending mail server and the domain name associated with the sender to make sure they are configured properly. Sender Checks can block a fair amount of legitimate email and should be used with caution.

Blacklists range from the highly conservative with very few false positives to the gung-ho that seem to block messages if they even look at another message that might be spam. More detailed explanations of the different types of spam and virus checks are available on the ipHouse support pages.

ipHouse has long provided customers running their own mail servers the ability to have their incoming email scanned by the MailFoundry system. Yet we have been frustrated by our inability to do more.  This summer, our engineers have been working on new anti-spam solutions for customers with their own mail servers.  We are just finishing the beta tests and hope to formally launch the product by the end of summer.

We are very excited about this upcoming feature.