Undeliverable mail
Jan 19th
One of the most common email related support calls I get is someone wondering why they’re receiving “Undeliverable Mail Returned to Sender” notifications when sending out email to a particular email address. These “bounce back” messages will tell you why the email couldn’t be delivered, but often times the language used isn’t immediately understandable by someone who isn’t an IT professional, or “into computer stuff”. I thought I’d go over some of the more common undeliverable errors, and explain what they mean. I made a test email address for this purpose, blogtest@iphouse.com – and then sent several emails to it that were intended to be bounced for various reasons. Here’s the first bounce back message:
This is the mail system at host smtpin-2.iphouse.net. I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.
<blogtest@iphouse.com>: permission denied. Command output: maildrop: maildir over quota.
Anti-spam Part 2, Bayesian Spam Filtering
Jan 5th
Well, Andrew and I kinda stepped on each others toes last month, but I’ll go into a little more depth on some of the things he touched on. Last month I talked about the frontend of our anti-spam filtering via Greylisting.
At the opposite end of our anti-spam system is content filtering. We use a third party vendor for this, MailFoundry in the form of two appliances. An appliance is a machine that you plug in, and is suppose to work with minimal configuration.
Now the MailFoundry appliances are “black box” systems. We don’t know how they work exactly, but we’re pretty sure that one of the techniques they use is Bayesian spam filtering.
Bayesian spam filtering uses the concept of probability to evaluate each token in a message, assign a weight to each, give the overall message a rating based on this weight, and evaluate the message based on a preset threshold.
Ok, unless you’re up on your statistics or logic based calculus, or a computer nerd with Wikipedia handy, I know your eyes just glazed over. Rest assured, you are not alone.
Why was my email flagged as spam?
Dec 8th
“Why was my email flagged as spam?” This is a very common question, and while it looks like a simple one on the surface, it’s actually not as easy to answer as you might think. Common misconceptions are;
1) If I’ve sent and received email from my friend for years, it shouldn’t get flagged as spam.
2) If I have their email address in my address book, their email won’t get flagged as spam.
3) If I avoid using certain words, my email won’t get flagged as spam.
None of these things are true. To understand why this is a tricky question to answer, it’s helpful to know a bit about what ISP’s are doing to filter spam. Most ISP’s have their own “custom blend” of what they do to filter spam, but it more or less boils down to using a combination of one or more of the following: Blacklists, Greylisting, enforcing RFC’s, and more traditional Content Filters.
Blacklists can be based on all kinds of things. They can be lists of IP addresses that have been reported as sources of spam, lists of mail servers that have been found to be capable of being used as open mail relays, lists of URL’s that have been “spamvertised”, or any number of other things. Not all blacklists are the same. Some are very aggressive in what they list, and some are very conservative. The aggressive lists might block a lot of spam, but they are also more likely to have “false positives” – as in they blocked something that the recipient really did want to receive. Whereas the conservative lists might not have many false positives, but they’re likely to let more spam through.
Greylisting is when a receiving mail server issues a temporary error, which causes the sending mail server to re-queue the email and send it once more. Being able to re-queue an email is something that any RFC compliant mail server ought to be able to do. Greylisting can drastically reduce spam sent through “spam zombies” – home computers compromised by viruses that send spam out directly from the PC instead of through a mail server capable of re-queuing email.
RFC’s are, in a nutshell, the basic minimum standards for anything Internet related. Enforcing RFC compliance for mail can cut down on mail sent out from compromised PC’s/servers, and cut down on spam sent out from “sketchy” mail servers.
And lastly, content filters are the more traditional form of analyzing the content of an email to determine the “spamyness” of the email. Each spam filter system has its own “custom blend” of techniques to identify spam. Some of these criteria include; spammy words/spelling (\/1agra), format of an email (lot’s of CAPITAL/BOLD/etc lettering), lists of “spamvertised” websites, know spammer addresses, etc. Some filters use a feedback system that allows end users to submit examples of spam to train the filter.
Because blacklists and content filters are dynamic in nature, it can be very difficult to determine what it was at that exact moment that caused a particular email to be tagged as spam.
Anti-Spam Part 1, Greylisting
Dec 5th
I’ve occasionally gotten calls from system administrators about a “mail bouncy thing” they notice in their logs when they send mail to us. They find it weird and sometimes frustrating and many consider it a silly anti-spam technique. Well, that would be greylisting, and while it’s weird, it also drops a lot of spam getting through to our customers.
It’s also our first line of defense against spam.
Greylisting is a very simple technique. It is a daemon attached to database that keeps track of who externally sent mail to whom internally, including from what IP address. When a new sender/recipient/IP-address (or triplet as it is called) combination pops up, it bounces the transaction with a temporary, 450/451 response code. This is per the RFC and any properly implemented SMTP server should adhere to it, re-queue the message, and send it again later. If the server sends it before a specified “too early” window (in my case on my personal server, 2 mins, but that’s fairly aggressive) it’s temp-failed (tech term for try again later) again. If the message comes back after this “too early” window, but before a 24 hour expiration window, the message is passed through, and an entry is made in the database allowing that triplet to send mail unhindered for a few days (depending on configuration). If enough messages come from the same ip address and domain pass Greylisting, that whole domain can be automatically white-listed through the check.
The goal of greylisting is not to penalize legitimate mail servers but only to stop non-compliant botnets from getting through.
Greylisting is very effective because it keeps non-compliant SMTP servers from sending mail to our (or even your) servers. Most virus infected computers that send or relay spam won’t re-queue messages, or will re-queue them for only the briefest amount of time. Why? Their goal is to blast as much email/virus payload as possible, and any slowdown or long retry time is very counterintuitive to this goal.
Problems with greylisting are legitimate, by mis-configured SMTP servers either not re-queuing the messages because they are set to treat 400 series bounces as 500 series (permanent) bounces. Or they re-queue the messages, but report to the original sender that the message bounced.
Yahoo implements a more esoteric set up, where they have 4 servers listed in the MX record, and at any time, any of them will bounce messages. This is another way to test for non RFC compliant servers, as a server is supposed to try all of the MX entries in turn, by weight value. Most virus infected computers won’t do that. At least that is what it looks like from the outside.
Because some of our users may have problems with receiving mail, our web-based interface, ipMom, gives you the option to disable greylisting. If you log into ipMom with your email address and password, you’ll notice a “Greylist” option . Set it to off, and greylisting is no longer affecting your mail. Keep in mind that this does let more spam into the system, although our other anti-spam protections may still catch them.
I hope that helps!
