We could eliminate SPAM tomorrow if...

We are all familiar with SPAM, its the huge volume of unsolicited crap that we have to wade through each day just to do our jobs, and yet there's no sign of it going away despite us all having the means to end it. So let's look at why we are all being subjected to the spam and then we'll look at why we don't end it when we all have the power to do so. 

The reason for SPAM

SPAM has three basic objectives and in order of volume, 

• Firstly the majority of SPAM is an attempt to infect your workstation, laptop, tablet etc with a virus and/or trojan. By doing this the spammers have (a) the ability to scan your system for card numbers, passwords, and of course email addresses from your email client, (b) steal the login credentials for your email account so they can use it to propagate more spam FROM YOU, and (c) in order to leverage DoS attacks. 
• Secondly, Spam will attempt to impersonate an organisation that you might expect an email from and then trick you into giving up your login, password, account and so on by taking you to a fake website. Whilst you may think most people are weary of this type of spam you would be surprised how many we still get at the helpdesk. 
• Finally, Some spam can actually be trying to sell you something, which is rare these days but does still happen. 

Current SPAM defences

• The blacklist: A number of worthy organisations like Spamhaus, SpamCop, etc are dedicated to maintaining lists of domains, hosts and subnets which are used to originate spam. Using these blacklists is an expensive but effective tool to eliminate a good percentage of spam at the first gate. Blacklists however are not realtime, and there is always a delay between a spammer launching a mass mailing and the blacklists listing it. 
• Authentication: Several technologies exist to verify sender domains and hosts such as SPF & DKIM and these can serve (where used by the receiving server) to block spoofed spam which constitutes the vast majority of scams. For example, the HMRC who are under constant attack from scammers specify in their SPF records two hosts that are allowed to send email for @hmrc.gov.uk and of course the spammers cannot originate email from those addresses so SPF wins the day and any email coming from, say This email address is being protected from spambots. You need JavaScript enabled to view it. that doesn't come from the two hosts listed in the SPF record are canned. This however all falls down when the receiving server doesn't check, the sending organisation doesn't use it, or the sending organisation has been compromised.
• DNS: The domain name system is that which coverts gen.net.uk to 212.140.242.10 and back again, and when you send email to someone @gen.net.uk DNS gives up the address of the mail server that is designated to receive that email, in this case farpoint.gen.net.uk. The RFC1124/1124 which form part of Internet Standard 1 specify clearly that every host on the internet should have forward and reverse DNS, that is gen.net.uk to 212.140.242.10 and 212.140.242.10 to gen.net.uk. So, when a host 'spammer.com' connects from 212.140.242.50 to our mail server, we (a) check that 212.140.242.50 corresponds to 'spammer.com', that 'spammer.com' has a valid MX record and that the host listed in the MX record actually exists on the internet. This is particularly hard for a spammer to forge and therefore this check eliminates a percentage of spam as well as a percentage of legitimate email from companies who don't know how to setup very basic DNS correctly. 
• Content Filtering: By far the most effective tool at eliminating spam which passes all the above tests is pattern matching. This involves looking and detecting elements in the body of an email and assigning a score to each detection. An example would be a HTML only email which scores 3 points, external links to pictures which scores 0.2 points each and so on. The more spammy the email the more points it will accumulate and once a threshold is reached the message is flagged as spam. Content filtering can make use of content lists which are maintained by third parties and provide known phrases and content to score. 
• Bayesian Probability Filtering: A gross simplification of this would be that email which is known to be spam can be 'learned' and that data used to identify 'similar' spam. The area of mathematics is complex and the techniques even more so, but the result is the same in that spam that looks like spam based on learned data can be flagged as such, usually by giving it a score, such as +10

And with these methods we can and do filter around 80% of your spam, but its never ever going to be 100% because SPAMmers spend a great deal of their time trying to circumvent these filters likewise costing us a great deal of money to continually adapt the filters for maximum effect. 

BUT, we do have the ability to stop the SPAM completely, 100% total removal of spam so why don't we? Well, quite simply we cannot because in this day and age everyone's an expert when of course they aren't. Using the current standards, and systems we could easily: 

• Eliminate the source of SPAM by authenticating the source of all email both by using DNS and SPF. This would mean that email can only be sent if it originates from an authenticated server and if all the ISP's got together an setup their systems in this manner (most already do) then spammers would ONLY be able to send spam by compromising users email credentials. That's going to immediately eliminate 67% of SPAM. 
• Use the tools we all have available to track, trace, and block email origination 'out of zone'. That is, for every email account the email server will ONLY accept email from the senders company LAN, or their country of residence. This kind of geolocation limiting is already built into all the modern mail systems, but its rarely used. 
• Use anti-hijack detection to automatically flag accounts that are likely to be compromised by looking for unusual email activity. For example, if a mailbox normally originates 50 email's a day and then suddenly originates 50 emails a minute then we have the systems to automatically block that behaviour until the mailbox owner contact's us.
• The use of S/MIME certification, which is free for individuals, and only a nominal charge for businesses not only provides transparent encryption of business email, but also provides authenticity to every recipient, so that when you receive an email from This email address is being protected from spambots. You need JavaScript enabled to view it., it comes with a 'seal' that confirms the email came from fred at bloggs.com. We've used these for the last decade, but we're pretty much alone in this. 

So, it doesn't sound that hard does it? Well its not, but unfortunately as an ISP with many customers there are always going to be the few who effect the many as in many business models. No matter how much you promise your customers a spam free life, a minority of customers don't want to hear that fredbloggs inc doesn't meet the standards and/or is blacklisted and therefore cannot send them email, they just insist how important it is that fredbloggs inc can email email them. This creates a real problem for ISP's who technically want to kill spam as promised to their customer base but are also aware of the real world cost of dealing with ticket after ticket of 'I can't receive email from xxx' and the time and effort spent identifying the sender doesn't comply or is blacklisted then trying to explain that to the customer.  

So our approach, which has been adapted over the years is to offer three levels of protection: 

1. No Filter - All email is accepted regardless. All Spam and Viruses are delivered untouched. 
2. Basic Filter - Some filtering is done, but spam is still delivered with [SPAM] in the subject line allowing customers to filter that into a spam folder if required. Some antivirus protection is enabled. 
3. Max Filter - All the above fully enabled and active both Anti-Spam and Anti-Virus. 

And as we expected the vast majority of business and corporate customers opt for the Max Filter, with only a very few opting for other options. The customers who opt for and stay with the Max Filter understand the issues and stand with us on the fight against spam. If a sender winds up blacklisted then they don't tell us, they tell the sender to sort it out. 

So what's the future? Well unfortunately as it stands with some ISP's favouring an easy life rather than deploying the available protections, with players like Microsoft and Google seemingly doing nothing to limit the spam they collectively originate, and with senders especially in the less advanced countries not able to configure even the very basic standard requirements we're going to be up to our armpits in spam for a good while to come but I do feel that things are changing as we're already seeing customers migrating to us solely for the benefits of our protection systems and that means we're doing it right. 

There are a number of articles on Blacklists, SFP, DKIM on our FAQ as well as the internet standards 1 RFC's. They are all technically orientated but available for anyone who's interested.