How many times have you read that? Personally, I’d rather get stuff I don’t want than not be able to find stuff I do want. Spam you can delete, but if you don’t know a message exists, then how can you tell if you want to read it? Perhaps its time for a re-think about spam. But it pays to understand what is going on first.
Spam is like weeds. You cannot classify a plant as a weed when its simply a plant you don’t happen to want in your garden. And some plants you thought were weeds turn out to be really lovely. Similarly, there is no formal definition of a spam message – and there never will be. There comes a time you might actually need Viagra!
Spam can be extracted or filtered at one or more of 3 levels (see diagram):
- Level 1: before it gets to your email server – by a spam filtering service or more technically, a transparent proxy spam filter.
- Level 2: Before it gets to your email client/app – by your company’s email server.
- Level 3: before it gets to you- by the email client/app on your phone or computer.
At each level, the spam filter will move the spam email to somewhere (usually called a “quarantine” folder) that it can be retrieved – should someone have the time and effort to go through it. If, as is often the case, companies have filtering at all three levels, there would be three different places to look. Of course, nobody ever does this unless they are absolutely desperate. And they never look if they don’t know they have something to look for.
The rules for deciding what constitutes spam can be complex. There are rules based on known spam sources, known spam content (eg subject contains “Viagra”) or something “intelligent” where the filter learns what spam means for that user or company.
Some time ago, we decided that the advantages of spam filtering were often outweighed by the disadvantages of losing or not being able to find non-spam email. We also discovered that yesterday’s “spammer” could become today’s customer and that by the time they had become the customer, we had lost the trail of communication that led to the sale. That meant we could not learn from the process.
So we started to abandon the idea of generic spam filtering. Instead, we began classifying email according to the participants – the senders and recipients. Basically, the participants fell into one of two categories:
- People we know
- People we don’t know
If a message comes from someone we don’t know, then mostly – but not always – it is unsolicited. Most traditional spam falls into this category. However, with some intelligence, you can add a third category:
- People we might know
A simple example of this might be a message from an unknown contact (say fred@somecompany.com) similar to someone we already know (say jim@somecompany.com).
By storing all messages – spam and non-spam – and allowing users to instantly change their view of messages according to contacts they know, might know and don’t know we found we didn’t need a spam filter. And what was great was that when a contact we didn’t know became a contact we knew (eg became a customer) then we could instantly discover what thread of communication let up to that.
Our system, called Threads, it not a spam filter and was never designed as one; we simply discovered that it made the spam filter redundant. Threads does not replace users’ email servers or apps – we knew it would be destined to fail if we tried to force users to abandon their favourite email system. Instead, it sits quietly in the background collecting, categorising (and deduplicating) all the company’s unfiltered messages. It collects email from any number of sources and even ingests VoIP phone calls – yes, you get phone spam too. As its name implies, its main task is to work out the threads of conversation. Dealing nicely with spam is a fringe benefit. Users are still free to apply their own spam filtering, but if ever they are trying to find something, they usually go to Threads first. Also, because they can search all of their company’s (non-confidential) messages, they can find important messages that were never in their own mailboxes in the first place.
After we had commercialised Threads and began to seek customers, we discovered something really interesting. Many companies using proprietary spam filtering services had absolutely no idea that ALL their messages were going to the service provider before they were getting to them. This is a called a Transparent Proxy Spam Filtering and technically, it makes excellent sense. But it amazed us how few customers understood what was happening. Redirecting all your messages to the service provider is not something you can easily turn off and on. And even fewer had read the service provider’s privacy terms which, while maintaining individual user’s anonymity, did not prevent them from extracting information from the customer’s message content. For example, this is an extract from Symantec’s privacy policy, a US company that provides spam filtering products and a popular service called Messagelabs…
The collected data may be transferred to the Symantec group in the United States or other countries that may have less protective data protection laws than the region in which you are situated (including the European Union), but Symantec has taken steps so that the collected data, if transferred, receives an adequate level of protection.
Oh really? It’s very reassuring that that the “collected” data receives an “adequate” level of protection, whatever adequate might mean. And I am not picking on Symantec. This sort of clause is not uncommon in the privacy policy statements of many large service and application providers – if only it was ever read. Also, although many providers will anonymise the customer’s data, their privacy policy does not preclude them extracting information from its content.
So next time you can’t find something that may have ended up as spam, remember your spam filter did its best – for you and perhaps for the spam service provider too!