LWN: Comments on "Spam avoidance techniques"

Spam avoidance techniques

patrickcurrier — 2009-07-15T19:46:38+00:00

Wow time flies. Here's a more updated list of spam filters http://www.theyellowlists.com/TYL/Spam_Filters.html

Spam avoidance techniques

andycreats — 2006-08-15T12:36:44+00:00

I can recommend a good outlook spam filter - Spam Reader. I used it about 6 months and very satisfied. I tried before also Spam Bully and it worked fine but there was one thing I don't like - it's too much complicated in its options. I needed very simple but effective spam filter - Spam Reader is the best in this category. If you need some fully customizable spam filter then of course you need to use Spam Bully, but be ready guys to spent about 3-4 hours to read its help file :)

Spam avoidance techniques

patrickcurrier — 2005-04-19T23:44:04+00:00

Sorry I meant to add the actual link --- Spam Filter - http://www.spamfilternews.com

Spam avoidance techniques

patrickcurrier — 2005-02-08T02:06:07+00:00

by the way there is great info about spam filter stuff at http://www.spamfilternews.com

Bayesian Spam avoidance possible hole ?

jerry — 2004-10-02T12:58:57+00:00

Theoretically, it will not defeat the bayesian filter, but in reality, it does affect the filter, especially considering the impact on speed and memory usage. personally, I think designing an algorithm that "forget" rarely used tokens may be tedious or costly/impractical...

The way we used in a real world implementation spamweed is to mix bayesian filter with other technologies, especially those that can extract useful information among spammer's decoys, which significanly increase bayesian filter efficiency and stabilily.

Jerry: Engineer SpamWeed.com

Spam avoidance techniques

RobbyRDG — 2004-05-04T09:47:57+00:00

The spam filter that I use - Spam Bully, has multiple filtering techniques like friends/spammers list; block email by country/language; block certain words or phrases; RBL integrationalso it allows you to see detailed information about each email you receive- IP address, country, character set, and how SpamBully ranked it. Tells you why a message was or was not blocked and how to correct this in the future. But it always blows me away how incredibly accurate a pure Bayesian approach is. All those other methods add complexity to the process and no measurable improvement in accuracy except when you first stat training your filter. also it allows you to see detailed information about each email you receive- IP address, country, character set, and how SpamBully ranked it. Tells you why a message was or was not blocked and how to correct this in the future.

Bayesian Spam avoidance possible hole ?

mattknox — 2003-04-15T04:27:49+00:00

Actually, this would not work all that well, unless the spammers chose a document that is in your field. If you get a lot of mail about scripting languages or kernel development, then an article about one of these topics might help spam get through(at least a few times). However, if a random article that you would not normally recieve in the mail was attached, it would do nothing, because the terms would neither look like spam nor like ham. So the only way for spammers to win on this strategy is to find words that are found in mail that goes to a lot of people, and has not appeared in spam yet. This will be, at best, an uphill battle for them.

Spam avoidance techniques

Waldo — 2002-12-02T23:17:57+00:00

Hi,
filter training must be done with two sets of mails, archives of spam and good mail. Someone has to classify all mail to one of these. This might be done automatically with SpamAssassin or by each user. If the sysadmin does this job he/she has to read someone´s mail and this is against the law, at least in my country. The next problem is the archive itself. This is stored private data from different individuals, that is the next criminal act. It is not allowed to store private data, not the spam mail, it is the loveletter for your collegue.
So the user has to classify and this does not guaratee a better result.

Greetings from Europe

How does online shopping work

mcisaac — 2002-11-04T01:31:47+00:00

Great article!

I'm worried that routine activities such as online shopping might be difficult with this approach. In the "definition of spam" section, the paper touches on what is and is not spam, referencing a merchant receipt as an example of something commercial that isn't spam.

My question is, does the receipt pass the Bayesian filter or get flagged as spam?

Sued ?!?

job — 2002-09-19T11:08:16+00:00

Oh, I have no idea you lived in the US. It was not an anti-American comment at all, just pro-free society, no matter where on Earth it may be. Don't take it personally.

Bayesian Spam avoidance possible hole ?

guybar — 2002-09-18T06:29:18+00:00

It seems a bit stupid to ask, but what's stoping the spammers from attaching a random scientific/financial/other serious/ article after the actual spam ?

wouldn't this defeat the bayesian techniques described ?

(statistically) biased tests?

ElMiguel — 2002-09-13T21:01:18+00:00

But the numbers most people will remember from this article will be the ones with the 100% of lwn@lwn.net messages, since they are the ones showing the most striking advantage in favour of Bogofilter. And, as Bockman says, that is the least realistic test case of all, since you previously optimized the filter for precisely that set of messages. Perhaps you should make a note in the article itself to warn people who don't read the comments of that circumstance?

(Otherwise than that and overlooking spamc/spamd, great articles, as always :-)).

Sued ?!?

gswoods — 2002-09-13T16:36:54+00:00

Yeah, right. And just where is this 'free' country?

US law has lots of problems, but so does anyplace else.

Besides, this is a gratuitous anti-American comment. There's no need for
that here. We all have valid reasons for living where we do. I think it
would be stupid to move to another country just so that I could be free
to filter content for an entire organization. And it's just as easy to
argue that content filtering restricts the freedom of employees to use
the Internet. I'm not saying I agree with that argument, but you have to
be very careful when glibly tossing around the word 'free'.

Lastly, content filtering is not illegal even here. It's just that if you
filter, then you're responsible for what gets through your filters.
Personally, I would agree that's silly, but that's what the courts have ruled.

machine learning techniques

robertb — 2002-09-13T13:05:31+00:00

There are lots of machine learning techniques which do have reasonable test run-times (the training run-times can be quite high for some, 'though). I'm sure we'll hear about spam filters based on other techniques over the coming years, and hopefully not based on only words or even combinations of words. (Razor may be going in this direction with its fuzzy matching techniques (sucking up swaths of text rather than individual words).)

On a different subject, it's surprising that there's been no mention of "white list keywords". I think this can be an effective technique, particularly on the individual level. (Eventually, using Bayesian techniques such as ifile, these may be able to be generated automatically and then pruned by the individual as necessary.)

<begin plug>
See this page for lots of spam fighting techniques/ideas.
<end plug>

Sued ?!?

job — 2002-09-13T10:16:50+00:00

It sounds like the problem is that you don't live in a free country, rather than being a problem with content filtering.

Idea for increasing effectiveness

gswoods — 2002-09-12T22:54:07+00:00

I am curious about the legal issues. I personally am not a lawyer, but
when I have taken tutorials at conferences on Internet legal issues, I
have been warned repeatedly about content filtering. SpamAssassin and the
Bayesian filters are content filtering, because they examine the content
of the message itself and filter based on that. This is fine for the end
user to do, but if you do it as an organization, you are potentially
opening yourself up to a big liability. Remember the Prodigy case? The
ruling there was essentially that, since they were doing content filtering,
they were liable for whatever *did* get through. So if you use SpamAssassin
on the organization's mail server, and one of your employees gets a kiddie
porn spam in spite of that and is offended by it, you could be sued.

We have started using IP blacklist filters here. This is safer from the
legal point of view, because the content of the message itself is never
examined. The message is rejected before it is ever sent. Our blacklist
filters have a lot of false negatives, but the problem with false positives
has been nearly nonexistent. Also, I used to get hundreds of bounced spams
every day, and the number has dropped to nearly zero since we started filtering.

I think IP blacklists still have their place.

Spam avoidance techniques

bitbytebit — 2002-09-12T19:07:49+00:00

that's actually BlackHole on Freshmeat.

Spam avoidance techniques

bitbytebit — 2002-09-12T19:05:46+00:00

A program I have developed in C and uses SpamAssassin and many other spam blocking techniques combined (even can run SpamAssassin or BogoFilter from it) is BlackHole, available at BlackHole on FreshMeat. It also combines free virus checking or Sophos/Mcafee/Trendmicro checking, and many more techniques of filtering email, hopefully it can help too. Thanks, Chris getdown@groovy.org

(statistically) biased tests?

corbet — 2002-09-12T18:26:51+00:00

"A better test was maybe to train the filter with half of the data set and then test it with the other half."

That was the first (15%) test, essentially. And the linux-kernel test too.

(statistically) biased tests?

bockman — 2002-09-12T17:49:13+00:00

For the little I know of statistic filters, if you train a filter with a set of data, then you should not use the same set of data to evaluate how good the trained filter is ( since you are testing on the training data, the filter obviously shows good results ).

A better test was maybe to train the filter with half of the data set and then test it with the other half.

spamc/spamd

corbet — 2002-09-12T14:28:41+00:00

You know, mentioning (if not trying) the spamc/spamd pair was on my list as I put the article together, but somehow in the excitement of ordering all those bulk email lists I dropped it. It's really true that reading your spam rots the brain.

I just ran the 5000-message linux-kernel test using spamd. The filtering results were the same, of course, and the run time dropped to 2400 seconds. That's a big speed improvement, but still an order of magnitude slower than bogofilter.

Here we go re-inventing the ego-wheel again

garym — 2002-09-12T13:55:35+00:00

Remember the Freshmeat editorial on how to pick an opensource project? The basic sarcastic rule was "Pick something already done, and do it the same way." Bogofilter may not be mature (and the less said about buffer-overrun and similar failures in its sibling software the better), but iFile is not; its probably unwise to chastize a net god, but even if Eric has black kettle concerns that it's not proper, I can't command him, but I'd rather he just contribute to the pre-existing software, standing on shoulders instead of standing on the toes of others. iFile is all ready, coded and deployed, and contributor packages include scripts for folding in with procmail or any of about half a dozen email readers, and, of course, it's free software.

Another paper

jrennie — 2002-09-12T13:17:53+00:00

FYI, k-nearest neighbors (kNN) is very slow compared to filtering by rules or Bayesian approaches (like Graham describes, bogofilter and ifile). For each message you want filtered, kNN compares that message to all messages in the training database. So, filtering n messages is O(nm) (where m is # of training messages). Bayesian approaches scale as O(n).

Jason Rennie
Author of ifile - the original intelligent e-mail filter

Another paper

armijn — 2002-09-12T12:21:28+00:00

At the SANE 2002 conference in the Netherlands a paper was presented
about a self learning content based spam filter. According to it you
can get quite some good results with it:

http://www.nluug.nl/events/sane2002/papers.html

Instead of Bayesian learning it uses the k-nearest neighbours algorithm.

Spam avoidance techniques

Dom2 — 2002-09-12T07:36:47+00:00

I do this and it makes spamassassin usable over my dialup. At this point, most of my runtime costs are DNS lookups from spamd.

-Dom

Spam avoidance techniques

fcrozat — 2002-09-12T05:35:07+00:00

Since SpamAssassin is written in perl, when you use it through procmail, a new perl intepreter is started for each message.. This is the main cause of the high "run time" figure.

To prevent this from happening, you should use the spamc/spamd tools which are shipped with SpamAssassin :

Spamd is a daemon which starts a spamassassin process which is kept in memory, fixing the startup latency problem of SpamAssassin.
Spamc is a client which connect to spamd and can be used instead of spamassassin in procmail rules (replace "spamassassin -P" with "spamc").

You should try it, runtime will probably more reasonable.

Spam avoidance techniques

dwheeler — 2002-09-12T04:12:33+00:00

It's certainly reasonable to combine multiple anti-spam techniques, in fact, a lot of people do exactly this.

Obviously, a lot of people only learned about this technique from Paul Graham's plan for spam (a well-written piece!). The LWN study shown here is wonderful confirmation that it has value. It's worth noting that there have been other studies on the topic, including An evaluation of Naive Bayesian anti-spam filtering, An Experimental Comparison of Naive Bayesian and Keyword-Based Anti-Spam Filtering with Personal E-mail Messages, Learning to Filter Spam E-Mail: A Comparison of a Naive Bayesian and a Memory-Based Approach, and information from lsi.upc.es and monmouth.edu; Slashdot has carried a discussion about it. Ifile implemented the idea many years ago - it claims a first release date of Aug 3 20:49:01 EDT 1996, and the author doesn't claim that this program is the first implementation of the idea, either.

A selected set from the newsgroup news.admin.net-abuse.sightings might be useful for initial training of a spam filter. That would eliminate the problem you mention.

I think every email reader should have a "big SPAM button" that adds an email to the "spam" folder (so it can be used for future analysis), as well as other configurable actions. See http://www.dwheeler.com/essays/stopspam.html for more information about this.

Idea for increasing effectiveness

Strike — 2002-09-12T03:25:39+00:00

Maybe I'm crazy, but I don't see why you can't simply daisy-chain the two together to provide even better results. This way you can tweak SpamAssassin to a good enough target score that won't produce false positives (I've found that an aggregate score of 8 or so without changing any of the test scores does a fine job, though does miss a few), and then the mails that have gone to great enough length to assure that all the header tests, MX tests, subject tests, and content (such as MIME type) tests that SpamAssassin does don't pump up the score very high will be subject to the Bayesian approach as well.

This way, spam mails that are clever enough to pass one but not the other, will be tossed aside.