| |||||||||||||
The Grumpy Editor's guide to bayesian spam filtersThe Grumpy Editor's guide to bayesian spam filtersPosted Feb 22, 2006 23:58 UTC (Wed) by njhurst (guest, #6022)Parent article: The Grumpy Editor's guide to bayesian spam filters
To be fair, people other than Graham built bayesian spam classifiers before his paper. I recall that Microsoft published a paper a good 6 months earlier. Here's one from 1998 (last millenium :):
http://research.microsoft.com/~horvitz/junkfilter.htm
Not sure why Graham paper gets such acclaim.
Currently I'm just using greylisting and getting good results. Spam Assassin is become unworkable. I'll probably link to SPEWS or similar when spammers cotton on to greylisting.
(Log in to post comments)
Filtering even earlier - 1996 Posted Feb 23, 2006 0:31 UTC (Thu) by dwheeler (guest, #1216) [Link] Ifile implemented the idea of automatically categorizing years earlier thatn that. It claims a first release date of Aug 3 20:49:01 EDT 1996. The author doesn't claim that is the first time this has been implemented, either.But Graham rightly gets credit for promulgating the idea. Having an idea isn't enough... other people have to know about it. Graham did a good job of connecting problem and solution, in a way that got others interested.
The Grumpy Editor's guide to bayesian spam filters Posted Feb 23, 2006 4:10 UTC (Thu) by FraserCampbell (guest, #33142) [Link] Greylisting is fantastic. I enabled greylisting on February 6th, since enablement I have averaged 2.88 spams per day. Prior to enablement I had averaged 121 pieces of spam per day (just looked at month of January).
So I get a 98% reduction in my spam and a pittance coming through, I don't have to hunt through a spam folder to see if things were missed and all my normal mails and mailing list traffic is totally unimpacted (I'm using postgrey's default of 30 day storage for ip/from/rcpt triplet).
I never liked the solutions that filter mail to folders (or /dev/null) because there are inevitably mistakes ... at least I always found mistakes (false positives). I don't recall all the different filters I've tried over the years but bogofilter was one of them. Perhaps things have improved but with greylisting so incredibly effective at this point I'll leave it to others to worry abou the spam problem.
The Grumpy Editor's guide to bayesian spam filters Posted Feb 23, 2006 8:46 UTC (Thu) by tzafrir (subscriber, #11501) [Link] It's only a matter of time until spammers learn to retry three times. And then we get even more useless spam traffic.
The Grumpy Editor's guide to bayesian spam filters Posted Feb 23, 2006 12:47 UTC (Thu) by shane (subscriber, #3335) [Link] It's only a matter of time until spammers learn to retry three times.Sure, but until then you drop 90% of spam without having to filter. Even if spammers do start to retry 3 times, they can be quickly identified unless the sender keeps state about the sessions, and retries only after a retry message from the server. For a spammer this means increased memory/disk useage, and more complicated spambots. Plus compromised hosts (which send a lot of spam) tend to get discovered, and taken off-line. If this happens before the re-send the mail never gets delivered. In theory there is no difference between theory in practice. But in practice, there is.The bottom line is that today, it works great.
The Grumpy Editor's guide to bayesian spam filters Posted Feb 24, 2006 1:06 UTC (Fri) by njhurst (guest, #6022) [Link] And when they are resending 3 times there is a high probability that they hit one of your honeypots. (SPEWS)
Actually they don't bother with state Posted Feb 24, 2006 2:32 UTC (Fri) by AnswerGuy (guest, #1256) [Link] ... they just retry everything three or four times at five minute intervals.
It's only an incremental extra effort to deliver multiple times regardless of whether the earlier copies reached you or not.
I get over a 1000 slices of spam per day through my personal inbox ... even after greylisting and some blacklisting (very limited blacklisting). SpamAssassing gets over 99% of that, but I still end up with 20 or so, per day, that make their way through SA. Often I get about 5 copies of any that do get through.
Periodically I go through the spam folder (and one "longlines" folder which contains spam with very long lines --- basically no standards compliant linefeeds at all, which have broken my procmail recipies for SA and YAVR in the past). (YAVR is an anti-virus recipe since SA normally doesn't catch those as "spam" per se).
I've only noticed a tiny handful of false positives dumped into my spam folder by SA. (Less than a half dozen in almost two years). This is a very unscientific measure since I am pretty heavy handed with the delete key when I do spot check the spam folder; and I'm only human with a limited about of time and energy to spend on rescuing mail sent by strangers who's content looks too "spammy."
I don't keep metrics on rejections ... nor on greylisting delivery deferrals that never get delivered. I only have one confirmed case of a bit of e-mail that was not spam but which got greylisted for 49000 seconds (wife was attempting a PayPal password change and their mail server didn't respect the conventional retry delays --- the Postfix greylisting daemon we're using punishes apparently attackers with an exponential back off). (She resolved that by simply whitelisting them and forcing another PW change; while also teaching the silly tech support person there all about proper MTA behavior, and greylisting over the phone).
So, greylisting helps a little ... but too many spammers have adapted and now simply, blindly try everything multiple times. (Also anyone that does successfully dump their spam on an open relay gets the delivery retries for free ... alll standards compliant MTAs do that, and most open relays are just old, unpatched, poorly configured copies of sendmail).
(I do NOT use ORBS type blocking ... I refuse to configure my MTAs to implement a set of dynamically changing policies that are set by strangers ... so I only add my own connection blocking sparingly ... so far).
The main thing that seems to limit my spam load is my own paltry bandwidth. My mailservers share bandwidth with DNS and web traffic (from my clients and servers) over a little old 144Kbps IDSL line. I have about the equivalent of two bonded 56K leased lines. Apparently a significant number of spam cannons time out and drop connections on such a slow link (they've got millions of other targets to get to).
JimD
reject on black, filter on grey Posted Feb 24, 2006 13:53 UTC (Fri) by copsewood (subscriber, #199) [Link] While greylisting makes sense for the reasons other contributors suggest, you might spend less time and have fewer manual false discardsgoing through your spam folder if you reject more of the very high probability spam at the MTA level. Checking manually what went into the spam folder is quicker and better on doubtful messages only or there will be too many in the spam folder to do this job accurately.
Also occasional genuine messages getting rejected by the MTA that accepts mail from across admin boundaries will result in a bounce to the sender, while not sending bounces to innocent victims, e.g. which happens if you reject at an internal incoming MTA. I set spamassassin score > 10 to MTA reject and > 7.5 to go to my spam folder via procmail. I also use the spamhaus DNSBL which currently rejects about 900 spams a week on my server and with which I have never seen a single false positive in about 2 years use, and use less reliable DNSBLs e.g. spews for spam folder filtering.
The Grumpy Editor's guide to bayesian spam filters Posted Feb 26, 2006 15:09 UTC (Sun) by job (guest, #670) [Link] If you'd actually read Grahams paper you'd know that Bayesian auto classification of mail is a lot older than that. What Graham did was point out that many of the existing attempts had been "too smart" and that results could actually be more accurate when not converting character sets, cleaning up headers etc. since their very existence adds to the amount of classification information.
So while not being in any way groundbreaking, he pointed out some interesting points that really re-ignited the idea. That, and that is was probably a good time when many people was unable to manually cope with all spam, was what makes people refer to the paper so often.
|
|||||||||||||