E-mail filters not fooled by signed spam (News.com) [LWN.net]

E-mail filters not fooled by signed spam (News.com)

Posted Oct 11, 2003 22:13 UTC (Sat) by arcticwolf (guest, #8341) [Link]

Actually, if you really want a good spamfilter, try bogofilter (http://bogofilter.sf.net), and train it a lot, with lots of ham and spam. I've been using it for almost a year, and I can count both the number of false positives as well as the number of false negatives I had in the last three months on one hand each, while getting around 40 real mails and 70 pieces of spam each day.

No problem with people sending html mail, either (even those who send *only* html and no plain text), no need for kludges like "if it contains words like penis, it's (probably) spam", it also sorts out worm emails and the like, and it trains itself while categorizing mail (so I only have to interact when it does something wrong).

I can only recommend giving it a try. It needs an initial training period to give good results, and will take a while until it gives great results, but it's worth it.

E-mail filters not fooled by signed spam (News.com)

Posted Oct 12, 2003 17:40 UTC (Sun) by RobSeace (subscriber, #4435) [Link]

Indeed Bayesian filters are definitely the way to go... I use bogofilter at
work, and SpamBayes at home, and both work wonderfully... I see almost NO
spam at all anymore... And, the only things that have ever gotten tagged
as spam that I actually wanted to see have been spammy-looking messages from
businesses which I've ordered things from, or certain spammy-looking mailing
list posts, etc... And, after a bit of retraining, all is well even there...
(In the case of a couple mailing lists, I set up explicit pass-throughs for
the addresses in my ".procmailrc", to just skip the filtering completely,
and always let them through... But, I'm sure if I retrained enough, I
wouldn't have even needed to bother with that...) That's the only downside
to Bayesian filters: the initial training time you have to put in before
they become fully effective... But, it's definitely more than worth it...
Because, once you're over that initial hump, they just train themselves, and
you don't have to do much of anything, other than correct the very rare
mistakes it makes... The main problem I've found is people who don't save
all their legit E-mail, so they don't have a good sized corpus of legit
mail to train it on... (You can get large amounts of spam from several
sources, and all spam is pretty much alike, so no need for that to be
personalized... But, legit mail really does differ quite a bit from person
to person...) In that case, they have to just train as they go for a
while, until enough messages have been received... (Or, you CAN start them
off with an initial database trained from someone else's legit E-mail,
which I've found does work relatively ok, for the most part... But, it
definitely requires a bit of tweaking work, and is far more likely to lead
to some false-positives at least early on... But, it's probably better
than starting from scratch, at least from the user's perspective, as it'll
wipe out most of the spam, right from the start...)

E-mail filters not fooled by signed spam (News.com)

Posted Oct 16, 2003 4:59 UTC (Thu) by arcticwolf (guest, #8341) [Link]

You're right, the initial training period needed is a bit of a problem with bayesian approaches. However, I think it hardly can be avoided; the reason why a bayesian filter actually works well, after all, is that it learns to distinguish between what the *user* considers spam and what he/she/shi considers legitimate email. And that - obviously - means that pretraining is not possible; if you started distributing tools like bogofilter, for example, with premade token databases, then you'd just create another weak link in the chain that spammers could attack, similar to SpamAssassin rules etc.

Getting an initial database from a friend might work; however, I, personally, would be reluctant to give anyone my token databases. Maybe it's just paranoia, but I prefur to keep them just as "secret" as my email.

It might be an idea, maybe, to use a distributed token database instead of per-user ones (P2P-based?), but I personally do not think this would work: it not only would allow spammers to pollute the database, it would also take away the individuality of users' databases that actually makes the bayesian filtering approach more effective.

The best way to train a bayesian filter is probably to just grit one's teeth and do it the hard way - put up with the spam and manually classify it until the filter starts working reasonably well, or - if you get too much spam to do this - use a tool like SpamAssassin to create an initial token database.

As far as setting up procmail rules to bypass filtering for messages you know will get misclassified is concerned - that works, of course, but the more elegant approach is still to train the filter, and I am happy to be able to say that it has worked in my case, too. I have one friend whose emails were notorious for being classified as spam; since he rarely ever sends one, the filter didn't get much exposure to them, either, so it wouldn't learn much about them, but by now, it classifies them correctly and leaves me with no known false positives.

It's amazing, really.

E-mail filters not fooled by signed spam (News.com)

Posted Oct 12, 2003 19:32 UTC (Sun) by nix (subscriber, #2304) [Link]

Er, SA's Bayesian algorithm is (an enhancement of) bogofilter's.

I think that any single-method attack is likely to fail; to catch things you really need every method you can find: so body-content heuristics and statistical methods and network checks and header analysis combined will be stronger than any one on its own.

(The immune system uses the same approach; strength in depth.)

E-mail filters not fooled by signed spam (News.com)

Posted Oct 16, 2003 5:14 UTC (Thu) by arcticwolf (guest, #8341) [Link]

Actually, I think that body-content heuristics and header analysis can be viewed as being included in statistical analysis, at least as far as bayesian filtering is concerned. Outside of that, I agree that having both depth and breadth in your approach to spam is a good thing; but for now, bayesian filtering (as implemented by bogofilter - I don't have experience with other tools) seems to do the job so well that there's no need to worry, and with the filter training itself automatically as it classifies messages, only requiring user interaction for false positives or negatives, it seems that there is little that spammers can do, either.

In fact, more or less the only approach I can think of right now would be to change spam characteristics so drastically that the (bayesian) filters wouldn't catch them anymore; however, this would require not only a concerted action in which most spammers participate (otherwise, only a few pieces of spam would get through), it would also be effective only for a very short amount of time, until the filters' token databases have been updated.

What else could a spammer do? Try to make messages look as much as legitimate email as possible, I assume, but then again, this likely won't be effective - spam is, after all, ultimately about advertising, and a message that does not advertise products anymore in any way does not justify being sent. The filters *will* catch on, and the fact that they are completely dynamical in generation (no static rules) and specific to each user means you can't just attack them.

Or at least that's what common sense tells me. Maybe the future will show that there is a fundamental flaw not only in the existing tools, but in the bayesian approach in general, but I can't see it right now; and even if there is, a better technique will follow. Ultimately, the war against spam can only be won.

(and I probably shouldn't post comments this early in the morning - or, rather, this late at night -; I seem to get a bit overdramatic. oh well.)

HTML mail...

Posted Oct 13, 2003 10:50 UTC (Mon) by eru (subscriber, #2753) [Link]

I believe we should combat HTML e-mail because it's primarily used by spammers. If your friends write you in HTML, please explain them that they are helping spammers.

I agree HTML mail is a bad idea, but getting rid of it is probably hopeless now, thanks to feature-happy e-mail client implementors. In too many mail clients it is the default, sometimes so that the default cannot be easily changed. Especially by technically unsophisticated friends or relatives... And I am not talking only about Outlook in its various incarnations. Some open-source mailers have the same flaw. For example, I have yet to find out how to tell Mozilla 1.0 "no, I don't ever want to send HTML mail to anyone, and if someone sends me one, I want to reply with plain text". Now it seems I have to use "options->format..." every time.

HTML mail...

Posted Oct 13, 2003 15:30 UTC (Mon) by kfiles (subscriber, #11628) [Link]

In addition to setting mozilla's send format to convert to plaintext, I use the following prefs. This combo gets rid of pretty much every case of rich-formatted email, and makes the result as close to Mutt+Emacs as possible.

user_pref("mail.quoted_graphical", false);
// To get rid of the sending window
user_pref("mailnews.show_send_progress", false);
// Change the reply header
// 0 - No Reply-Text
// 1 - <Author> wrote:   - Netscape 3.xx/4.xx style
// 2 - On <date> <author> wrote:
// 3 - user-defined string. Use the prefs below in conjuction with this.
user_pref("mailnews.reply_header_type", 3);
// If you set 3 for the pref above then you may set the following prefs.
// The end result will be <authorwrote><separator><ondate><colon>
user_pref("mailnews.reply_header_authorwrote", "%s wrote");
user_pref("mailnews.reply_header_ondate", "on %s");
user_pref("mailnews.reply_header_separator", " ");
user_pref("mailnews.reply_header_colon", ":");
// This should change attached image and text files from inline to attachment.
user_pref("mail.content_disposition_type", 1);
// To change the color of the quote bar
// Replace #0000A0 with the colour of your choice.
user_pref("mail.citation_color", "#0000A0");
// Format=flowed prefs, RFC 2646
pref("mailnews.send_plaintext_flowed", false);
user_pref("mailnews.display.disable_format_flowed_support", false);
pref("mail.display_struct", true);
pref("mail.send_struct", false);

--kirby

HTML mail...

Posted Oct 13, 2003 15:37 UTC (Mon) by proski (subscriber, #104) [Link]

If you are using Mozilla Mail, perhaps you should try Thunderbird. In Thunderbird, you select Tools->Account Settings->Composition & Addressing->Compose Messages in HTML Format. It should be similar in Mozilla.

As for the default, see bug #115439. I'm sure there are other bugs filed for this issue, it's just the first one I could find.

HTML based mail

Posted Oct 13, 2003 12:57 UTC (Mon) by Duncan (guest, #6647) [Link]

> I believe we should combat HTML e-mail because it's primarily used
> by spammers. If your friends write you in HTML, please explain
> them that they are helping spammers.

Add crackers to that list as well. Anyone using HTML mail is helping spammers
AND CRACKERS. When this comes up, I simply ask folks to consider how many
exploits OE and Outlook proper have had, and how many they WOULD have had,
if they'd stuck to plain text. That should persuade virtually ANYONE (with any
tech knowledge, or who knows how to look it up, anyway).

My top priority rule deletes HTML formatted mail, whether or not it includes a
plain text version also. Yes, that's prioritized ABOVE the whitelist rules, even if
I'm not affected by HTML vulns, I wouldn't accept an offer to shake hands if I'd
seen someone slime their hand with snot before they offered it to me, even if I was
wearing gloves to protect myself, and I'm not going to accept HTML mail,
regardless of my client's vulnerability to it, for the same reason. If they want to
shake hands, they can learn not to be disrespectful of my health in the process. If
they want to exchange messages, they need to respect the health of my computer as
well. To do otherwise is simply rude, and they can just go be rude to someone else.

"Security hole"?

Posted Oct 12, 2003 1:58 UTC (Sun) by Jilks (guest, #12291) [Link]

Since Outlook. ;-)

"Security hole"?

Posted Oct 12, 2003 14:44 UTC (Sun) by TwoTimeGrime (guest, #11688) [Link]

I wish people would stop posting this FUD about Outlook. Outlook is very secure once you install the patches. You do install the patches on *your* computer don't you? Or do you just install your software off a CD and hope that everything is secure forever?

"Security hole"?

Posted Oct 12, 2003 16:29 UTC (Sun) by dsime (guest, #5764) [Link]

Guess again.
Outlook will execute arbitrary commands, arbitrary to me but not to the sender, when the only action I take is to open the note.

In point-of-fact I don't even have to do that as the default configuration for Outlook is to display the inbox in such a way that it opens notes so you can see the "first few lines", just by having them highlighted on the list.
And the first one is always highlighted.

So in order for Lookout to execute arbitrary code all I have to do is start it.

THAT I would not think would classify as secure in anybody's book.

"Security hole"?

Posted Oct 12, 2003 16:43 UTC (Sun) by TwoTimeGrime (guest, #11688) [Link]

Care to tell us what these arbitrary commands are? I've had no problems with Outlook wanting to execute attachments or run code since I've installed the available patches.

"Security hole"?

Posted Oct 13, 2003 7:45 UTC (Mon) by diegor (guest, #1967) [Link]

It's a old Outlook express bugs. I don't know exactly where is fixed. BTW:
the trick is that outlook was configured in highly insecure way, so the autopreview open any attachment that looks like a image.

So if you made an attachment that is a executable, named pippo.gif.exe (to fool the user) and mime type 'image/gif', outlook open it using run32.dll.

But run32dll it recognizes that it is an executable, and run it. Be happy, the nice autopreview feature have installed a new virus, even if you haven't clicked on the image....

The problem with outlook (and many other office program) is that until now they are not designed with security in mind.

Regards,
Diego.

"Security hole"?

Posted Oct 12, 2003 22:49 UTC (Sun) by Ross (subscriber, #4065) [Link]

FUD?! First of all I don't use Outlook so I don't think it need to patch
it.

Second of all it is full of problems. I remember when the only email
"viruses" were fake warnings and (remember GOODTIMES?). I told users not
to worry because viruses only spread by infecting executable code and only
got a chance to run when you ran executable code which might be infected,
text like their email was certainly not a problem. The funny thing being
that users didn't realize it but _they_ were the ones spreading the "virus"
(the fake warning), not their email programs.

Then along came Microsoft with the great idea of making everything into
code. Macros for every file! Their idea was that reading email should
be the same as executing it. Random instructions from a stranger should
be executed by clicking to read a message. Kinda like their idea that
opening a spreadsheet or a word processing document should mean that you
exectute it. Great idea guys. My overflowing mailbox does not thank you.

Users don't have to click an attachment to run it. Outlook will do it
for them _automatically_ just to "help out". Unless recent patches
finally disable that (that feature has been there for years so I'd be
surprised is MS suddenly changed their mind) I don't think you are correct.

"Security hole"?

Posted Oct 13, 2003 2:38 UTC (Mon) by TwoTimeGrime (guest, #11688) [Link]

> First of all I don't use Outlook so I don't think it need to patch
> it.

Then you're probably not qualified to talk about it. I do use Outlook. All of the issues that you cited in your post have been addressed in the security patches that Microsoft has released for it. If you install them then you don't have those problems with marco execution or being able to run executable files from within Outlook unless you specifically disable the security checks.

> Users don't have to click an attachment to run it. Outlook will do it
> for them _automatically_ just to "help out". Unless recent patches
> finally disable that

Recent as of September 2000, yes.

I know Microsoft is an easy target and drags their feet on patches until someone embarasses them into releasing them, but get your facts straight before you reply. What you should be doing is encouraging users of older copies of Outlook to install the latest patches on their system. Newer versions have the patches incorporated. If the security checks have been disabled then it's been done by the system administrator.

We could make the same arguments about people running older vulnerable versions of Apache or Sendmail or SSH but you'd tell those people that they should patch their systems. Yet when it comes to Outlook you'd rather point and hiss at it and Microsoft rather than the user who isn't patching their system or is administrating it poorly.

RE: I.E. "Security hole"?

Posted Oct 13, 2003 17:04 UTC (Mon) by scripter (subscriber, #2654) [Link]

I advise my relatives not use use outlook, period. I tell them to use Mozilla Mail instead. Why? Basic risk assessment:

1. Outlook is one of the most common email clients, thus it is a better target for exploits. Running a lesser known, or lesser used email client generally translates to less risk.
2. Outlook has a track record of serious security problems. I'm sure more will be found. Other email cleints have had problems, but not as high-profile, and not as damaging (partly due to #1).
3. Relatives like the pretty look of Mozilla Mail.
4. Mozilla keeps its email in a standard format, not a proprietary format.
5. Mozilla imports outlook email and the outlook address book, so switching is easy.
6. Mozilla isn't just an email client, it is also a web browser. Why not replace their web browser with something far more secure than I.E.? (I.E. continues to have a horrible track record for security)

Despite all of the above, most relatives prefer to keep using Outlook and Internet Explorer. Why?

1. It's what they know.
2. It's fast.
3. It's what their friends use.
4. They don't have to install something new.

So, I tell them how to install patches. Do they do it? No. Why?

1. They can't remember how to do it.
2. They don't want to tie up the phone while downloading updates. (Fortunately, most of them have dial-up instead of always-on connections, which limits their vulnerability to some degree).
3. They forget to do it.
4. They are lazy. It works, doesn't it?

RE: I.E. "Security hole"?

Posted Oct 13, 2003 17:46 UTC (Mon) by TwoTimeGrime (guest, #11688) [Link]

> Despite all of the above, most relatives prefer to keep using Outlook and
> Internet Explorer. Why?

Do you mena Outlook Express? Outlook is a completly different product than Outlook Express for which there is currently no open-source equivilent on Windows.

Outlook vs Outloook Express

Posted Oct 13, 2003 20:43 UTC (Mon) by pflugstad (subscriber, #224) [Link]

Please correct me if I'm wrong, but Outlook is the product that comes with Office, correct? It costs $$$$.

And Outlook *Express* is the free one. And the one that has all the bug, holes, exploit of the week.

You can praise Outlook all you want, but Outlook *Express* is what the vast majority if users use, and it's a pile of crap. And Outlook itself, while it may be "safe" with all the patches applied - how many people got nailed by viruses even when the patch has been available for months. Fact is, people don't patch, so even if Outlook is safe, it's not for the vast majority of people who actually use it instead of Outlook Express.

So, either pay $$$ for Outlook, or since Outlook itself has had a large number of security holes, use something like Mozilla Mail. As a side benefit, use Mozilla instead of IE and stop all the spyware from being automagically downloaded onto your computer via ActiveX, DCOM and all the other idiocies M$ has foisted off on us as "useful".

Outlook vs Outloook Express

Posted Oct 13, 2003 21:44 UTC (Mon) by dlang (subscriber, #313) [Link]

Outlook (the full version) has had another hole found in it within the last month (Ok, technicly it was a hole in IE as used by outlook to process mail, but since you have no other choice it made users of Outlook vunerable), so it may be safe if fully patched, but you had better keep checking becouse next week it may have another hole discovered with no patch for it yet.

"Security hole"?

Posted Oct 13, 2003 2:46 UTC (Mon) by proski (subscriber, #104) [Link]

Please, don't take so seriously comments ending with a smile. It was a joke, maybe not a good joke, but let's not make it a discussion about Outlook.

Passing a message is not a security hole in the usual sense, but spamming everybody in a large company with a "signed" message can be just as costly as defacing the webserver.