Weekly Edition
Return to the Security page
| |||||||||||
82% of email is spam
According to this
eSecurity Planet article, 82% of all email which was sent in April was
spam. That is the highest level ever measured - so far. Informal
measurements here at LWN suggest that the 82% figure could even be a little
low; some of our accounts here are receiving well over 1200 spams per day.
The cost of this endless stream of garbage is eventually going to push some part of the system to the breaking point. And the results may not be good. As the spam problem gets worse, most email users will be willing to accept almost anything from their ISPs or legislators which promises to improve things. It would not be surprising to see power grabs coming from several directions as the usual cynical forces try to take advantage of the situation. Are we ready for a world of centralized email systems, proprietary protocols which limit bulk mailing to "authorized" merchants, and new laws giving governments power to monitor and restrict email content? If we're not ready for those things, we're going to have to think again about how to fight this problem. Filtering can be highly effective, but it does little for many of the costs of spam, including bandwidth usage and compromised servers. Filtering also does not work for all users. Somehow, a way must be found to keep spammers and their output off the net. If we can't come up with a way to do that which preserves the freedoms that have made the net what it is, we're likely to see rather less palatable attempted solutions imposed by others. (Log in to post comments)
Solution Posted May 5, 2004 20:57 UTC (Wed) by yodermk (subscriber, #3803) [Link] The solution, of course, is Dan Bernstein's IM2000. Read it. Be happy.http://cr.yp.to/im2000.html I've posted this on Slashdot a few times, and whiners always come back with reasons why it won't work. I have, with very little difficulty, been able to think of a reasonable answer to every objection. So before you gripe here about why it won't work, please spend a few minutes yourself trying to think of a way to get around your supposed problem. The biggest problem by far is getting people to switch mail systems. But if we do nothing, spam will keep getting worse. IM2000, along with a good blacklist system (which would work MUCH better than blacklists with SMTP), should go a long way to stopping spam. No centralized mail servers, certificates, or government intervention required. Last I checked a few months ago, there were some good proposals swimming around on the Net. Need to look again for progress. Unfortunately, I don't know of an implementation yet, but it shouldn't be *that* hard. Once we get that, I bet people will start switching. Then as they drop SMTP accounts, others will need to switch to IM2000 to communicate with them. :)
Problem with your 'Solution' Posted May 5, 2004 21:42 UTC (Wed) by lakeland (subscriber, #1157) [Link] Ok, I've spent your few minutes thinking of flaws...A large percentage of spam is sent by zombies. Your solution would enable these zombies to continue spamming as before. I also wonder: if the spammer can afford the bandwidth of delivering each message to the ISP of the user, why it would be so much more expensive delivering them directly to the users. After all, the spammer could trivially manage to only store one copy of the message on their ISP. Though I admit I haven't thought this one through so carefully.
Problem with your 'Solution' Posted May 5, 2004 23:05 UTC (Wed) by yodermk (subscriber, #3803) [Link] First, I said to spend a few minutes thinking of a solution for any flaws before mentioning them, not spend a few minutes thinking of flaws...> A large percentage of spam is sent by zombies. Your solution would enable First, what do you mean by zombies? I know (roughly) the tactics of spammers, but I'm not familiar with which one(s) the term "zombie" applies to. But no, I don't believe you're correct that it would enable them to keep spamming as before. Here's how I understand it: Say a user of an ISP sends a million spams. All million (short) notification messages go out to the clients. BUT, someone at the ISP would almost certainly get some clue rather quickly that a spam was sent. *ZAP* and the message is gone from the ISP's server, and whoever hadn't checked their mail yet will be fortunate enough not to see it or waste bandwidth downloading it! Also, all mail will need to go through some permanantly connected mail server. Dialup certainly won't work. A cable modem/DSL mail server might work, but would possibly be less reliable. But virtual servers are cheap enough that anyone who really needs their own mail server should be able to afford one. Blacklisting as I understand it in the IM2000 model: If joe@isp.com sends a spam, his particular address is added to the blacklist, which would probably be a "push" type deal. Eventually it would trickle down to all the clients. (Ok, a "pull" system might be better, where it queries for every email, that's up for debate.) If, say, three or more users of a certain ISP or domain name or maybe subnet send spam, that entire network goes onto the blacklist. It probably won't stop 100% of spam, but I don't think our dear grumpy editor would be receiving 1200 spams a day with this protocol. I can also see how it could cause some minor inconvenience for people in some circumstances, say, those in the middle of nowhere with spotty satellite connections. But, I'm 100% sure that it can be made to work somehow, even if it looses a feature or two of this setup. Above all, please ask yourself this: Is any conceivable disadvantage to this system anywhere near as big a problem as 1200 spams a day??? > After all, the spammer could Yeah, think you need to read and think about this method more. :-) Actually one of the great things about this system is that it allows only one master copy to be stored on the ISP. For legitimate bulk mail, such as mailing lists, that's a rather significant benefit.
Problem with your 'Solution' Posted May 5, 2004 23:29 UTC (Wed) by dlang (subscriber, #313) [Link] a zombie is a users machine that has been taken over by the bad guys.get a few thousand machines controlled by worms to each send a few dozen messages and you have a system that won't trigger any alarms at any one ISP. also what happens to mailing lists? I have a low bandwidth DSL line to my mail server (144k IDSL, the only thing I can get to my location) if I sendd a message to the linux-kernel mailing list do I now have several tens of thousands of people trying to download the message over my slow link? what happens if the senders server is down or unreachable when I want to read the message? this idea would work in a world where everyone has pleanty of bandwidth and storage and everything is always up, but in the real world it's little better then a dream to toy with. some good may come of it, but this is nowhere near being a deployable system.
Problem with your 'Solution' Posted May 6, 2004 2:13 UTC (Thu) by rjw (guest, #10415) [Link] Do you have an ISP?You get them to store the mail. Normal caching strategies also work. The mail you send would be stored in the same place the mail you receive is stored today.
You need to allow for people running there own servers Posted May 6, 2004 4:30 UTC (Thu) by alex (subscriber, #1355) [Link] You don't always want an ISP to process your mail. I run a mail server of my cable modem because I host several domains and mailing lists. Its cheaper to use my system at home where I have control rather than pay an ISP for stuff I can do myself.
You need to allow for people running there own servers Posted May 6, 2004 5:51 UTC (Thu) by copsewood (subscriber, #199) [Link] Unfortunately for this kind of mail system, one of the most effective means of blocking spam is to see if it comes directly from the kind of d IP address generally allocated to broadband users, by doing a PTR DNS lookup on the IP and looking for distinctive patters ( e.g. 4 numbers between . Addresses in this category are already getting into a number of DNS blacklists, and if more people get sufficiently fed up with spam to adopt this solution, legitimate mail systems hanging on this kind of connection without using a smart host are likely to become collateral damage. You would avoid this by configuring your ISPs outgoing SMTP server as your outgoing smart host, which would avoid this problem, and make better use of your limited bandwidth (only 1 copy of each list message uses your uplink).
You need to allow for people running there own servers Posted May 6, 2004 17:21 UTC (Thu) by yodermk (subscriber, #3803) [Link] Get a virtual server to run your mail server. More expensive than running it over a cable modem? Yeah, but you can afford it. And as another reply said, running SMTP over a cable modem isn't reliable anyway, since they tend to get blacklisted.
Problem with your 'Solution' Posted May 6, 2004 13:08 UTC (Thu) by shapr (subscriber, #9077) [Link] These are actually part of the advantages of this system.If a zombie sends notifications, it must be at the same hostname or IP for them to be picked up. that also means that blacklists become much more effective when a server is 'accountable' for its actions. As for mailing lists, I think the list host would pick up and then host the mail, allowing for pushed spam. But then this isn't a silver bullet, just an improvement. Right, once a mail is in the system it's trusted and pushed. The cost of storing and delivering mails is on the system, not the sender. If you move those costs to the sender, spam becomes less economically viable. The essence is that you and I pay for spam in a push system like we have now, and we only pay for notifications in im2000. One major advantage here is that those most likely to respond to spam, namely Internet newbies who check their mail once a week, are much less likely to get spam, since geeks like us will have gotten a spammed notify, and had time to do something about it (update blacklist, remove virus, etc). Anyway, that's just my take on how to make spam less profitable for the spammers.
Problem with your 'Solution' Posted May 6, 2004 17:19 UTC (Thu) by yodermk (subscriber, #3803) [Link] Ok, by that definition of zombie, IM2000 would mostly stop it, if not totally. In most cases, it would have to provide the IP address of the compromised box in order to "work", and the box would have to run a mail server, and not have a firewall. That kind of thing would be easy to detect, but of course I suppose a few idiots will let it go on unnoticed. And a compromised box IP would likely be blacklisted quickly.With the distributed blacklist (assuming it is administered in a trustworthy manner) would almost certainly make the cost to spam way too high. As for mailing lists, the protocol changes somewhat drastically. See Bernstein's site, he talks about it. > what happens if the senders server is down or unreachable when I want to read the message? That is the second biggest problem with the system, after transition difficulties. Basically, someone (your ISP or you) needs to run a reliable mail server. That's a minor disadvantage perhaps, but keep in mind how it compares to 1200 spams a day. > this idea would work in a world where everyone has pleanty of bandwidth and storage and everything is always up, but in the real world it's little better then a dream to toy with. some good may come of it, but this is nowhere near being a deployable system. I really don't see how you can come to that conclusion. Of course it's deployable. It doesn't even use more bandwidth and storage in the long run. It will use much less bandwidth because of less spam. And storage would be less because it would only store one copy of an outgoing message to many users. Receiver-side storage would only be on his local computer, not on the server. (Ok, an IMAP-like mode could change that.)
Problem with your 'Solution' Posted May 6, 2004 6:25 UTC (Thu) by pizza (subscriber, #46) [Link] >Say a user of an ISP sends a million spams. All million (short) notification messages go out to the clients. BUT, someone at the ISP would almost certainly get some clue rather quickly that a spam was sent. *ZAP* and the message is gone from the ISP's server, and whoever hadn't checked their mail yet will be fortunate enough not to see it or waste bandwidth downloading it!Except you are forgetting one very important point. The majority of these spams now contain unique subjects and/or bodies, be it via random dictionary words or whatever. So much to only one copy, eh?
Problem with your 'Solution' Posted May 6, 2004 17:26 UTC (Thu) by yodermk (subscriber, #3803) [Link] 1. These "unique" subjects are only to get around mail filters with SMTP. I don't think those type of filters will be as necessary with IM2000.2. In the long run it's no real disadvantage over SMTP even if you *do* have to store a separate copy for each recipient. And, if joe@isp.com sends a million distinct messages, some flag would still be raised, and it would be easy for an admin to nuke all his messages. Heck, you could set a space limit for outgoing mail ... maybe 100MB or so (up to the ISP of course). That should eliminate a million distinct messages.
Problem with your 'Solution' Posted May 6, 2004 7:20 UTC (Thu) by fergal (subscriber, #602) [Link] Say a user of an ISP sends a million spams. All million (short) notification messages go out to the clients. BUT, someone at the ISP would almost certainly get some clue rather quickly that a spam was sent. *ZAP* and the message is gone from the ISP's server, and whoever hadn't checked their mail yet will be fortunate enough not to see it or waste bandwidth downloading it!Your faith in the attentiveness of ISPs is a little misplaced. Why do you think ISPs will keep their spam spotting software right up to date when even today we still encounter the trivially fixable problem of open relays?
Problem with your 'Solution' Posted May 6, 2004 17:28 UTC (Thu) by yodermk (subscriber, #3803) [Link] > Why do you think ISPs will keep their spam spotting software right up to date when even today we still encounter the trivially fixable problem of open relays?Because they'd enter blacklists if they don't.
Problem with your 'Solution' Posted May 7, 2004 2:19 UTC (Fri) by fergal (subscriber, #602) [Link] Are you suggesting we would have blacklists for ISPs that don't keep their spam filters up to date?How exactly would you keep that blacklist up to date? Wouldn't you have to aim spam probes at various ISPs and see if they block them. Good luck with that project.
Problem with your 'Solution' Posted May 17, 2004 5:50 UTC (Mon) by coolian (guest, #14818) [Link] The original "IM2000" guy is obviously a hard-core democrat. I'm no Republican, but this "legislate my butt out of existence" and "trust the ISP not to go down" thing is WAY too much trust for me.
Problem with your 'Solution' Posted May 6, 2004 9:55 UTC (Thu) by mmarsh (subscriber, #17029) [Link] >First, I said to spend a few minutes thinking of a solution for any>flaws before mentioning them, not spend a few minutes thinking of flaws... That's an unfair request. If it's all been thought of and debated before, provide a URL or something. You're making a proposal, so it's up to you to defend it. A capsule description provides insufficient detail to support a position or adequately describe a system. >Say a user of an ISP sends a million spams. All million (short) How is this different than a spam where the body is essentially just an image tag (or a redirect) to an advertisement on a remote server? Since notification latency will vary from recipient to recipient and not everyone checks email immediately on arrival, there might never be a noticeable spike. Combine that with an ISP that just doesn't care about bandwidth usage, and you're really no better off than when you started. >Also, all mail will need to go through some permanantly connected mail So email availability is dependant on the reliability of the sender? If the sender's ISP has to keep track of whether all of the recipients have retrieved the message, how does that affect forwarding? Will mail that gets forwarded be retrievable at all? Here I don't mean A sends a message to B who then forwards it to C. That's a no-brainer: B has to store it. I mean A sends a message to B at foo.com, but B has a .forward file passing it along to bar.com. I've had forwarding chains of about four hops, and I suspect there are many people with longer chains. Does each hop have to cache the message? What about messages of the type "Our server will be down for maintenance on Thursday"? If a recipient doesn't check mail on Wednesday, he'll see that there's a message waiting from him, possibly flagged as important, but he won't be able to retrieve it until it's no longer relevant.
Problem with your 'Solution' Posted May 6, 2004 17:37 UTC (Thu) by yodermk (subscriber, #3803) [Link] Dan Bernstein made the proposal, not me. Granted, his page isn't that detailed. If you Google for "IM2000" you can find a few other more specific proposals.> How is this different than a spam where the body is essentially just an image tag (or a redirect) to an advertisement on a remote server? That would still end up as an email in someone's box. An IM2000 spam would have a subject, but may be unavailable upon mail check. Even though that type of spam is "short", it's still far better that it never reaches the recipient, to make spamming less profitable. > Combine that with an ISP that just doesn't care about bandwidth usage I do think it would be detectable, and any ISP that simply didn't care would be blacklisted. > but B has a .forward file passing it along to bar.com. I've had forwarding chains of about four hops, and I suspect there are many people with longer chains. Does each hop have to cache the message? Couple possibilities ... 1) each hop caches the message, 2) each hop notifies the originating ISP of the new recipient. That may have privacy concerns though. > What about messages of the type "Our server will be down for maintenance on Thursday"? That would be something to think about. This really does depend on mail servers being reliable. But, for the most part, they are.
Problem with your 'Solution' Posted May 17, 2004 5:52 UTC (Mon) by coolian (guest, #14818) [Link] " Dan Bernstein made the proposal, not me."But you suggested it and said it would "of course" be the answer. If you don't have any actual knowledge of it, then how could you know it's the answer? Jesus, you must be 12.
Problem with your 'Solution' Posted May 17, 2004 5:45 UTC (Mon) by coolian (guest, #14818) [Link] It's idiotic. You said the answer was "of course" this guy...It wouldn't work. Why? Because it wouldn't.I think the only thing that will work without legislation is filtering/whitelist/challenge-response.
Why does everyone bring this up every time? Posted May 6, 2004 10:24 UTC (Thu) by Ross (subscriber, #4065) [Link] I haven't read an article about spam in the last 6 months where therewasn't a post extolling the virtues of DB's proposal. And every time people point out major reasons why it isn't going to work. Those issues go unresolved and yet when I read the next article...
Why does everyone bring this up every time? Posted May 6, 2004 17:41 UTC (Thu) by yodermk (subscriber, #3803) [Link] There's a difference between "isn't going to work" and a few minor disadvantages. The way I see it, we have a choice:1. Regulating all mail servers with government intevention Which is least evil?
Why does everyone bring this up every time? Posted May 17, 2004 5:58 UTC (Mon) by coolian (guest, #14818) [Link] 1. Regulating all mail servers with government intevention>> Oh yeah, I don't think so. 2. Using only centralized mail servers >> Meaning what? 3. Using certificates for mail servers, with each server needing to pay out the arse for a certificate >> Why? Do you understand how certs work? 4. IM2000, which has a couple minor disadvantage for a few uses, and which can be worked around >> Minor disadvantages? The whole system not working is more than minor. There are MAJOR unresolved issues with it, that you have zero answers for. They can be worked around? By who? You?
>> This is the least evil of the ones you list. Sorry, but most would agree. I don't want people like you, who have zero answers, running my mail. Thanks, but no.
Solution Posted May 6, 2004 13:19 UTC (Thu) by melauer (guest, #2438) [Link] >The solution, of course, is Dan Bernstein's IM2000. Read it. Be happy.> >http://cr.yp.to/im2000.html I'm not convinced by this "solution". Those "brief notification" messages must either contain a little information, like a subject line, or people will have to click on them not knowing what they're going to get. Either way, these will become the new method of delivering spam. Then we'd be back where we started from. If a spammer runs their own mail server, we'd need to blacklist it. If there's an open relay out there for "brief notification" messages, it would have to be closed so spammers don't send fake ones. And so on. The proposed method would cut down on the total bandwidth used by e-mail, though. That's kind of nice. Incidentally, this system has basically already been implemented. Any number of private web forums use this. When one forum member sends a private message to another, the recipient gets a "brief notification" in the form on an e-mail. Then they login using a link provided in the e-mail and view the contents of the private message, which is of course stored on the forum's server.
Solution Posted May 6, 2004 17:46 UTC (Thu) by yodermk (subscriber, #3803) [Link] > Those "brief notification" messages must either contain a little information, like a subject line, or people will have to click on them not knowing what they're going to get.I'd suggest they contain a subject line and sender name. Any obvious spam would then not need to be transmitted over the Net at all, at least to users who care, which would take care of a lot right there! > Either way, these will become the new method of delivering spam. But remember that it would be much easier for a responsible ISP to stop this before the worst part of the problem than it is under SMTP. If an ISP detects spam, it deletes the message at the source before most people have downloaded it. If a customer's computer is spamming with its own IM2000 server, it could simply block the receiving port to that IP. Under SMTP, if it was caught at any point after the spam was sent, it's too late. > If there's an open relay out there for "brief notification" messages, it would have to be closed so spammers don't send fake ones. It would be impossible to send fake notification messages, because the end-user's box needs the IP of the server from which to fetch the mail!
Solution Posted May 7, 2004 15:02 UTC (Fri) by brouhaha (subscriber, #1698) [Link] OK, so I've spent a few minutes thinking about problems with IM2000 in terms of spam avoidance, and trying to think up a solution.The problem I see is that this doesn't address spam at all. It just changes the delivery mechanism. Today, the spammers bombard your MTA (e.g., Sendmail, Postfix, or Exchange) with spam. Your MTA uses a lot of resources (CPU, memory, disk) dealing with this. Depending on the MTA and configuration, it may or may not be able to do some filtering to avoid actually storing some of the spam in your mail queue. Then you check your inbox using an MUA (mail client such as Evolution, Mozilla Mail, Eudora, or Outlook). The MUA may also do some filtering, but at the very least has to inspect the headers of each message the MTA has received for you. With IM2000, the spammer's machine doesn't directly send the spam to you. Instead, it tells your MTA (or whatever the IM2000 equivalent of an MTA is) that there is mail waiting for you on the spammer's machine. So now instead of getting 1200 spam emails a day, you get 1200 email waiting notifications from random machines all over the internet, many of which are probably zombies. Ultimately your software doesn't have any better way to tell which are spam than in today's architecture; it will simply have to poll each of the 1200 sender mail servers to get the mail headers and try to filter them. This has *perhaps* reduced the internet backbone bandwidth consumed somewhat, if the filtering can be done with inspection of headers only and not the full message body, but it has not solved the spam problem. In fact, given that the IM2000 model provides for the sender's mail server to repeat the "mail available" notifications to the receiver, it may not even affect the bandwidth consumption. And it certainly does not improve the bandwidth consumption on the user's local link. As far as I can see, IM2000 is a solution to a non-problem. It has attempted to solve the problem of reducing the amount of disk space the recipient needs to store the email, but disk space is the least significant problem associated with spam. IM2000 appears to solve the "mailing list problem", except that there isn't really a "mailing list problem" either.
Part of the solution: laws that work Posted May 5, 2004 21:11 UTC (Wed) by dwheeler (subscriber, #1216) [Link] Part of the solution will need to be laws that work. Unsolicited bulk email (UBE) needs to be made simply illegal. No more winking at "marketing approaches" -- just treat it like the theft of service that it is. Unsolicited email is not the same as unsolicited paper mail, because the sender pays almost none of the costs, so we shouldn't expect that unsolicited bulk email is permitted the same way that it is with paper mail.Will people disobey the law? Sure. But after a few people lose all their money, or go to jail, this exploitation will quickly reduce to levels that can be handled by current technology. We need both laws and technology. One without the other is like fighting with one hand tied behind our back. Europe is already leading this way; they've already passed these kinds of laws. But without cooperation from the US, it won't be enough. Once there are only a few rogue states, their cooperation can be more easily acquired... by refusing to accept any of their email until they update their laws. This is just like the computer break-in laws a few decades ago; at first, many people didn't even understand that break-ins were like trespass. Then, over time, laws were passed to forbid people from harming others using computers.
Part of the solution: laws that work Posted May 5, 2004 22:33 UTC (Wed) by jamesh (subscriber, #1159) [Link] As well as making spamming ilegal, you'd probably need some kind of penalty for having your products advertised via spam. Otherwise companies will just blame it on over-eager affiliates, or similar.Of course, the penalties for having your product advertised via spam would need to be a fine rather than jail. In the case where it really is done by a third party without consent, the company should have some way to recover the fine from the spammer.
Part of the solution: laws that work Posted May 7, 2004 15:09 UTC (Fri) by brouhaha (subscriber, #1698) [Link] you'd probably need some kind of penalty for having your products advertised via spam.Some people have proposed the death penalty for spammers. Most such proposals are probably somewhat tongue-in-cheek. I used to think that the death penalty for spam was too severe, but after a small amount of back-of-the-envelope analysis, I'm not so sure. A single spam campaign easily wastes 10 seconds each of the lives of 100,000,000 people. In total, that's more than a third of a human lifetime. So one spam campaign is equivalent to murdering a 50 year old person. The death penalty (or life imprisonment with no possibility of parole) thus seems entirely reasonable.
Part of the solution: laws that work Posted May 6, 2004 5:30 UTC (Thu) by copsewood (subscriber, #199) [Link] Forging paper mail sent out using someone else's address is rightly considered a criminal act of deception. The same should apply in the email world. Spammers would be less willing to do this if they faced jail time when caught. Suitable laws should also making those whose products are advertised in this manner criminal associates and conspirators. It would be easier to trace spammers if those obtaining advertising services in this manner could only avoid jail time by actively assisting the authorities in tracing those sending out this stuff.Blacklisting and blocking can become much more effective when spammers have to use their own domains and give valid return addresses.
82% of email is spam Posted May 5, 2004 22:09 UTC (Wed) by freethinker (guest, #4397) [Link] Hmm...Wood also attributes the rise in spam to the huge number of open proxies on the Net. Virus writers began teaming up with spammers last year, and so far it's been a dangerous combination. Virus writers send out malicious code that infects computers and opens a back door in the machine. Wood estimates that 70 percent of spam is sent through open proxies. Well, then, this isn't a problem, because as everyone knows, Microsoft is now taking security seriously! So I'm sure all those zombies will be dead real soon now.
82% of email is spam Posted May 6, 2004 9:31 UTC (Thu) by mmarsh (subscriber, #17029) [Link] Microsoft's product quality, from a security standpoint, would likely improve more quickly if there were a demand for it from customers. I don't mean "supply and demand"-type demand, I mean "fix your damn software!"-type demand. The best way to do this is for ISPs to go back to TOSing users whose machines are compromised and used as zombies. It all goes back to good digital hygiene, and most users today have never had an Internet Health class. One of the reasons that FLOSS has a better security record is that the Unix community went through its baptism by fire a few decades ago, and now the bulk of the Unix-ish systems users _know_ that they want a secure system and _expect_ it from their providers. FLOSS is even more susceptible to this, since there's a lot less lock-in with it, making it easy for users to switch to alternatives if they're not satisfied.
82% of email is spam Posted May 17, 2004 6:01 UTC (Mon) by coolian (guest, #14818) [Link] Why do you think Microsoft will EVER change, unless given a monetary reason to do so? They dominate for now, and they rarely change until they are forced to. Bush couldn't force them, but maybe in a few years, when Linux is making MS toss its salad...
82% of email is spam Posted May 13, 2004 6:47 UTC (Thu) by Wol (guest, #4433) [Link] My box got compromised (W2K server).Indeed, it was so bad, that it took me about four attempts before I could download the necessary security patches to fix the problem. The spam software - which I didn't know what it was although I knew something was wrong - was pinching nearly all available cpu and bandwidth so the system had nothing left to download the patches with :-( Cheers,
Solution: Sender Policy Framework Posted May 6, 2004 5:39 UTC (Thu) by copsewood (subscriber, #199) [Link] Some very good work is advancing quickly on DNS technology to enable domain owners to publish details of which addresses legitimate mail from their domains is allowed to come from. When enough people implement SMTP + SPF this is likely to make senders more accountable for what they send. Check out the SPF site for details. Any domain owner who can publish TXT records can help move adoption further and reduce the likelyhood that spammers will forge mail envelopes using their domain by publishing SPF records stating where you send your mail from.This isn't a complete solution by any means, but it is likely to be a neccessary step in making a complete solution possible which doesn't break too many things worth keeping.
Solution: Sender Policy Framework Posted May 6, 2004 6:07 UTC (Thu) by hingo (subscriber, #14792) [Link] One thing I've always wondered about SPF is this. When more people start using it, isn't it then just as easy for the spammers to make their worms such, that they check the DNS record to learn what SMTP server they should use and then send their mail through that server. The server has no chance to realise that it is not the user of the computer sending normal email.Of course, if we have SPF, then we could start adding other things to SMTP, like the server requiring a password before accepting mail. In combination these kinds of techniques might do it.
Solution: Sender Policy Framework Posted May 6, 2004 10:28 UTC (Thu) by Ross (subscriber, #4065) [Link] Maybe. It depends on what type of authentication the server requires.Assuming the worst case, no authentication, it would still be an inprovement because the messages would really be from the domain they appear to be from. This makes it easier to contact admins, implement filters, etc.
82% of email is spam Posted May 6, 2004 6:29 UTC (Thu) by pizza (subscriber, #46) [Link] Perhaps the most promising solution I've seen is greylisting. The idea is that all "unknown" senders get sent a temporary "retry later" error, and legit MTAs will do just that, but spammers/worms/virii/etc won't bother, and will require a much more sophisticated "zombie" to send mail.This has cut the spam I receive down to a small trickle from the couple hundred a day, to say nothing for the other users of the system I admin. (The only spam I get now is stuff sent via massive BCCs using a rather traditional e-mail client) The nice thing about this is that it happens before mail is delivered, so you save on the bandwidth, storage, and cpu costs of post-delivery filtering. http://projects.puremagic.com/greylisting/ There are multiple implementations of this technique now.
82% of email is spam Posted May 6, 2004 7:32 UTC (Thu) by rwmj (guest, #5474) [Link] My standard response to spam (originally from here: http://yro.slashdot.org/comments.pl?sid=104138&cid=8868069 ) ...--- quote: Your post advocates a ( ) technical approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.) ( ) Spammers can easily use it to harvest email addresses Specifically, your plan fails to account for ( ) Laws expressly prohibiting it and the following philosophical objections may also apply: (X) Ideas similar to yours are easy to come up with, yet none have ever Furthermore, this is what I think about you: (X) Sorry dude, but I don't think it would work.
82% of email is spam Posted May 6, 2004 7:59 UTC (Thu) by alspnost (subscriber, #2763) [Link] Well, the 82% figure sounds about right for me. For a while, it seemed to be getting better, but it's now dramatically worse again. It's depressing, because SpamAssassin used to work brilliantly, but it just doesn't seem to be coping any more. Its success rate seems to have dropped from about 95% to around 50%.At work, we use a 3-tiered strategy that's pretty effective. Firstly, we use RBL blacklists, and reject connections from malconfigured mail servers (eg with DNS problems); secondly, we use SpamAssassin on everything that reaches our queue; thirdly, we've put most of our users onto Thunderbird, and the built-in adaptive filtering is pretty good at mopping up anything that gets through SpamAssassin, once it's been trained. For me personally, the biggest problem is accessing my (POP-based) mail via the webmail gateway when I'm away from home: with no spam filtering at that stage, I have to wade through 3 screenfuls of crap to find the 1 legitimate mail waiting for me....
Bandwidth Posted May 6, 2004 8:21 UTC (Thu) by vondo (guest, #256) [Link] I've heard this bandwidth argument from the Cantor and Seigel (sp?) days, and never been convinced. Even if 82% of e-mail is spam, (and for me it's about 1/2 that), all the e-mail I get in a typical day probably doesn't add up to the amount of bandwidth I consume when I visit CNN's homepage. If I watch streaming video or audio, or download 5 ISOs for the latest version of Mandrake, e-mail is just a drop in the bucket.Besides, spam e-mails are generally small. I have 13 in my filtered area now, the largest is 12 KB. People regularly send me 1MB files in the mail.
Bandwidth Posted May 6, 2004 10:10 UTC (Thu) by smoogen (subscriber, #97) [Link] If you think only about yourself then it is small. Here at a place with 12000 email accounts it actually goes over the amount that video etc are using according to our plots and graphs. I think that may be due to the fact that we can cache/proxy the video/web much better than 12000 slightly different emails to 12000 accounts
Thrusted mail ? Posted May 6, 2004 12:50 UTC (Thu) by bockman (subscriber, #3650) [Link] IF (but what a big IF it is) everyone started to use signature in their mail (I don't, actually), then one could think of mail clients that only accept mail with given signatures, or better yet we could instruct our ISP mail server to reject any e-mail addressed to us but not coming with a 'good' signature.Mailing lists also could work this way, carrying both the mailing list signature and the signature of the poster. This would work for 90% of users or more. Those needing to receive truly anonymous mail, could set special mailboxes for that. Thinking of that, is anyone aware of any anti-spam filter that works on signatures? (Just curious: my personal small spam problems are easily solved by pressing 'd' in mutt some 8-10 times per day ).
Thrusted mail ? Posted May 7, 2004 2:14 UTC (Fri) by dd9jn (subscriber, #4459) [Link] Sorry, that does not work.The problem is the definition of what makes up a "good signature". Yeah, we know about the Web of Trust and hierachical trust models but this will only work between people who know each other. This would lead to a system of closed groups like BBSes decades ago. If you don't care about the validity of the signature, but merely check for the mathematical correctness, spammers will simply create new keys for each spam and sign it. No, there is no performance problem given the legions of zombies waiting for their evil masters.
Thrusted mail ? Posted May 7, 2004 5:22 UTC (Fri) by bockman (subscriber, #3650) [Link] Uhm, maybe my usage of e-mail is anomalous, but 99% of the mail I receive, at work and at home, belongs to one of these categories:- people I know (and then I can get their publick key) - mailing list or newsletted which I subscribed to (and also here, a public key could be distributed by the mailing list ) - Spam/viruses (ok, viruses could also come from known people, but spam usually doesn't) Once per month, or even less, I may receive a mail from an unknown person, that wants to get in touch with me. Therefore, for me would be very easy to define a list of 'good signatures' that could be used to filter my mail (possibly at server level). If I don't want to loose the mail from unauthenticated sources, I could isolate it in a separate mail folder, to check when/if I want. This mail folder could collect spam, but at least it is nicely isolated (and if I get too much annoyed, I could simply bounce unauthenticated mail). Now, I understand that people, and especially business, can use e-mail to get in touch daily with unknown people. But I believe that a lot of people have an e-mail behaviour very similar to mine.
Thrusted mail ? Posted May 7, 2004 6:37 UTC (Fri) by dd9jn (subscriber, #4459) [Link] It is not only businesses receiving a lot of mail from yet unknown people but also people from the Free Software community, especially authors and maintainers. There is as well the problem of resending and forwarding messages.Mailing lists are another problem. Of course the mailing list software could sign all message to be send out but that won't help. For a closed mailing list this will currently help but I have already encountered faked From addresses (which are the current way of authenticating subscribers) which led spam slip through. Open mailing lists (everyone is allowed to post) are already nice spam exploders and it won't help to have a signature applied by the ML software. Over short or long we have to change the authentication of mailing lists anyway to a stronger one (i.e. only accept signed posts), but this will require manual approval of subscription requests to sort out spammers. For some mailing lists this will not be possible at all - think of a help list for the signing or MUA software ;-).
Thrusted mail ? Posted May 8, 2004 10:56 UTC (Sat) by giraffedata (subscriber, #1954) [Link] Your system doesn't even require signatures. It's exactly what I do today, based on the from: header. The from: header is successfully forged (i.e. has one of the 2,000 addresses in my white list) in less than 1% of my spam.But I still have a big problem. I route the stranger mail to a special folder, which I check daily, but there are 250 spams a day. I've had to start automatically deleting some of it (e.g. any all-html email), but that's risky. I think people who believe the only interesting email they will get is from acquaintances lack imagination. You can't just delete mail from strangers; you have to spend some time looking at it to see if it's spam. I occasionally have my outgoing emails bounced by overzealous spam filters, and in every case the recipient would have been glad to get my email.
Trust metrics are the FUSSP Posted May 6, 2004 17:42 UTC (Thu) by raph (guest, #326) [Link] I've given a lot of though to e-mail spam as one of the possible applications of trust metrics. I've done a piss-poor job of self promotion, but if you look at the eigenvector-based diary ratings on Advogato, they're fabulously accurate, and the site itself runs a near-zero spam/abuse rate, in spite of the very open access policies, and the near total neglect of administrative duties on my part.Unfortunately, trust metrics are difficult enough to understand (and I've done such a poor job evangelizing people so they're motivated to understand), that they haven't made much impact on the world. I resemble a few of those check-boxes on the form posted above, but I do suspect that one of two things is true: the spam problem won't be solved in our lifetime because of a combination of technical difficulty and lack of willpower; or that the solution will use some form of peer certification to distinguish spammers from legitimate email correspondents. These days I'm more inclined to believe the former, but that's probably just my legendary bitterness shining through.
82% of email is spam Posted May 6, 2004 21:13 UTC (Thu) by neilbrown (subscriber, #359) [Link] Maybe the best way to beat this statistic is to simply generate more genuine mail. If everybody subscribed to a few high-volume mailing lists..... :-) (I must admit that linux-kernel is one reason that my spam ratio is quite low. "host -t MX cse.unsw.edu.au" might give you a hint at another).But on a more serious side, I'm coming to the conclusion that the only long-term solution to SPAM must involve white-listing. i.e. I *only* accept mail from addresses (SPF-verifiable addresses) that I trust. This would require MUA support so that anyone I send mail to automatically gets white-listed, and things like that. It would also mean that when people (e.g. companies) ask for your Email address, their would need to give your their email address in return (We will only send you mail from info@clever-company.com.) which you would have to white-list. If people who you don't know want to send you mail, that has to be possible, but it also should be expensive (roughly the cost of a postage stamp or a 'phone call). This might involve finding a common correspondant to introduce you, or it might involve some "proof-of-cpu-power-spent" similar to HashCash, or it could even involve an exchange of money (e.g. I will read your mail if you can prove that you have bought a $1 e-postage stamp from World Vision). It could use any other challenge-response that a recipient is happy to impose on potential senders. The big problem today with challenge-response is that you risk sending challenges to innocent third-parties whose address has been used inappropriately. This is where I think SPF really gives value. With SPF, I can tell if I can trust a return-address, and so I can know if it is safe to send a challenge. If the MTA add an appropriate header with SPF status, this can all be done in the MUA. This doesn't address the bandwidth/server-load problem. It isn't clear to me that there can be a better total solution to that than pushing the whitelist+challenge-response into the MTA (there are lots of partial solutions which can heurisically drop a lot of bad mail, but they will eventually cause the spammers to get smarter).
82% of email is spam Posted May 7, 2004 15:21 UTC (Fri) by rgoates (subscriber, #3280) [Link] My 2 bits. Most of the spam I get is very repetitious in form, disregarding the list of bogus words used to bypass filters. I suspect a little human intervention added to filters can get rid of most spam, and doing it at the ISP level can reduce the impact on the net. Here's how it might work:1) I get spam in my mailbox. So what do we gain? The spammer has to work a lot harder (harder than adding some bogus word strings, anyway). With adequate speed of response from spam recipients, many of the spammer's targets would never see his spam. And the pattern building is largely centralized and so can be more easily improved as we become more clever. Additionally, catching spam at the network edge allows easier identification of the spam source. The monetary cost is primarily in the "spam alert" service. I suspect that cost will be very low compared to the size of the spam problem. The social cost (or effort) will include getting most of the ISPs to use the pattern filtering, but I suspect seeing success at a few ISPs will provide a lot of momentum. I for one am certainly willing to badger my ISP to use it. There will be no silver bullet. As we all know, there are always antisocials with too much time and misdirected energy. The best we can do is to keep far enough ahead of them to keep their damage to a tolerable level. I think that will be good enough, if we can do it.
82% of email is spam Posted May 8, 2004 11:13 UTC (Sat) by giraffedata (subscriber, #1954) [Link] It seems to me that that doesn't have to be done with a complex inter-ISP system. It can be done within each individual ISP (and I thought they were already doing it). I believe the vast majority of the world's email accounts are within giant ISPs. If we can filter spam from giant ISP users, the spammers will give up.So why isn't it working? Is it technologically infeasible? Are big ISP customers not availing themselves of the filtering?
82% of email is spam Posted May 15, 2004 11:00 UTC (Sat) by bronson (subscriber, #4806) [Link] Big ISPs never look at the content of what they are transmitting, lest they lose their common carrier status. Also, the tinfoil hat crowd claims that the big ISPs make good money on spam so it's not in their best interest to stop it.
spam filtering by ISPs Posted May 15, 2004 11:19 UTC (Sat) by giraffedata (subscriber, #1954) [Link] We may be using different definitions of ISPs; I believe this thread is about providers of email mailboxes. AOL, Yahoo, Hotmail, Earthlink, ... I know these guys filter spam based on content. I see it in my own Yahoo mail account, and Earthlink advertises it. So why is spam still effective? Are they not filtering enough? Are users opting out?
82% of email is spam Posted May 10, 2004 9:17 UTC (Mon) by ikm (subscriber, #493) [Link] I think the roots of the problem are very simple. Spam exists just because it is welcomed by most people. That's right -- they read it, and in case they are not interested about the offering they read about, they pronanly just think -- "what the hell, spam sucks!"... but if they are interested about the offering, they go for it! They go for it! That way spam proves as an effective marketing solution. And the only real way to stop it is to make it ineffective.This all is about humans, not about the computers. But computers can help achieve the goal. Say, any mail server that detects a spam messsage attaches a promotion text just above the text of the message, explaining the problem briefly and saying that the only way to stop spam is to stop accepting *any* offerings in that spam. That is, attaching anti-spam to any spam. The more spam one gets, the more probably he will read the banner at last and understand the problem. Well, in case most people are failing to read that banners and understand the problem, all the humanity will keep getting spam -- in that case, it actually deserves it. That sounds bitter, of course, but that's the economy -- while there is an interest, there is an offering.
|
|||||||||||