|
|
Subscribe / Log in / New account

A note for Hotmail users

The wise folks at Hotmail have decided to start blocking email from LWN's server; they also have not really bothered to inform people of how to get themselves unblocked. As a result, anybody who has subscribed to an LWN mailing list from a Hotmail account has been unsubscribed. It must be said that we were surprised by just how many of those there were. Please accept our apologies for the inconvenience.

to post comments

A note for Hotmail users

Posted Mar 10, 2011 18:18 UTC (Thu) by clugstj (subscriber, #4020) [Link] (4 responses)

Cue the conspiracy theorists.

A note for Hotmail users

Posted Mar 10, 2011 18:38 UTC (Thu) by wallmari (guest, #72956) [Link] (2 responses)

No, Hotmail are consistently useless. I'm responsible for a set of servers that deliver several million emails a day, and occasionally, despite following every single best practice and being registered to every Hotmail bulk sender policy, they still to occasionally blacklist a server or two of ours.

The worst part is they'll return 5xx codes for perfectly valid email addresses if they decide they don't like the IP address, which can be pretty nasty. It's now one of the first support questions for people who suddenly find their email settings changed - "Are you using Hotmail? You are? Okay, they reported your email account was closed, so you need to take it up with them."

A note for Hotmail users

Posted Mar 10, 2011 18:40 UTC (Thu) by ESRI (guest, #52806) [Link] (1 responses)

Even worse is when they return no error codes, accept the email but silently throw it away...

A note for Hotmail users

Posted Mar 11, 2011 2:25 UTC (Fri) by miguelzinho (guest, #40535) [Link]

Been there, I know the pain.

Users come to you, you check the logs, message delivered. You politely say that there is absolutely nothing you can do.

User blames you, "our mail server sucks", she/he complains about you and the poor work you are doing to your manager.

A note for Hotmail users

Posted Mar 10, 2011 19:28 UTC (Thu) by clugstj (subscriber, #4020) [Link]

I was kidding.

A note for Hotmail users

Posted Mar 10, 2011 18:40 UTC (Thu) by ESRI (guest, #52806) [Link] (1 responses)

Hotmail is horrible about this. At least once a year I have to go through their postmaster pages and get myself unlisted.

Setting up DomainKeys, SPF and such helps, but ultimately they appear to just randomly start blocking things.

Fortunately, it's pretty easy to get in touch with a real live person via the postmaster pages above and they'll most likely resolve the issue for you.

A note for Hotmail users

Posted Mar 17, 2011 9:47 UTC (Thu) by Seegras (guest, #20463) [Link]

Hah!

Everytime Microsoft implements something whose name it prefixes with "Smart", you know it's totally dumb. "SmartScreen" in that case.

And here's my story: http://seegras.discordia.ch/Blog/abused-by-microsoft/

A note for Hotmail users

Posted Mar 10, 2011 18:56 UTC (Thu) by jwb (guest, #15467) [Link] (111 responses)

Assuming you are sending mail from your MX (tex.lwn.net), your IP block reputation is unfortunately quite low. According to sendmail.com, your IP block is medium risk (60% reputation where 100% is best). It is likely that other people at your ISP are spamming.

(disclaimer: i am an operator of gmail.com)

A note for Hotmail users

Posted Mar 10, 2011 19:26 UTC (Thu) by michel (subscriber, #10186) [Link] (64 responses)

Seems like the system is being gamed in favor of large SaaS vendors and against small businesses who are just trying run run their own SMTP server.

When I checked that tool, my poor SMTP server sits in a pool with more than 500K other IPs. Good luck on me getting the reputation of that block higher. Geez.

A note for Hotmail users

Posted Mar 10, 2011 19:41 UTC (Thu) by jwb (guest, #15467) [Link] (18 responses)

"Small businesses who just want to run their own mail server" are predominantly idiots. People don't want to get spam, so you have to do something to turn off all the botnets.

A note for Hotmail users

Posted Mar 10, 2011 20:25 UTC (Thu) by csamuel (✭ supporter ✭, #2624) [Link] (1 responses)

I've got an idea to improve that; the US is responsible for most of the spam in the world [1] so based on that way of thinking we should be blocking US IP addresses.

I guess I can no longer recommend Gmail to people to people who may actually want to receive email now.. :-(

[1] - http://www.spamhaus.org/statistics/countries.lasso

A note for Hotmail users

Posted Mar 17, 2011 18:48 UTC (Thu) by bboissin (subscriber, #29506) [Link]

> I guess I can no longer recommend Gmail to people to people who may actually want to receive email now.. :-(

Why would that be? The parent didn't say that gmail was using the sendmail.com data, and it is very likely true that they are "predominantly" idiots (not all, but most of them).
This doesn't mean gmail cannot differentiate between the idiots and the others.

A note for Hotmail users

Posted Mar 10, 2011 21:27 UTC (Thu) by lutchann (subscriber, #8872) [Link] (5 responses)

So what do you recommend to small businesses that need to be able to send and receive email? For us, saying "use Gmail" isn't an option because of contractual confidentiality requirements.

A note for Hotmail users

Posted Mar 10, 2011 23:05 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

Forward outgoing mail via your ISP's server? Or is that likely to be blocked too :-)

At least it's likely to get unblocked quickly, should that happen.

Cheers,
Wol

A note for Hotmail users

Posted Mar 11, 2011 0:02 UTC (Fri) by lutchann (subscriber, #8872) [Link]

Our mail server is co-located in a commercial data center. So, there's no "upstream" mail server to forward to.

A note for Hotmail users

Posted Mar 11, 2011 8:39 UTC (Fri) by lkundrak (subscriber, #43452) [Link] (2 responses)

So, you're using plain unencrypted e-mail messages for confidential internet communication?

A note for Hotmail users

Posted Mar 11, 2011 15:32 UTC (Fri) by lutchann (subscriber, #8872) [Link] (1 responses)

We're required to store confidential data on servers that we control. Whether that data is encrypted on disk or in transit is irrelevant, as far as our legal obligations go.

A note for Hotmail users

Posted Mar 13, 2011 4:06 UTC (Sun) by no_treble (guest, #49534) [Link]

And this is why I despise businesses or organizations asking me for unnecessary personal information for simple transactions. Pair this kind of attitude with the increasing willingness of millions to give all of their personal information to third parties for free (hello Facebook), and it's no wonder identity theft is the fastest growing criminal activity.

A note for Hotmail users

Posted Mar 11, 2011 1:00 UTC (Fri) by tpo (subscriber, #25713) [Link] (4 responses)

> "Small businesses who just want to run their own mail server" are
> predominantly idiots. People don't want to get spam, so you have to do
> something to turn off all the botnets.

Does that mean that small businesses souldn't bother sending mail to Gmail accounts?

A note for Hotmail users

Posted Mar 11, 2011 11:59 UTC (Fri) by endecotp (guest, #36428) [Link] (3 responses)

>> "Small businesses who just want to run their own mail server" are
>> predominantly idiots. People don't want to get spam, so you have to do
>> something to turn off all the botnets.
>
> Does that mean that small businesses souldn't bother sending
> mail to Gmail accounts?

Yes, basically. I run my own mail server on a machine in a co-located facility. Although I can currently send email to random joe@gmail addresses, I don't seem to be able to email companies who use the "gmail for domains" service, or whatever they call it. I've given up trying to email such people.

Note to jwb: is your Google boss OK with your "idiots" comment? Is that an official Google statement?


A note for Hotmail users

Posted Mar 11, 2011 15:12 UTC (Fri) by foom (subscriber, #14868) [Link] (1 responses)

> Note to jwb: is your Google boss OK with your "idiots" comment? Is that an official Google statement?

And what is *that*? A threat to go tell his boss on him for expressing a opinion strenuously? Sheesh!

A note for Hotmail users

Posted Mar 11, 2011 18:55 UTC (Fri) by rahvin (guest, #16953) [Link]

I would take it more as a warning that if you are going to comment on who your employer is and what you do for a living you probably shouldn't be using inflammatory language that can come back and get your butt fired. There is a reason for anonymity, when you breach that anonymity you risk exposing yourself to real world consequences for behavior on the internet. I'll never understand the personal crap people discuss or the inflammatory language they use that can be tied back to their real life identity. It would be trivial for Google to figure out who he/she is and react to calling their potential customers idiots when some journalist looking to create a name publicizes the quote.

A note for Hotmail users

Posted Mar 11, 2011 15:16 UTC (Fri) by salimma (subscriber, #34460) [Link]

I had an interesting experience recently with two separate Google Apps-managed domains that I manage. I could send emails to one of them, but to the other it just silently vanishes. Same DNS registrar for both...

Ended up resolving it by just making our MTA forward its mail through Gmail, by creating a user for it, with aliases for each service on our server that needs to send email.

A note for Hotmail users

Posted Mar 11, 2011 7:17 UTC (Fri) by rilder (guest, #59804) [Link] (1 responses)

You sound more like from a gmail marketing team than from operations. You do know that there are other mail servers than gmail right ? Also, spam also emerges from many gmail accounts. Turning off an entire IP block to prevent spam sounds less of an engineering solution and more of a shortcut solution.

Now that I remember, there was a solution flouted few months back about quarantining certain parts of internet since Microsoft couldn't prevent their PCs from getting infected. Do you see the parallel here ? Needless to say incredibly stupid, but what is surprising is that people in other areas coming up with nincompoop solutions to similar issues.

Also, why don't you block China/US as mentioned by others, it will zero the spam for the most part.

Gmail is part of the spam problem!

Posted Mar 11, 2011 18:18 UTC (Fri) by tmassey (guest, #52228) [Link]

For *weeks* now I've been getting blowback from spam using my e-mail address as the From:, so I get the bounceback messages. Examining the header, what is the first non-forged hop? gmail.com. All of the spam is going through Gmail first.

In Google's insatiable desire for data, they are willing to let virtually anyone to send through their SMTP servers. To me, they're nearly indistinguishable from an open relay.

A note for Hotmail users

Posted Mar 11, 2011 8:19 UTC (Fri) by spaetz (guest, #32870) [Link] (1 responses)

> "Small businesses who just want to run their own mail server" are predominantly idiots.

So according to sendmail.com reports, my webhoster has a risk class of 70, my employer's mail server (a large University) has a 70 risk class and my ISP is using gmail which resolves ultimately to ghs.l.google.com gives me a risk class of 70... ?

100 means blocked, according to them. So if that sendmail.com report is any good, I have no means to send email reliably?...

A note for Hotmail users

Posted Mar 11, 2011 15:55 UTC (Fri) by michel (subscriber, #10186) [Link]

One wonders who the idiots are in this context.

A note for Hotmail users- not idiots

Posted Mar 17, 2011 16:09 UTC (Thu) by RogerOdle (subscriber, #60791) [Link]

If you want secure email inside your company then you want your own email server. This email has to be able to forward email addressed outside of the company but should never forward local email.

I setup an email server (MTA) years ago and didn't know how to configure it to tell other MTAs that it was an end node and could not relay email. The result was that my ISPs MTA pushed other people's email into the cache of my email server. I think that the system has gotten better now.

If you have a properly configured smtp server then what belongs in you company, stays in your company.

A note for Hotmail users

Posted Mar 10, 2011 19:43 UTC (Thu) by pzb (guest, #656) [Link] (44 responses)

I don't think it is so much in favor of large SaaS vendors as it is just getting harder and harder to run a MTA on today's spam filled internet. No longer is simply implementing plain SMTP (RFC 821) enough, you need to support SPF, SenderID, DomainKeys and DKIM and ensure your MTA's hostname and greeting are "right" (where "right" is not defined in an RFC).

Looking at lwn.net, basic DNS looks "right", but there SPF records at all, which is pretty much a minimum requirement these days.

A note for Hotmail users

Posted Mar 10, 2011 20:00 UTC (Thu) by jwb (guest, #15467) [Link] (3 responses)

A note for Hotmail users

Posted Mar 10, 2011 20:43 UTC (Thu) by pzb (guest, #656) [Link] (1 responses)

I didn't know about that one. Is anyone actually checking _dmarc records yet?

In a similar vein, do you know if anyone is using client certs + STARTTLS to help verify hosts?

I know that spam filtering is one of those areas, like code exploits, where there is a tendency to consider security through obscurity a good thing, but it would really help the average mail admin if more detail was published how to be a good email citizen.

A note for Hotmail users

Posted Apr 3, 2011 10:27 UTC (Sun) by darylt (guest, #74039) [Link]

> I didn't know about that one. Is anyone actually checking _dmarc records yet?

Google/Gmail seems to be. I've noticed hits on my name servers for _dmarc records for my domain that appear to be coming from their IP block, starting around early February this year.

The link for me is dead as well - any other pointers to the draft-draegen-dmarc document? (Google has ironically failed me).

A note for Hotmail users

Posted Mar 31, 2011 14:51 UTC (Thu) by mfedyk (guest, #55303) [Link]

That link is broken right now.

Where can I find more information about DMARC?

A note for Hotmail users

Posted Mar 10, 2011 20:27 UTC (Thu) by foom (subscriber, #14868) [Link]

Well, I run my own mailserver for my personal mail (on a static IP address from RCN), and don't implement any of that crap except SPF (v=spf1 mx a -all), and I have never had problems emailing people...although I don't email people on hotmail very often, so who knows.

I'm a bit surprised to hear that anyone actually uses spf for anything. I only added the SPF record in the first place because I didn't realize how stupid the spec actually was, back when it was first announced.

Not SPF again!

Posted Mar 10, 2011 21:25 UTC (Thu) by job (guest, #670) [Link] (19 responses)

SPF breaks forwarding for zero gain. It is also badly designed and overloads txt records with a crummy format instead of specifying a real data type. The sooner it dies the better.

It does absolutely nothing to prevent spam. Reputation works better on an IP basis than a domain basis, as the latter is trivial to renew (even free if you're large enough). When you point this out to its proponents you get a blank stare back, "of course, SPF is not designed to prevent spam" they say.

Then what is the gain? Nothing. Some people mention false bounces but these can be trivially filtered at MTA level without complex filters stuffing rules in DNS.

Of course this is nothing new. You can find the usual arguments by asking Google "SPF considered harmful".

Tell me again why SPF is "pretty much a minimum requirement these days"?

Not SPF again!

Posted Mar 10, 2011 22:11 UTC (Thu) by HenrikH (subscriber, #31152) [Link] (17 responses)

SPF prevents spoofing, that is why it's of interest to companies.

Not SPF again!

Posted Mar 10, 2011 22:19 UTC (Thu) by job (guest, #670) [Link] (16 responses)

Then why is it brought up in a thread about reputation systems for spam?

It's symptomatic for SPF. It's being sold as a panacea for spam, for which it clearly is broken.

Spoofed emails is not a problem. Have to ever heard of false emails being sent, unless it was spam or phishing which we take care of with our normal reputation systems? If it were a problem we'd all be using certificates just like we do with HTTP. That would be both simpler and work better than SPF.

Not SPF again!

Posted Mar 11, 2011 8:43 UTC (Fri) by epa (subscriber, #39769) [Link] (13 responses)

A large proportion of spam is spoofed, so if you get rid of the spoofed messages, you get rid of most spam.

Not SPF again!

Posted Mar 11, 2011 11:11 UTC (Fri) by dwmw2 (subscriber, #2063) [Link] (12 responses)

No, you just move the goalposts so that most spam will no longer be spoofed.

Not SPF again!

Posted Mar 11, 2011 12:19 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

Wasn't there a study which found that at some point most SPF-valid email was spam? To think spammers can't setup SPF records, if they needed to, is very very naive.

Not SPF again!

Posted Mar 11, 2011 19:57 UTC (Fri) by nof (guest, #61716) [Link]

Well, for the servers I manage:
2% with a correct SPF is spam (good, I can now blacklist those servers as they identified them selves!)
20-25% of all good email have a valid SPF record.
1-3% email have faulty SPF record. These are almost 100% mass mailing 'news'. No-one will miss these mails if the are denied into the network.

All in all, I find SPF as a rather good tool to have.
SPF is NOT a solution to spam ALONE.
Enable it in spamassassin and see for yourself.. .. be sure to give SPF fault a borderline reject score. Give no score for a correct SPF.

I think of SPF as a tool for (validating) the SENDER. Not into it for the receiver.

So, if you publish a SPF record, you better have a corporate policy to back it up.
I have and it works great. The numbers of emails faked with user@ourdomain as sender dropped like a stone.

Not SPF again!

Posted Mar 16, 2011 12:10 UTC (Wed) by epa (subscriber, #39769) [Link]

No, you just move the goalposts so that most spam will no longer be spoofed.
All spam filtering has that property: if you do Bayesian filtering, you ensure that most spam will not contain keywords such as 'Viagra' (they are obfuscated instead); if you drop messages from known spamming hosts, you ensure that most spam comes from widely spread botnets, and so on.

Factually, I completely agree with your statement. If you stop spoofing of mail, then most spam will no longer be spoofed. That doesn't exclude the possibility that the total amount of spam falls. No anti-spam measure so far really fixes the problem; all you can do is try to block 80% of it

Not SPF again!

Posted Mar 17, 2011 5:53 UTC (Thu) by zlynx (guest, #2285) [Link] (8 responses)

And this is a great thing, especially against phishing spam.

If the phishing can't get away with claiming to be from microsoft.com or amazon.com or chase.com then it makes it that much harder to convince people to click and give away passwords.

Not SPF again!

Posted Mar 17, 2011 9:17 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (7 responses)

And this is a great thing, especially against phishing spam.

If the phishing can't get away with claiming to be from microsoft.com or amazon.com or chase.com then it makes it that much harder to convince people to click and give away passwords.

The problem is, with SPF the phishing can still get away with claiming to be from microsoft.com or amazon.com or chase.com. SPF only validates the envelope sender as used in SMTP. It doesn't affect the From: header of the mail, which is what the user actually sees, at all.

Imagine you receive a letter, but your secretary has opened the envelope and thrown it away, and just put its contents in your in-tray.

SPF is equivalent to validating the sender's address as it appears on the back of envelope that your secretary just threw in the bin. It buys you absolutely nothing when you're actually looking at the letter.

Not SPF again!

Posted Mar 17, 2011 13:36 UTC (Thu) by zlynx (guest, #2285) [Link] (6 responses)

I am pretty sure SpamAssassin has rules to score on mismatched From and envelope headers. If it doesn't then eSoft must have invented their own because I know it can be done and can work pretty well.

So, no. The phishing emails cannot get away with it.

Not SPF again!

Posted Mar 17, 2011 13:38 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (5 responses)

Congratulations. You've just invented a way to break fairly much all known mailing lists.

Not SPF again!

Posted Mar 17, 2011 14:03 UTC (Thu) by zlynx (guest, #2285) [Link] (4 responses)

Give me some time to dig up the actual rule. I know it works, I just don't remember the details.

With some creativity I'm sure you could figure out your own way to score on envelope and From mismatch without breaking mailing lists.

Not SPF again!

Posted Mar 17, 2011 14:11 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (3 responses)

I could go down a huge rathole trying to fix all the things that SPF breaks with its fundamental misunderstanding of how mail actually works in the real world. Or I could just ignore it as a bad idea altogether and concentrate on something that actually validates the From: header directly. Like DKIM or S/MIME.

"When banks start DomainKeys or S/MIME signing all outbound mail, I promise to give up SPF and Sender ID."
— Meng Weng Wong, inventor of SPF.

Not SPF again!

Posted Mar 17, 2011 14:15 UTC (Thu) by zlynx (guest, #2285) [Link] (1 responses)

And when everyone in the world who has an SPF record also implements those then there will be rainbows and unicorns for all.

In the meantime people who care can gain a few more spam accuracy points by doing a bit of extra work.

Not SPF again!

Posted Mar 17, 2011 14:17 UTC (Thu) by dwmw2 (subscriber, #2063) [Link]

s/care/don't mind throwing the baby out with the bathwater by losing some genuine mail/

Not SPF again!

Posted Mar 17, 2011 22:14 UTC (Thu) by spaetz (guest, #32870) [Link]

Hey, my german bank sends me emails encrypted with my gpg key. Does this Count?

Not SPF again!

Posted Mar 12, 2011 20:57 UTC (Sat) by marcH (subscriber, #57642) [Link]

> Spoofed emails is not a problem.

If you want to filter based on the emitter, then spoofing is obviously a problem.

Not SPF again!

Posted Mar 13, 2011 1:43 UTC (Sun) by HenrikH (subscriber, #31152) [Link]

Spoofing can be a big problem, especially since most people does not check PGP-signatures (or even inserts them in the first place). The good thing about SPF is that it is handled automatically by the mail server.

Before activating SPF I actually received quite a lot of "your mail was detected as spam" replies from other people receiving spam orginating from my e-mail address. After SPF this has dropped significantly.

And I don't want our customers (or other people for that matter) to receive mails that they think is coming from me due to spoofing.

Granted that certicates etc are way better to fix this but SPF is far from worthless.

Also regarding normal spam, SPF has forced some of the spammers to actually register and publish their domains so blacklisting gets easier. Just a minor benefit though since they keep chaning domains quite rapidly.

Phising is also a good candidate for this. Considering the amount of people beeing fooled by mail from "almost-the-domain-of-your-bank.com", consider the amount of people that would be fooled if it really came from the correct domain!

Not SPF again!

Posted Mar 11, 2011 20:14 UTC (Fri) by rqosa (subscriber, #24136) [Link]

> overloads txt records with a crummy format instead of specifying a real data type

That's not true anymore; there's an SPF record type now.

Er, not SPF

Posted Mar 10, 2011 22:09 UTC (Thu) by dwmw2 (subscriber, #2063) [Link] (10 responses)

You absolutely should not implement SPF. It is broken snake oil based on a fundamental misunderstanding of how email works.

http://david.woodhou.se/why-not-spf.html

Er, not SPF

Posted Mar 11, 2011 14:04 UTC (Fri) by pboddie (guest, #50784) [Link] (9 responses)

I had an argument with the infrastructure bureaucracy at work about this, and that document came up. But SPF most certainly eliminated "backscatter" from spam for my personal mail, so unless people can actually point to a different actually-deployed technology that achieves this, I guess I'll be keeping the SPF records, thanks.

As for the argument at work, I mentioned SPF in a sort of "what about" or "for example" context when a mail went out warning everyone about a previous phishing mail, telling everyone not to send their passwords to the perpetrators, and that got me flamed. But then I may be alone in finding it absurd that you have phishing attempts claiming to be from admin@org.xyz being delivered by mail servers belonging to org.xyz to users at org.xyz, and no apparent authenticity check was being made on the originator's e-mail address and whether the mails actually came from such an account, if it even existed.

Er, not SPF

Posted Mar 11, 2011 14:37 UTC (Fri) by dwmw2 (subscriber, #2063) [Link] (8 responses)

Publishing SPF records alone doesn't stop you getting backscatter. It only actually protects you from backscatter from those people who are daft enough to actually reject mail for an SPF failure.

My solution to backscatter is different, and doesn't require anyone else to participate in any hare-brained scheme that tries to change the way that email has worked for decades.

It's really simple: I just never send MAIL FROM:<dwmw2@infradead.org>. And thus I never accept bounces to that address either. Any mail I did genuinely send will be from an automatically generated address of the form <BATV+be504084107f+2756+infradead.org+dwmw2@phoenix.srs.infradead.org> instead. Those addresses have a date encoded into them, and I accept bounces to each address for about two weeks.

These auto-generated addresses are only in SMTP; the "envelope" of the mail. You still see my proper address in the From: header, of course. Under normal circumstances, users never see those generated addresses.

The additional benefit is that anyone who does happen to be doing sender verification callouts will manage to discard faked mail from me. But that's just a side-effect; the main effect of banishing backscatter is achieved all by myself, without anyone else having to participate.

Er, not SPF

Posted Mar 11, 2011 15:10 UTC (Fri) by pboddie (guest, #50784) [Link] (7 responses)

Publishing SPF records alone doesn't stop you getting backscatter. It only actually protects you from backscatter from those people who are daft enough to actually reject mail for an SPF failure.

But how does that explain me getting backscatter before I had an SPF record? Nobody was rejecting mail based on an SPF failure at that point. And when I did introduce an SPF record, I stopped getting backscatter completely. How does your assertion explain that?

Er, not SPF

Posted Mar 11, 2011 15:27 UTC (Fri) by dwmw2 (subscriber, #2063) [Link] (6 responses)

Nothing explains that. If true, and if nothing else on your side changed, then it's a complete coincidence.

Er, not SPF

Posted Mar 13, 2011 1:51 UTC (Sun) by HenrikH (subscriber, #31152) [Link] (5 responses)

It's hardly a coincidence. Even if few people discard mail due to SPF mismatches, spamassassin have been SPF aware for years so those mail have had their point being added above the spam threshold due to not matching the SPF records.

SPF helps enormously with backscatter/forged spams.

Er, not SPF

Posted Mar 13, 2011 10:07 UTC (Sun) by dwmw2 (subscriber, #2063) [Link] (4 responses)

I didn't say that SPF will have no effect. It will definitely stop some people accepting certain mail claiming to be you — a lot of it fake, and some of it genuine.

But if someone "stopped getting backscatter completely" after publishing SPF records, then that is a coincidence. You might expect a reduction, but certainly not a complete stop.

SPF isn't entirely ineffective. But it does break genuine mail too, and thus it isn't widely implemented in its original intended form, with an outright reject for SPF failure. And if it *was* implemented that way, you'd just see even more spam with SPF pass.

There are much better ways to achieve what SPF sets out to achieve, without throwing the baby out with the bathwater. And if it's just backscatter that you want to eliminate, you don't even need a scheme which is implemented by anyone else; you can do that completely for yourself.

Er, not SPF

Posted Mar 13, 2011 16:15 UTC (Sun) by pboddie (guest, #50784) [Link] (2 responses)

But if someone "stopped getting backscatter completely" after publishing SPF records, then that is a coincidence. You might expect a reduction, but certainly not a complete stop.

Alright, it may not have stopped backscatter completely but it most certainly appeared to reduce it to a trickle from what were effectively fire-hose levels. That is, I started getting hundreds of "returned mails", but this ceased after publishing an SPF record (as actually recommended by my hosting provider). Perhaps something got switched off in my mail provider's own spam countermeasures and then got switched on again, but there's a limit to how much investigation I can do to find the real cause.

There are much better ways to achieve what SPF sets out to achieve, without throwing the baby out with the bathwater.

I've seen (and been inconvenienced by) measures that you have suggested, but the big question is this: can people actually do such things with vanilla mail clients without messing around with MTAs and the like?

Er, not SPF

Posted Mar 13, 2011 17:49 UTC (Sun) by dwmw2 (subscriber, #2063) [Link] (1 responses)

I've seen (and been inconvenienced by) measures that you have suggested, …
It shouldn't cause any inconvenience to any well-behaving mail systems; why should it matter to you if the SMTP reverse-path on my messages is different? The localpart is supposed to be opaque data interpreted only at my end.
… but the big question is this: can people actually do such things with vanilla mail clients without messing around with MTAs and the like?
Why would you want to? It's not that hard to set it up in a decent MTA, and that's where it lives. You guarantee that all messages sent through the server are properly handled, from all clients (including phones, etc.)

Besides, decent spam filtering has to live in the MTA, for inbound messages. By the time the MUA sees it, it's too late. If you're trying to set up a decent mail system, but using a crippled MTA {,config} and trying not to change it, then you are screwed from from the very start.

If you're arguing for SPF, it seems like a disingenuous requirement — it's not as if an individual user can set up SPF from the mail client either.

I suppose you might be able to do some of it on the MUA side. If you're willing to drop the automatic generation of the reverse-paths (and hence the time limit on them) then you may be able to set up a MUA to send messages with a reverse-path different to the one in the From: address.

You'd still need to configure the MTA so it doesn't accept bounces to the "real" address though.

Er, not SPF

Posted Mar 13, 2011 23:04 UTC (Sun) by pboddie (guest, #50784) [Link]

It shouldn't cause any inconvenience to any well-behaving mail systems; why should it matter to you if the SMTP reverse-path on my messages is different? The localpart is supposed to be opaque data interpreted only at my end.

OK. I got what you suggested mixed up with schemes where people actually did change the From address. Still, for those of us who just want to use some mail provider, I guess we must either insist on a provider who does this, or we just make do with whatever other techniques are available to mitigate the problem.

I never had a real problem with handling spam in the user agent, but then my provider may well be preventing delivery of tons of blatantly bad messages for all I know. The backscatter issue has been the only time where the existing anti-spam measures have not been effective or appropriate in dealing with the problem.

Er, not SPF

Posted Mar 14, 2011 0:00 UTC (Mon) by HenrikH (subscriber, #31152) [Link]

>And if it's just backscatter that you want to eliminate, you don't even need a scheme which is implemented by anyone else; you can do that completely for yourself.

Of course, but one of the things that made me happy with less backscatter was not that I received less backscatter :) but that it implied that far less people received fraudulent mail from "me" (since their mailservers thus filtered them out).

A note for Hotmail users

Posted Mar 12, 2011 22:46 UTC (Sat) by marcH (subscriber, #57642) [Link] (7 responses)

> it is just getting harder and harder to run a MTA on today's spam filled internet. No longer is simply implementing plain SMTP (RFC 821) enough, you need to support SPF, SenderID, DomainKeys and DKIM and ensure your MTA's hostname and greeting are "right" (where "right" is not defined in an RFC).

Sending email has to be expensive one way of the other. When it is too cheap spam happens. It is probably not expensive enough yet.

A note for Hotmail users

Posted Mar 13, 2011 1:47 UTC (Sun) by HenrikH (subscriber, #31152) [Link] (6 responses)

Doesn't help since no spammer is sending the mails from their own machines. All a cost-per-mail solution would bring would be added expenses for the Windows zombies out there.

A note for Hotmail users

Posted Mar 13, 2011 19:50 UTC (Sun) by nhippi (subscriber, #34640) [Link] (1 responses)

..which would give windows pc owners an incentive to secure their machines.

A note for Hotmail users

Posted Mar 14, 2011 0:04 UTC (Mon) by HenrikH (subscriber, #31152) [Link]

One would think that having virus-ridden machines already would be a great incentive. But it apparently isn't.

Also with a large enough botnet you would probably send such a low amount of mails per zombie that the cost would go unnoticed by the users.

A note for Hotmail users

Posted Mar 14, 2011 0:47 UTC (Mon) by marcH (subscriber, #57642) [Link] (3 responses)

It helps because botnets do not implement SPF, SenderID, etc. so they are more easily blacklisted (just like small businesses who still dream sending email can be cheap). I do not care if small businesses like LWN cannot mail me anymore. Gmail solved my 100 spams per day problem and i do not care about the rest. I do not care about people who try to still live in the past, when everybody was nice and everyone was allowed to send email for free.

A note for Hotmail users

Posted Mar 14, 2011 2:44 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

if you think the spammers (including those that use botnets) don't implement SPF, senderID, etc then you have been ignoring the news of what's going on.

at one point, the existance of SPF records for a domain had a very good correlation to that message being spam, because the spammers set their scripts to create the SPF records.

these things aren't even slowing down the spammers.

by the way, who should I have to pay money to for the privilage of sending mail in your new world?

A note for Hotmail users

Posted Mar 14, 2011 7:12 UTC (Mon) by marcH (subscriber, #57642) [Link]

> if you think the spammers (including those that use botnets) don't implement SPF, senderID, etc then you have been ignoring the news of what's going on.

Indeed I do not know what is going on (except gmail solved my spam problem). I am not interested in this or that technology, I am just happy to see that it is harder and harder to send email for small players. Things should never have been easy in the first place; SMTP's design is a joke not even worth discussing about.

> by the way, who should I have to pay money to for the privilage of sending mail in your new world?

Do not take "expensive" too literally; I mean "hard" and costing time (see the first post I answered to). And if it also costs a little bit of money (think DNS, or certificates), then all the better.

A note for Hotmail users

Posted Mar 14, 2011 3:32 UTC (Mon) by foom (subscriber, #14868) [Link]

I guess you should just disable the receiving of mail from non-gmail users.

After all, anyone who doesn't use gmail is stupid and living in the past, and nobody should want to talk with them anyways, right?

Of note, of course, is that gmail isn't banning mail from lwn.net, hotmail is (and they have a long history of rejecting massive amounts of valid mail). That they have a really crappy spam filter is really not LWN's fault...

gmail is frustrating too...

Posted Mar 10, 2011 19:30 UTC (Thu) by knobunc (guest, #4678) [Link] (32 responses)

Any email from my home server gets flagged as spam by gmail.

It is not clear to me why they hate me, I've checked everything, but unless a user has flagged me as safe then all my email to them evaporates.

fiji@limey.net is the source address.

gmail is frustrating too...

Posted Mar 10, 2011 19:40 UTC (Thu) by jwb (guest, #15467) [Link] (31 responses)

Well, why would any mailer want to accept mail from an IP that usually does not send mail? You have to build up your reputation before anyone will believe it's not spam.

gmail is frustrating too...

Posted Mar 10, 2011 19:49 UTC (Thu) by xorbe (guest, #3165) [Link] (27 responses)

That's the point. The e-mail system is basically captured by big players only now, and it's nearly impossible for the little guy to send out legitimate e-mails that arrive at the big players. And yet, large retail chains are allowed to plop spam right into my big player e-mail inbox. Follow the $$$.

gmail is frustrating too...

Posted Mar 10, 2011 19:57 UTC (Thu) by jwb (guest, #15467) [Link] (26 responses)

Email is just broken. Unless you want an unbelievable amount of spam in your mailbox, you'll just have to cope. 999 times out of 1000, when an unknown or low-reputation IP address starts sending mail, it's because it's a Windows PC that was captured by a botnet. Rejection based on IP reputation is one of the best spam signals around.

gmail is frustrating too...

Posted Mar 10, 2011 20:15 UTC (Thu) by dbruce (guest, #57948) [Link] (17 responses)

"999 times out of 1000, when an unknown or low-reputation IP address starts sending mail, it's because it's a Windows PC that was captured by a botnet."

So when is the world going to wise up to the glaringly obvious enabler of spam, and start dumping Windows?

I've seen hundreds of articles about malware, and I have yet to seen anything in the mainstream media that even mentions, let alone suggests "don't use Windows".

gmail is frustrating too...

Posted Mar 10, 2011 20:23 UTC (Thu) by knobunc (guest, #4678) [Link] (1 responses)

As an aside, I'm running the Amavisd OS fingerprinter (based on p0f). It watches the incoming email traffic and guesses at the OS. Then Amavisd calls out to get info on the IP address and adds it to the mail header. So the comment notification I received for your post says:

X-Amavis-OS-Fingerprint: Linux 2.6 (newer, 3) (up: 2113 hrs), (distance 11, link: ethernet/modem), [72.51.34.34:59434]

Which is rather nice. So I can weight certain OSes more heavily in SpamAssassin.

gmail is frustrating too...

Posted Mar 10, 2011 20:33 UTC (Thu) by jwb (guest, #15467) [Link]

Good idea. I think you might find that certain, very old versions of Windows are 1) identified by p0f, and 2) almost all owned by botnets.

gmail is frustrating too...

Posted Mar 11, 2011 1:06 UTC (Fri) by HelloWorld (guest, #56129) [Link] (14 responses)

> So when is the world going to wise up to the glaringly obvious enabler of spam, and start dumping Windows?
Oh come on, we've been through this. The reason why malware is written for Windows is that writing it for Mac OS or Linux isn't worthwhile due to their low market share. You only have to take a look at lwn's security section to see that Linux-based operating systems have just as many security flaws as Windows.

gmail is frustrating too...

Posted Mar 11, 2011 7:08 UTC (Fri) by hozelda (guest, #19341) [Link] (5 responses)

Are you aware that Linux is open source and that many people discover problems early on in the process and post openly? And by posting openly many more people gain insight into the problem so they can seek superior solutions over the primary players?

Are you aware that Windows has tons of security problems that key groups know about but are not published in the open (though we get suggestions of these with large regular Windows security updates)? Closed source Windows doesn't give the customer or outside experts much insight into problems.

Are you aware that Microsoft is a single entity to manage most of the security for many people while Linux is managed by many different groups, some of whom take security as a very high priority over decisions that would maximize say profits? Thus the more security conscientious has more superior customized options with Linux.

Are you aware that Windows details are largely managed by a single company while Linux distros include a lot of variation that makes achieving a wide malware success rate more difficult?

In contradiction to your claims,
Are you aware that Linux is used by a large number of servers, yet it has an established track record of experiencing significantly less wide-scale technology specific security issues than does Windows?

Finally, if Linux is safer because it is used less as a Desktop, isn't that a legitimate reason to switch to it today? By some measures, Linux has been at 1% for many years, so there is apparently no worry that it will leave the 5% boundary any time soon.

gmail is frustrating too...

Posted Mar 11, 2011 12:02 UTC (Fri) by HelloWorld (guest, #56129) [Link] (4 responses)

Are you aware that repeating the same phrase over and over is really poor style?

> Linux is open source and that many people discover problems early on in the process and post openly? And by posting openly many more people gain insight into the problem so they can seek superior solutions over the primary players?
I don't believe this argument, because code review is hard and tedious and _very_ few people are likely to do it except when paid for it.
The sad truth is: open source is not inherently more secure than closed source software. What matters is solely the development process and the tools and techniques employed (and of course whether you have smart developers writing your software). By the way, did you know that Microsoft employed pair programming in the development of Windows 7? That's like having _all_ of your code reviewed at least once. How many Open Source projects can say that of themselves?

> Are you aware that Windows details are largely managed by a single company while Linux distros include a lot of variation that makes achieving a wide malware success rate more difficult?
Perhaps, but I wouldn't want to rely on this in order to keep my systems secure.

> In contradiction to your claims, Are you aware that Linux is used by a large number of servers, yet it has an established track record of experiencing significantly less wide-scale technology specific security issues than does Windows?
When you make such claims, you have to back them up somehow.

gmail is frustrating too...

Posted Mar 11, 2011 16:59 UTC (Fri) by tuos (guest, #43318) [Link] (3 responses)

> The sad truth is: open source is not inherently more secure than closed
> source software. What matters is solely the development process and the
> tools and techniques employed (and of course whether you have smart
> developers writing your software)

When you make such claims, you have to back them up somehow.

> By the way, did you know that Microsoft employed pair programming in the
> development of Windows 7? That's like having _all_ of your code reviewed
> at least once.

When you make such claims, you have to back them up somehow.

> Are you aware that repeating the same phrase over and over is really poor
> style?

Sometimes it's just necessary.

gmail is frustrating too...

Posted Mar 11, 2011 17:16 UTC (Fri) by HelloWorld (guest, #56129) [Link] (2 responses)

> When you make such claims, you have to back them up somehow.
I already did, why do you want me to repeat myself? Read lwn's security section where dozens of security flaws in all kinds of open source software are being published every week. Open Sourcing your software doesn't magically make it secure, cope with it.

> When you make such claims, you have to back them up somehow.
I read it here:
http://www.spiegel.de/spiegel/0,1518,634334,00.html
If you don't understand german -- tough.

gmail is frustrating too...

Posted Mar 12, 2011 8:34 UTC (Sat) by jthill (subscriber, #56558) [Link]

code review is hard and tedious and _very_ few people are likely to do it

Your characterizations imply things your facts contradict.

And for any comparison of counts to be valid, you'd have to argue that Microsoft publicly lists bugs as security flaws using (even remotely) the same criteria as the projects you're comparing against.

gmail is frustrating too...

Posted Mar 13, 2011 14:20 UTC (Sun) by henning (guest, #13406) [Link]

> I read it here:
> http://www.spiegel.de/spiegel/0,1518,634334,00.html
> If you don't understand german -- tough.

Well, the article is from summer 2009.. And Spiegel is IMHO not a good source for informations about technology and open source.

gmail is frustrating too...

Posted Mar 11, 2011 7:36 UTC (Fri) by rilder (guest, #59804) [Link] (6 responses)

"You only have to take a look at lwn's security section to see that Linux-based operating systems have just as many security flaws as Windows. "

I was wondering whether anyone would misunderstand those. You may be the first one. The idea in Linux/BSD is that these vulnerabilities are patched as soon as they are discovered and deployed. Making all the vulnerabilities open helps in discovering them sooner, exposes them to more eyes and gets patched sooner. There are daily security updates on any linux distro.

Also, considering Linux security model, a single application vulnerability affecting a single point on an attack surface is unlikely to compromise the whole system enough to make it send spam or lock up the whole system to make it a botnet.

I presumed that after reading lwn for this long, people would be slightly more aware of these aforementioned basic assumptions about Linux, FOSS in general. But you never know ;).

Also, many distros now include Selinux/Grsecurity and prevention against stack smashing, RELRO, PIE and so on, the vulnerabilities in that page are on the assumption that these are non-existent. So, do your research before you comment.

gmail is frustrating too...

Posted Mar 11, 2011 8:27 UTC (Fri) by spaetz (guest, #32870) [Link] (1 responses)

>Making all the vulnerabilities open helps in discovering them sooner,
>exposes them to more eyes and gets patched sooner. There are daily
>security updates on any linux distro.

Right, many botnets are being installed through social engineering, users locally installing poisoned malware or clicking otherwise on crap. I don't see how the choice of operating system changes that.

> Also, considering Linux security model, a single application
> vulnerability affecting a single point on an attack surface is unlikely > to compromise the whole system enough to make it send spam or lock up
> the whole system to make it a botnet.

Compromising a single user is enough to send spam from that user account and run a botnet while that user is logged in. Most of the time you don't need to compromise a whole system to cause havoc, run a phishing operation or DDOS other sites. Compromising a single user on a single-user desktop is enough, independent of the underlying OS.

gmail is frustrating too...

Posted Mar 11, 2011 9:56 UTC (Fri) by cate (subscriber, #1359) [Link]

I agree, and none of the security models in Linux corrects the PHP programmers.

IMO the spam and botnet problems don't depends only on using the wrong OS.

gmail is frustrating too...

Posted Mar 11, 2011 11:26 UTC (Fri) by HelloWorld (guest, #56129) [Link] (3 responses)

> I was wondering whether anyone would misunderstand those. You may be the first one. The idea in Linux/BSD is that these vulnerabilities are patched as soon as they are discovered and deployed. Making all the vulnerabilities open helps in discovering them sooner, exposes them to more eyes and gets patched sooner.
I don't believe this works. Reviewing code for security flaws is hard, and unlike writing new code, it's also tedious. Very few people are likely to do it unless they get paid for it.

> There are daily security updates on any linux distro.
If Linux-based operating systems were more secure than Windows, there'd be no need for daily security updates.

> Also, considering Linux security model, a single application vulnerability affecting a single point on an attack surface is unlikely to compromise the whole system enough to make it send spam or lock up the whole system to make it a botnet.
That's outright bullshit, compromising a single user's account is totally sufficient for sending spam.

> Also, many distros now include Selinux/Grsecurity and prevention against stack smashing, RELRO, PIE and so on, the vulnerabilities in that page are on the assumption that these are non-existent.
Do you actually think that current Windows versions don't include comparable technology? Do your research before you comment.

gmail is frustrating too...

Posted Mar 11, 2011 20:24 UTC (Fri) by rqosa (subscriber, #24136) [Link] (2 responses)

> Very few people are likely to do it unless they get paid for it.

> If Linux-based operating systems were more secure than Windows, there'd be no need for daily security updates.

First you say that open source software is less secure because not enough developers are making vulerability fixes, and then you say that open source software is less secure because vulnerability fixes are published too often? You're contradicting yourself.

gmail is frustrating too...

Posted Mar 11, 2011 21:37 UTC (Fri) by HelloWorld (guest, #56129) [Link] (1 responses)

First you say that open source software is less secure because not enough developers are making vulerability fixes
I never said anything remotely like that. What I said is that the supposed advantage of having more people that review the code (as claimed by rilder) doesn't exist in the real world, as people don't review open source code just for the fun of it. Heck, high-profile projects like the GIMP even have trouble finding developers who write the code. And just to make this clear: I also don't believe that closed source software is more secure or anything like that. Security just has nothing to do with whether the source is available or not.
and then you say that open source software is less secure because vulnerability fixes are published too often?
Again, I never said anything like that. I said that if open source software were inherently secure, there'd be no need for security fixes (duh).

gmail is frustrating too...

Posted Mar 12, 2011 7:27 UTC (Sat) by rqosa (subscriber, #24136) [Link]

> the supposed advantage of having more people that review the code (as claimed by rilder) doesn't exist in the real world, as people don't review open source code just for the fun of it.

If there are in fact "daily security updates", then that implies that there are people reviewing the code and finding vulnerabilities.

Remember this nVidia driver vulnerability that didn't get fixed until a long time after it was known?

> I said that if open source software were inherently secure, there'd be no need for security fixes (duh).

Obviously a piece of software that has no vulnerabilies needs no security fixes (and since it's obvious, there's no point in saying it); but that's not what you said above. You said that "If Linux-based operating systems were more secure than Windows, there'd be no need for daily security updates". That suggests that you're claiming that Windows has fewer vulnerabilities, and citing as evidence for that the lower frequency of Windows security fixes. But that evidence doesn't support the conclusion, because:

  1. With most proprietary software, no one other than the copyright owner can make fixes, even for known vulnerabilities like the aforementioned nVidia driver one. (I say "most" because maybe some "source-available" proprietary licenses, like the original Qt license or the old Pine license, would allow for other people to make security fixes.)
  2. Unlike Windows, most Linux distributions include a lot of software other than operating system components, thereby inflating the amount of security fixes per-distribution.

gmail is frustrating too...

Posted Mar 13, 2011 16:47 UTC (Sun) by jrigg (guest, #30848) [Link]

>The reason why malware is written for Windows is that writing it for Mac OS or Linux isn't worthwhile due to their low market share.

I suspect the fact that for years most Windows systems had no way of setting file permissions to prevent executable programs from being installed merely by opening an email attachment or clicking on a web link is also of some relevance.

Email is far from broken

Posted Mar 10, 2011 21:34 UTC (Thu) by job (guest, #670) [Link] (6 responses)

That's just false. Most businesses does not use Google or Microsoft servers but their own server or their local ISP's.

I've run my own SMTP for over a decade and I've never had a single problem sending email to Google, Yahoo or any of the other big email providers (with Hotmail as the sole exception). Obviously my box is unknown to them but they still do not reject my mail.

On the incoming side a standard Spamassassin configuration throws away thousands of mail each day for me with very few to none false positives. With Spamassassin and bayesian filtering when necessary, spam is a solved problem. Email is far from broken.

Email is far from broken

Posted Mar 10, 2011 22:04 UTC (Thu) by jwb (guest, #15467) [Link] (3 responses)

I suppose you have done copious amounts of market research to back up your claim?

Email is far from broken

Posted Mar 10, 2011 22:13 UTC (Thu) by job (guest, #670) [Link]

Which claim? That most businesses use their own or their ISP's mail servers?

Actually yes, email is part of what I do for several customers and I tend to read their logs. If you think webmail took over, you will be shocked to discover how many companies use Notes or Exchange.

Email is far from broken

Posted Mar 11, 2011 16:07 UTC (Fri) by michel (subscriber, #10186) [Link]

Perhaps the same copious amount of market research you did to come to the conclusion of '"Small businesses who just want to run their own mail server" are predominantly idiots'. Your posts here are insightful, but it would be nice if those small business owners who are not idiots have a way to understand how to configure their email in such a way that the larger player will accept and handle the mail correctly. Of course, those large players certainly give the impression that they have no interest in that.

I certainly don't consider the folks at LWN.net to be idiots, so they must be part of the minority I guess.

Email is far from broken

Posted Mar 18, 2011 13:32 UTC (Fri) by jschrod (subscriber, #1646) [Link]

Well, the latest Gartner report that I've seen on this topic is from 2010; and then 15% of US companies outsourced their email. Last year I read a Forrester report about the situation in Europe, there it was even lower. (That's in line with overall outsourcing numbers that are significantly lower in Europe.)

So, copious amounts of market research don't agree with your opinion.

That may be because you have obviously lots of technical experience how to handle large numbers of private email accounts. Equally obviously you have no business experience. And that's no wonder -- I'm an CEO, and I would never let a staff member with your behaviour near any customer of mine.

Just FYI: People who call their potential customers idiots are not appreciated in the business world. My company does IT consulting, especially in outsourcing management. Your behaviour here is not a good example for the type of service that one has to expect from Gmail engineers in case of problems and speaks against using that outsourcing provider.

Email is far from broken

Posted Mar 11, 2011 2:26 UTC (Fri) by fandingo (guest, #67019) [Link] (1 responses)

Same experience here. I've setup Nagios to email my gmail account, without problems. Just last nightI setup a Redmine server on my home ISP and no problems at all.

I think that it's because Google does email filtering right and isn't overly dependent on source address/domain.

Email is far from broken

Posted Mar 11, 2011 12:58 UTC (Fri) by knobunc (guest, #4678) [Link]

They do it right... until your mail mysteriously stops going through. And there is no way to work out why they have decided you aren't cool. There are no contact addresses that I could find, and posting to their forums is the suggested recourse, and that usually is as good as catting into dev null.

And then... your mail may start going again. It's all rather frustrating.

gmail is frustrating too...

Posted Mar 11, 2011 5:50 UTC (Fri) by cmccabe (guest, #60281) [Link]

I have to give you credit for being honest, but the picture you're painting of the hosted email business is not a pretty one. Sounds like a lot of small businesses are effectively locked out while a few big players consider dropping valid email no big deal as long as it keeps their servers lightly loaded...

gmail is frustrating too...

Posted Mar 10, 2011 19:55 UTC (Thu) by knobunc (guest, #4678) [Link] (2 responses)

I've had this domain for 16 years and have been sending email for me and my friends who have accounts (100ish accounts total and about 30 active email addresses) for all of that time. No one sending from my machine, or using email addresses from my domain have ever sent Spam out. There have probably been a few wide-distribution mails that someone may have flagged as spam, but I obviously haven't watched all of the outgoing mail. There certainly have not been emails to postmaster@ or abuse@.

In that time I have had four static IP addresses (from three different providers). The most recent one is Verizon, and I've had the current IP address for over two years. And it is not in VZ's dhcp pool...

Precisely how much "reputation" must I amass before I am allowed to send to the hallowed gmail addresses?

My mail peeve is that gmail makes it very hard to work out if you are blocked and if so, why. I can no longer tell if my mail is blocked (without signing up for a separate gmail account) since I flagged mail to my gmail address from my domain as "good". So I can no longer tell...

gmail is frustrating too...

Posted Mar 10, 2011 21:15 UTC (Thu) by lutchann (subscriber, #8872) [Link] (1 responses)

There's really no way to use "test" accounts to see if Gmail is filtering you. It seems like there's a different threshold for each user, so for some users you'll end up in their inbox and for other users you'll end up in their spam folder, even when sending more-or-less the same email from the same outgoing mail server.

(I'm basing this theory on "cold-call" emails I send to people I meet at trade shows, conferences, etc, who I've never emailed before and therefore wouldn't yet have added me to their address book/personal whitelist.)

In fact one Gmail user tells me all my messages still end up in his spam folder, even though he has added me to his address book and clicked the "not junk" button on all my messages. I just don't get it.

gmail is frustrating too...

Posted Mar 11, 2011 7:16 UTC (Fri) by hozelda (guest, #19341) [Link]

If it is true that so few people notice such a problem with gmail, the most likely possibility is that the reason has little to do with gmail and that the error lies elsewhere. I think the hotmail case had lots of users complaining.

A note for Hotmail users

Posted Mar 10, 2011 21:16 UTC (Thu) by wtogami (subscriber, #32325) [Link]

How does sendmail.com decide on a reputation rating?
How can the general public query this for their own private or small business MX?

A note for Hotmail users

Posted Mar 10, 2011 21:20 UTC (Thu) by wtogami (subscriber, #32325) [Link]

I'm not sure how sendmail.com would decide upon a 60% out of 100% reputation. http://multirbl.valli.org/lookup/70.33.254.29.html for example shows lwn's outgoing MTA as completely clean.

A note for Hotmail users

Posted Mar 10, 2011 21:24 UTC (Thu) by wahern (subscriber, #37304) [Link] (9 responses)

I presume you used this database?

http://www.sendmail.com/sm/resources/tools/ip_reputation/

I just ran the query against 209.85.212.52, the most recent gmail.com sender in my inbox. It says the weighted risk was 68%, where 100% is a blacklisted IP.

I co-locate my own server and my weighted risk was 61%.

(NOTE: You have to enable HTTP Referer headers. I normally keep that disabled in my browser, and puzzled over some confusing error messages for a bit.)

A note for Hotmail users

Posted Mar 10, 2011 21:42 UTC (Thu) by jwb (guest, #15467) [Link] (8 responses)

A note for Hotmail users

Posted Mar 10, 2011 22:23 UTC (Thu) by job (guest, #670) [Link] (5 responses)

... which lists LWN just as safe as GMail.

That sort of disproves the original point that LWN would have bad reputation.

A note for Hotmail users

Posted Mar 10, 2011 22:27 UTC (Thu) by jwb (guest, #15467) [Link] (4 responses)

Are you sure you clicked on the right links? For me, it shows the Google IP as 97, the LWN IP as 58 ("High risk").

A note for Hotmail users

Posted Mar 10, 2011 22:54 UTC (Thu) by job (guest, #670) [Link] (3 responses)

I entered lwn.net as opposed to tex.lwn.net since I do not know which outbound mail server LWN uses. It was given a rating of 99, nota bene better than a lot of GMail. That the reputation differ that much may put the relevancy of the data in question, but either way it pretty much disproves your point that individual servers should not be trusted. They are, and it works.

A note for Hotmail users

Posted Mar 10, 2011 23:43 UTC (Thu) by jwb (guest, #15467) [Link] (2 responses)

It doesn't put the relevancy of the data in question at all. It simply means that the IP block of tex.lwn.net contains some spammers, while the IP block of lwn.net does not.

It's one of the major risks of shared hosting environments. You don't have complete control over the reputation of your IP. IPv6 should fix this by making the attribution of addresses to organizations much more granular.

A note for Hotmail users

Posted Mar 11, 2011 7:24 UTC (Fri) by joern (guest, #22392) [Link] (1 responses)

> It's one of the major risks of shared hosting environments. You don't have complete control over the reputation of your IP. IPv6 should fix this by making the attribution of addresses to organizations much more granular.

I cannot quite follow your reasoning here. Today we have a 32bit address space. Even if it were fully populated, with a bit of trickery we can store the reputation in 8 Bytes or so, resulting in 32GB total. That will easily fit on a hard disk, an ssd or, if absolutely necessary, into RAM. Yet, somehow, reputation does not work on IP basis but some number of IPs get bunched into "blocks" and reputation is shared among the block.

What makes you think that with a 128bit address space, things will improve? If programmers of reputation systems are clueless today, how will IPv6 make them smarter (or make the task for their poor overworked brains easier)?

A note for Hotmail users

Posted Mar 12, 2011 20:48 UTC (Sat) by marcH (subscriber, #57642) [Link]

> Yet, somehow, reputation does not work on IP basis but some number of IPs get bunched into "blocks" and reputation is shared among the block.

Maybe you missed the real reason for this aggregation.

A note for Hotmail users

Posted Mar 10, 2011 22:33 UTC (Thu) by dskoll (subscriber, #1630) [Link]

We run our own reputation list based on this software and protocol and the LWN server hasn't been seen enough to be statistically significant. The small amount of reputation data we do have, however, indicates that it's clean.

That's a lot better than the average Hotmail or Yahoo outbound server.

A note for Hotmail users

Posted Mar 13, 2011 20:00 UTC (Sun) by nhippi (subscriber, #34640) [Link]

Senderbase otoh seems to give "good" reputation for *.lwn.net.

http://www.senderbase.org/senderbase_queries/detailip?sea...

when looking at the network, there is two "bad" listings in the same /24 but in general seems like good "neighborhood".

A note for Hotmail users

Posted Mar 14, 2011 2:29 UTC (Mon) by i3839 (guest, #31386) [Link]

Okay, then I have a complaint for you:

It's way too easy for bots to create and use gmail accounts, which are
then used to create bot accounts on forums and used for spamming.

Please randomize the sign-up page totally for every new user, so it's
harder for bots to automatically create an account. Just captcha's is
not enough, have a text box where they have to write a specific text
or a checkbox they need to tick or not.

Also send them an email where they have to do something before they
can send or read other emails.

A note for Hotmail users

Posted Mar 10, 2011 19:14 UTC (Thu) by pheldens (guest, #19366) [Link]

It's easiest to just switch outgoing mta IP if you have spares.

A note for Hotmail users

Posted Mar 10, 2011 22:34 UTC (Thu) by jiu (guest, #57673) [Link] (5 responses)

to Jon: I'd be interested to know what percentage of LWN subscribers use hotmail. I can imagine these people as either:
- very conservative, change-averse, resilient
- scared of disclosing their real email
- young and uninformed
otherwise, why?

A note for Hotmail users

Posted Mar 10, 2011 23:00 UTC (Thu) by marrusl (guest, #67123) [Link] (2 responses)

There's also just being old.

I didn't use it for LWN, but I still have a Hotmail account from before Microsoft bought them. They were one of the first web-based email providers and also one of the first free ones.

I still use it as kind of my 3rd level account, behind my company and real personal email addresses.

Fwiw, Hotmail was a Solaris/FreeBSD shop when they were bought by MS. There were stories at the time, perhaps apocryphal, about just how much trouble MS had trying to port everything to NT. I do believe it took a while at the very least.

A note for Hotmail users

Posted Mar 11, 2011 19:44 UTC (Fri) by oblio (guest, #33465) [Link] (1 responses)

Stories? You mean official documents? :)
http://technet.microsoft.com/en-us/library/bb496478.aspx

A note for Hotmail users

Posted Mar 11, 2011 19:53 UTC (Fri) by oblio (guest, #33465) [Link]

A note for Hotmail users

Posted Mar 11, 2011 15:35 UTC (Fri) by pzb (guest, #656) [Link]

I recently ran the percentages for one of the sites I help run (susestudio.com). It isn't as technical as lwn.net, but is not a general interest site either.

62.7% Gmail by Google
 9.6% Yahoo Mail
 6.1% Hotmail
 1.1% GMX

All other providers are below 1%. I did normalize some of the domains (GMail is googlemail.com in some countries; Hotmail has live.*, hotmail.*, msn.* and windowslive.* for a bunch of TLDs; etc)

We haven't had issues with being blocked, but right now we only send transactional email (that is one email at a time, based on user action).

A note for Hotmail users

Posted Mar 12, 2011 2:49 UTC (Sat) by tonyblackwell (guest, #43641) [Link]

Use it while on international travel as a low-security easy to use account, accessed from public terminals. Use it creating accounts for some websites where don't want to give them my main email and where I judge it's at risk of incurring spam.
White hair, but not change-averse!

Curious to see "very conservative, change-averse and resilient" in the same group! Do these all mesh well?

Not an isolated problem

Posted Mar 11, 2011 2:31 UTC (Fri) by mikov (guest, #33179) [Link]

Our email service (hosted by SendGrid) also had a problem with Hotmail in the beginning of the week, so perhaps it is a universal problem affecting everybody.

FYI, this is the error we were getting:
550 SC-001 Unfortunately, messages from <our-dedicated-ip> weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

A note for Hotmail users

Posted Mar 11, 2011 2:58 UTC (Fri) by tdwebste (guest, #18154) [Link] (3 responses)

Why I have more than one email address. Because I use some to give away when I need to provide an email address. Which means I have gmail, yahoo, hotmail, mail.com, .... email addresses.

Having one email address is like having one user account. A dangerous foolish idea. After a dirty website to infects your account with a logger are you going to use that same account for your banking transactions?

A note for Hotmail users

Posted Mar 11, 2011 14:10 UTC (Fri) by mjthayer (guest, #39183) [Link] (2 responses)

> Why I have more than one email address. Because I use some to give away when I need to provide an email address. Which means I have gmail, yahoo, hotmail, mail.com, .... email addresses.

There are enough sites doing one-time e-mail addresses not to need this - unless you want semi-throw-away, as in it isn't terrible if the address gets out but there might still be interesting stuff coming in.

A note for Hotmail users

Posted Mar 11, 2011 19:18 UTC (Fri) by rahvin (guest, #16953) [Link] (1 responses)

I thought that's what aliases were for. I run my own postfix server and simply setup aliases when giving addresses to people. Spam starts hitting the alias and it gets deleted and the mail is rejected. My real email address is never given out.

A note for Hotmail users

Posted Mar 13, 2011 17:33 UTC (Sun) by jrigg (guest, #30848) [Link]

I have a few web mail accounts for semi-throw-away stuff. For more important stuff I use aliases on my own server, which get changed when spam starts getting sent to them. Hardly anyone gets my real email address.

I had a Hotmail account for email which I thought would attract spam, but stopped using it some time ago when it stopped accepting new messages from my Linux system.

A note for Hotmail users

Posted Mar 17, 2011 16:20 UTC (Thu) by RogerOdle (subscriber, #60791) [Link]

What bothers me is that when the community started talking about replacing the email system with a more secure mechanism, the first thing the big companies did was run out and patent every lame idea they had. They would only get behind the reform movement if they could put themselves in a position to collect a per-email fee. The result was that the idea of real email reform died at the hands of software patents.

Is it really possible to fix email in this environment?


Copyright © 2011, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds