LWN: Comments on "A SpamAssassin surprise"

collaborate on email server maintenance/setup?

Klavs — Fri, 15 Nov 2024 15:15:17 +0000

We use mailcow - its released as open source docker images.. so you can easily update - and they maintain the configs for battleing spam etc. Seems to work well.

Before that we've used Kolab - which sadly went unmaintained.. But with Graylisting alone - we've had very little spam come though actually.

better use spamassassin to block the query only if necessary

jahlives — Tue, 22 Oct 2024 16:22:50 +0000

I think it's better to dynamically disable the rule, depending on the return code to the query of the A record. As a unique return value is given if the query limit is exceeded. Although I do not know from which exact spamassassin version this feature is supported. In my case I simply added

header RCVD_IN_VALIDITY_BLOCKED eval:check_rbl('zen-lastexternal','sa-trusted.bondedsender.org.','^127.255.255.255$')
describe RCVD_IN_VALIDITY_BLOCKED ADMINISTRATOR NOTICE: The query to sa-trusted.bondedsender.org was blocked
tflags RCVD_IN_VALIDITY_BLOCKED net
reuse RCVD_IN_VALIDITY_BLOCKED

and then a

if can(Mail::SpamAssassin::Conf::feature_dns_block_rule)
dns_block_rule RCVD_IN_VALIDITY_BLOCKED sa-trusted.bondedsender.org
endif

which let spamassassin "ignore" the result of the lookup

Oct 22 18:06:16.337 [623] dbg: dnseval: checking RBL sa-trusted.bondedsender.org, set zen-lastexternal, rule RCVD_IN_VALIDITY_BLOCKED
Oct 22 18:06:16.337 [623] dbg: dns: launching rule RCVD_IN_VALIDITY_BLOCKED, set zen-lastexternal, type A, subtest (?^aa:^127.255.255.255$)
Oct 22 18:06:16.337 [623] dbg: async: blocked by dns_block_rule: A/REDACTED.sa-trusted.bondedsender.org, rules: RCVD_IN_VALIDITY_BLOCKED

I'm using spamassassin 4.0.1 so maybe this feature is new in the 4.X SA. But the steps above can be repeated for any list that returns a unique code for excessive querying

Badly coded websites

Wol — Wed, 02 Oct 2024 10:34:21 +0000

Actually, I would have thought it would break for every CV2 >= 100. M can't be 0, but YY could be anything.

And why use YYYY for user interaction when all dates are "about now"? Credit card dates are all within a 10-yr windows, so YY is not ambiguous.

Cheers,
Wol

Badly coded websites

farnz — Wed, 02 Oct 2024 10:12:46 +0000

The CV2 is three digits; I had a CV2 (at the time) of 821, and the site rejected this on the grounds that this "must" be the issue date of my card, not the CV2.

I assume that they were looking at the last 2 digits, and deciding if it looked "too close" to a plausible expiry or issue year in order to determine if the CV2 was valid, but the error message told me that it was rejecting it because I'd supplied the issue date, not the CV2. Experimentally, it also rejected 828 as my card expiry date (even though I had put in the actual expiry date elsewhere in the form).

Badly coded websites

taladar — Wed, 02 Oct 2024 08:22:25 +0000

> "CV2 can't be interpreted as a date in the format M/YY"

Wouldn't that break for the last three months of every year? Not to mention that 4 digit years should really be used for everything by now.

Effectiveness of greylisting

farnz — Tue, 24 Sep 2024 10:36:53 +0000

I'm using exim, not postfix, in part because exim's ACLs are more capable than postfix's filtering options. I use the acl_smtp_data ACL to do my decision about greylisting.

Downside is that you've paid the full cost of receiving the mail by the time you get here; upside is that I can base the decision to greylist on the contents of the mail, not just hosts and headers. And it's important if you're doing this that you record which mails are retried from a different IP, and whitelist the original sender, not just the new IP - otherwise you're creating pain for yourself later. I find that this exim config snippet is a good example of how to start doing this.

Effectiveness of greylisting

mgedmin — Tue, 24 Sep 2024 09:56:02 +0000

This sounds interesting. How have you hooked it up?

I'm currently using postfix + postgrey + amavis for spamassassin/clamav, and I don't quite see a way of making the postgrey conditional on spamassassin's results, given my understanding on how these all are hooked up together.

The problem is greater than just spam

fest3er — Thu, 19 Sep 2024 14:00:31 +0000

A few thoughts:

Was it a SA logic error (any response means the addr is OK)? or was it Validity documentation error (no response means not OK, any response means it's OK)? Said differently, did Validity properly document their API?

gmail is the only 'free' email service that allows ClawsMail to function. MS/Yahoo and others allow only their favorites to work. This further deepened my distrust/dislike of companies that offer 'free stuff'. I run my own email server now; blocking IP addrs I want *zero* contact with is a good start.

Recently, I noticed some spam was coming from cloud hosting services like google, digital ocean, censys, microsoft, and others. I also noticed that some of those services run port scanners (with noble-sounding excuses I mean reasons). Alas, some honest servers also run on them, so it gets a little tricky.

My philosophy is that *any* site that probes my network for services I do not make publicly available is nefarious; I do not want my systems to have *any* contact with such sites. Port scan me? You're out. Send me spam? You're out. Fart in my general direction? You're out. Host pron/warez/etc.? You're out.

IMO, this is one of the reasons I truly dislike end-to-end encrytpion: it prevents network owners (mostly SOHO) from examining data to prevent malware/maldata from entering *AND* leaving their networks. Network owners are responsible for the safety of their networks *and* they have a social responsibility to the internet to keep the internet 'clean'.

Badly coded websites

farnz — Sat, 14 Sep 2024 16:27:03 +0000

The trouble is that, while I've encountered cases that are the payment card industry's fault (Amex CV2 being 4 digits and in a different place on the card, some cards having a 19 digit PAN), I've also encountered a lot that add complexity (like "CV2 can't be interpreted as a date in the format M/YY"). And while I get the temptation to simplify in ways that you think don't matter, adding complexity because you assume things outside the spec doesn't make a huge amount of sense to me.

Badly coded websites

Wol — Sat, 14 Sep 2024 14:09:30 +0000

> First, the spec is easily found and clearly defined - it's not rocket surgery to get the whole thing right (including knowing if the card is a visa or amex or whatever, so don't ask the user!)

It's amazing the number of people who it never occurs to to try and get it right. I've just been asked to implement a check on lorry driver hours, and my FIRST reaction was to ask the requester to send me a copy of the appropriate regulations.

I think this will be the third regulation I've implemented a check for - the previous one was "a driver cannot work for more than 13 calendar days without a 45hr break". The guy who implemented the previous version of the check thought all he needed to look at was the previous, current and next calendar week. How can you check a 13-day rule if at the start of the week you can only see back 8 days! And he just couldn't see what was wrong with it.

Cheers,
Wol

I feel with you

sammythesnake — Sat, 14 Sep 2024 11:38:09 +0000

I always read "Contact us for pricing" as "We assumed you'd ruin away screaming of we told you the price, so we hid it from you"...

Badly coded websites

sammythesnake — Sat, 14 Sep 2024 11:25:23 +0000

I'm in two minds about this one, having implemented the entire spec twice for different employers:

First, the spec is easily found and clearly defined - it's not rocket surgery to get the whole thing right (including knowing if the card is a visa or amex or whatever, so don't ask the user!)

On the other hand, the spec itself is a bear, with 1001 different historical warts, so if you think you find a way to simplify it (which inevitably means being wrong, of course) I can absolutely understand the temptation to do so...

Ideally, the banks etc. would all get together and agree to retire all but one checksum algorithm, standardise on one number length etc. so that the spec can be implemented in half a dozen lines of code, but that's herding cats :-(

test your email hosting

johnjones — Fri, 13 Sep 2024 10:45:14 +0000

you can test your provider

https://internet.nl/

https://checkcybersecurity.service.ncsc.gov.uk/

@corbit

There's a DNS response code for this...

grifferz — Thu, 05 Sep 2024 13:29:43 +0000

> Of course, when it's a free service, the most you can reasonably expect is a full refund of the $0 you paid for it, but it would still be nice if people made a slightly greater effort to comply with standards.

There are actually informal standards for DNSBL return codes to signify thinsg like permissions error or that the DNSBL has shut down. This is necessary because a usual failure mode of a domain is to expire and gain a wildcard DNS for a web landing page, which would otherwise result in the world being listed. Validity did/does use these codes. Unfortunately the tests in SpamAssassin did not check for these and three months of notice wasn't enough time to get the tests modified and into the hands of users.

https://github.com/apache/spamassassin/commit/37f0cf7591d...

Validity did also consult the mailop community about what there were going to do and were advised to return A records in the 127.255.255.0/24 space as is the convention here.

https://www.mail-archive.com/mailop@mailop.org/msg20711.html

The free limit for Validity by the way is 10,000 queries per 30 days. You can easily exceed that if you've left your system using a public resolver (or your local resolvers forward to a public resolver, which is the same result).

I should imagine that LWN does quite a high volume of email and so does need to keep on top of this sort of thing more than an individual or most other small businesses would.

Spamcop was caught out, too

grifferz — Thu, 05 Sep 2024 12:43:12 +0000

The Validity change caught Spamcop out as well. It thought everything was listed so would only report things to bondedsender instead of the actual source.

https://forum.spamcop.net/topic/73357-internal-spamcop-ha...

I feel with you

taladar — Thu, 05 Sep 2024 12:18:27 +0000

On the other hand asking for my email or "Contact us for pricing" are two of the main reasons I leave any possible interaction with your company early on before you even get a chance to convince me your product is worth consideration.

I feel with you

rwmj — Thu, 05 Sep 2024 07:34:22 +0000

I run my own mail server and it's really not that bad. It probably depends on the amount of email sent and received and if you have to handle multiple email accounts though.

selective greylisting

philh — Wed, 04 Sep 2024 08:55:47 +0000

I use http://www.mailavenger.org/ as my SMTP listener (in front of postfix), which provides a lot of nice features (like allowing individual users to script their preferences on how to deal with incoming mail, at SMTP-time).

On the mail server, one can setup user-specific handling under '~/.avenger/'.

The script that deals with my most spam-targeted addresses includes the line:

match -q "*Windows*" "$CLIENT_SYNOS" && greylist

CLIENT_SYNOS is a variable that's populated with mailavenger's guess at what the remote client's OS seems to be, based on SYN packet analysis, so this line results in me only greylisting people who appear to be running windows, which covers approximately all of the spambots, and none of the people I actually want to get mail from.

I've got a lot of other complicated things going on with my own mail scripting (including spamassassin , crm-114, and issuing different addresses to different correspondents etc.), but other users on the system almost all have a very simple setup, and don't have much of an issue with spam AFAIK, while having the chance to tune their anti-spam response if the need were ever to arise for them.

The best thing about mailavenger for me is that it means I have no spam folder. If my systems detect something as spam it gets rejected at SMTP time, and the sender gets to deal with it (the baysian-based SMTP rejections include instructions on how to get past the filters, for the occasional real person who manages to provoke a rejection -- only 3 spammers have ever bothered with that extra step, and now they have a couple of lines of script in their honour ;-) ).

(Hmm, I notice that mailavenger has dropped out of Debian -- I guess I need to work out why and see if I can fix it ... )

I feel with you

khim — Tue, 03 Sep 2024 14:08:09 +0000

Whenever thinks worked that way? Things like !, /, or even @ are in the original RFC, too. Yet web sites don't support them (or, at least, rarely do).

Why do you think + is an exception?

I feel with you

brunowolff — Tue, 03 Sep 2024 13:44:30 +0000

> I say last @ character because "@$foo"@example.com is theoretically a valid form of email address, though I doubt it'd actually work in many places.

This illusrates another issue with email addresses. There are at least three different encodings for them (raw, 821 and 822). "@$foo"@example.com might be a raw address or it might be an 821 or 822 encoding of @$foo@example.com .

I feel with you

dskoll — Tue, 03 Sep 2024 12:55:24 +0000

But when you chased up, and got negative feedback, I presume you spent no further effort following that prospect up?

It depends. Usually we did not. If the negative feedback was about the product itself, we did consider it, and if we felt the criticism was valid, we'd use it to improve the product, and let the prospect know we were doing that, if we thought they'd reevaluate.

I feel with you

Wol — Tue, 03 Sep 2024 07:41:25 +0000

> Gmail is large enough and common enough to force developers to accept these.

That "+" thing is in the original RFC. Gmail didn't force others to accept it - they would have *been* forced to accept it.

Cheers,
Wol

I feel with you

Wol — Tue, 03 Sep 2024 07:38:28 +0000

I'm not denying that. But when you chased up, and got negative feedback, I presume you spent no further effort following that prospect up?

I think you miss the point of what I'm saying - that sort of follow-up - that you were doing - is (a) great at weeding out false prospects (giving you more time to focus where you *were* likely to get results), and (b) handled properly can leave a great feeling in the prospect who is then more likely to cold-call when they see a use for your technology.

I'm not surprised the company did well. (Provided you didn't spoil it some other way :-)

Cheers,
Wol

The origins of spam

jd — Tue, 03 Sep 2024 06:33:32 +0000

IIRC, spam originated from a Utah lawyer firm bombarding USENET with advertising, then selling a how-to book on how to abuse both it and email.

Naivety

marcH — Tue, 03 Sep 2024 05:03:42 +0000

> Those admins didn't just handle the technical side either; they were also responsible for controlling their users and ensuring they abided by the "gentlemen's club" rules. Since access to the systems was tied to employment or enrollment in a University, the admins had a lot of power to enforce the rules.

You just put a smile on my face by reminding me of the admin in my college who was totally out of his depth when the Internet arrived and ridiculed by the gang of l33t students. The only way he managed to get back some level of control was by... allying with them in the end.

But these were college kids == "gentlemen thieves" so your point very much stands.

I feel with you

khim — Tue, 03 Sep 2024 00:21:37 +0000

Gmail did that. In Gmail everything after + is ignored so you may used JoeAverage+shopping@gmail.com on one web site and JoeAverage+work@gmail.com on some other web site (and can use filters to label them differently on the receiving site).

Gmail is large enough and common enough to force developers to accept these.

I'm not sure smaller mail providers can do that.

I feel with you

dskoll — Mon, 02 Sep 2024 23:33:40 +0000

What can I say? We ran the company successfully for 19 years, so our techniques worked.

I feel with you

Wol — Mon, 02 Sep 2024 22:53:41 +0000

But an important part of getting sales is to sell the customer what they want.

One of my friends was a very good salesman. His technique was to go in and ask "What's your problem?". He then followed up with "we have these products that look like what you're looking for".

MUCH more effective than "we have this wonder product, now what's your problem".

And even if you get the feedback "this is a lovely product. I just don't see how it's any use to me", it tells your sales guys to leave alone and don't waste effort. AND THAT CAN GET YOU SALES. On several occasions, I've gone back to a company that's treated me with respect, and taken a previous "no I'm not interested" with good grace. Because I feel I can trust them!

Cheers,
Wol

I feel with you

dskoll — Mon, 02 Sep 2024 22:20:23 +0000

Wol, we didn't want good honest feedback. We wanted to make sales. :)

I feel with you

dskoll — Mon, 02 Sep 2024 22:19:32 +0000

We would have lost many sales that way. Much as many people don't like to admit it, getting in touch with people partway through a trial is a very effective sales technique.

I no longer have any skin in the game (sold the companies ages ago) but do speak from experience.

Naivety

rgmoore — Mon, 02 Sep 2024 16:24:43 +0000

Email was not designed for any specific times: it was designed for a gentlemen's club.

Except that the gentlemen's club atmosphere was tied to a specific time: early in the life of the Internet, back when only a comparative handful of powerful institutions (the government, military, higher education, or big businesses that were connected to government, military, and education) were allowed in. Not only that, but the systems themselves were big and complicated enough to have full-time administrators. Those admins didn't just handle the technical side either; they were also responsible for controlling their users and ensuring they abided by the "gentlemen's club" rules. Since access to the systems was tied to employment or enrollment in a University, the admins had a lot of power to enforce the rules. Of course this was true of all of Internet access, not just email.

The system wasn't perfect. For one thing, it was reactive rather than proactive, and someone could cause a lot of damage before their admin figured out what was going on and reined them in. The Morris Worm was a good example of how much damage a single incident could cause. More importantly, it completely failed in the face of commercial ISPs. The ISPs were being paid by their users rather than employing them, so they lacked the same kind of leverage in the face of bad behavior by their users. The big ISPs got big so fast the traditional Internet group really didn't have a chance to respond before their nice gentlemen's club was destroyed. We've spent the past 30+ years patching all the flaws in the original Internet infrastructure that were papered over by the power of traditional admins.

I feel with you

Wol — Mon, 02 Sep 2024 16:18:24 +0000

That doesn't work if you want good, honest feedback.

Yes, from the customer's point of view, it might suck to be contacted after you've decided you don't want the product. But from the vendor's p-o-v, most customers will not provide feedback unless they are actually interested in the product. So how does the vendor tell the difference between the customer thinking (a) this product is crap, (b) this product doesn't fit my requirements, (c) this product is great but I can't see how we would use it, and (d) the customer employee thinking "that was a nice waste of time getting paid for doing nothing". I'm sure you can think of many others which - without the vendor being able to pro-actively reach out - would all be conflated as "prospective customer never came back".

I've been on the receiving end of that, and once I'd downloaded the pdf, I wasn't impressed. I only responded because I received a chase-up email, and politely said that on reading the pdf, it was clear it would be of no use to my in my job. But without the nice personal chase, they would have heard nothing.

Cheers,
Wol

I feel with you

taladar — Mon, 02 Sep 2024 14:51:09 +0000

That honestly sounds less like a need and more like something your sales & marketing department wanted. You could have just offered a visible way to contact you instead to the customers who are interested in your product after the trial.

I feel with you

dskoll — Mon, 02 Sep 2024 13:59:57 +0000

Yes, people don't understand email. It's very annoying.

Web developers: The only thing you should do to validate an email address is (1) check that it contains an @ character, and (2) check that everything after the last @ character is a domain name that has either an A or an MX record that ultimately points to at least one routable IPv4 or IPv6 address. That's it. Nothing else.

I say last @ character because "@$foo"@example.com is theoretically a valid form of email address, though I doubt it'd actually work in many places.

I feel with you

pbonzini — Mon, 02 Sep 2024 10:26:40 +0000

I suppose that's easily fixed with a better regular expression.

(That's sarcasm, or rather disillusionment, if that wasn't clear).

I feel with you

anselm — Mon, 02 Sep 2024 09:56:44 +0000

I'm generally happy that most places seem to have caught on to the idea that foo+bar@example.com is, in fact, a valid e-mail address in spite of the “+”. This used to be a major annoyance a few years ago.

I feel with you

yeltsin — Mon, 02 Sep 2024 09:34:08 +0000

To give a concrete example, sourcehut uses '~' and '/' in the left part of the address, which breaks some badly written email clients and even some mail providers. Their response is always "wontfix, report it to your email client/mail provider", which is probably for the best.

e.g. ~sircmpwn/sr.ht-packages@lists.sr.ht

I guess it's one more reason we need diversity in both, because it forces everybody to comply with the standards. Gmail is very specific on what you can (or cannot) do with your username; other large providers are similar. If that's the only thing everyone is using, it is the new de-facto standard.

Badly coded websites

farnz — Mon, 02 Sep 2024 08:42:09 +0000

Combine that with an effort to minimise the number of back end errors you see (certainly the case with the e-mail and credit card ones I saw), where you're trying not to send data to the back end if you "know" it's invalid, and you have something that apparently works until you get real users on it.

The trouble is that when you try to fix "mistakes" by the user (and I have no doubt that some users would type their expiry of 1/23 into the CV2 box as 123), you have to ensure that this can't be a genuine user - asking "are you sure - 123 looks like it might be your card expiry" is OK to reduce back end error rates, but "123 is not a valid CV2" is simply wrong, when it's the number on my card.

I feel with you

NYKevin — Mon, 02 Sep 2024 02:29:00 +0000

The other problem is web developers who blithely assume that an email address is suitable as a username. The standards give you basically no guarantees at all about the semantics of the local part (the part before the @). A mail server is fully within its rights to decide that foobar@example.com, FOO.BAR@example.com, and something+completely+different@example.com are all the same mailbox, or all different mailboxes, and can even follow different rules for different mailboxes (perhaps in order to transition to a new set of semantics, while grandfathering old mailboxes to avoid inconvenience to users), as well as change all of these rules whenever it feels like it. The RFCs are completely clear that nobody other than the recipient server should be parsing or interpreting the local part, for the explicit purpose of allowing mail servers to do things like this. So if you use email addresses as usernames, you have accepted the reality that multiple "different" accounts may send notification emails to the same mailbox in some cases. Now you throw in attackers trying to take over an innocent user's account (or maybe just cause annoyance), and the fact that the average user has no idea that this feature exists (and is therefore perfectly happy to click a "confirm your email address" link from a service where they really do have an account), and all sorts of unpleasant things might come of that.

Best of both worlds

kvv — Sun, 01 Sep 2024 22:42:44 +0000

I decided to do both: I run my own mail server, while at the same time I am using the mail-services of the domain hosting provider.

It works as follows:
The provider receives and sends mails for my domain. This is the destination as per mx-record. It means the provider will sign (dkim, spf, etc.) outgoing mails, and deliver incoming messages after spam-filtering to a catchall mailbox. Hence the provider takes care of interfacing with big-tech and their requirements of the day.

My mail system behaves more or less like any other mail-client. On the perimeter I run fetchmail, every minute, to get the mails from the provider's smarthost and send them by smtp to localhost (perimeter) postfix. Then perimeter postfix forwards these messages to internal postfix, which passes them through rspamd (local spam-filter) and let's dovecot do the delivery to the proper mailbox, where the mail-client picks it up.
Outgoing mail is passed from the mail-client to internal postfix to perimeter postfix to provider smarthost and onto the internet.

The one disadvantage is that postfix cannot bounce mails. Mails are considered "delivered" once they are in the provider's catchall mailbox, any attempt to bounce is in fact back-scatter.
This mainly surfaces for mails to a non-existing addressee where it would be helpful to inform a sender, but that is back-scatter, so the mail must be put it in /dev/null and hence we leave the sender in the dark.

The benefits of this mail systems are that there is full control over the user-side of things: personal-mailboxes, aliases, shared-mailboxes, distribution lists, auto-responders, etcetera. While the internet-facing side of mail is taken care of by the domain hosting provider.
While this may not scale to larger sites, it has proven to work fine for smaller setups and it takes away quite some burden of a self-hosted mail system.