A SpamAssassin surprise

Back in April, an important anniversary went mostly unremarked: it was a full 30 years since the sending of the still-famous "green-card spam". Those of us who were recipients of that missive were horrified at the time, but nowhere near as horrified as we would have been had we known what was coming. Almost overnight, email was transformed from a collaborative service to a pipeline that would be constantly abused by ill-meaning actors. Spam now dominates the email stream; this problem has driven many other changes to email, not all of which are good.

For most of the last three decades, anybody wanting to run their own email system has needed some way of filtering out spam — at least, if they want that email system to actually be useful. There are a number of tools out there that can take on that task, but many of us rely on Apache SpamAssassin to keep the trash out of our inboxes. SpamAssassin combines a long list of heuristics (does the email start with "Dear beneficiary"? does it have a strange date? does it reference large amounts of money?) with a trainable Bayesian filter and the ability to query reputation services ("blocklists") on the Internet. It does not do a perfect job, but a well-trained SpamAssassin installation will identify most spam with a negligible false-positive rate.

Recently, though, vast amounts of obvious spam started evading SpamAssassin and landing in our inboxes. The investigation provoked by that onslaught turned up an interesting failure mode for SpamAssassin — one that other users may want to be aware of.

Validity confusion

A close look at the spam that got through revealed, along with numerous opportunities to get rich, deal with chronic health issues, or make improbable anatomical changes, that a couple of rules (RCVD_IN_VALIDITY_CERTIFIED and RCVD_IN_VALIDITY_SAFE) were firing in all of them. These rules are meant to fire if the machine that presented the message under consideration appeared in a couple of reputation lists maintained by a company called Validity. These are not blocklist databases, though; instead, they are allowlists, indicating that the email is highly likely to be trustworthy and should be delivered normally.

SpamAssassin works by calculating a score for each message based on the rules that trigger; each rule will adjust the score up or down. These two rules, combined, will reduce an email's spam score by five points. Five is also the number of points required for an email to be judged as spam in the default SpamAssassin configuration; if all emails start triggering those rules, that threshold is effectively raised to ten points. The result is that a lot of email that would otherwise have been properly classified as spam now looks legitimate; you really did have an unknown rich uncle whose estate can be yours if you just pay a small advance fee.

One can imagine all kinds of unpleasant outcomes that can result from giving a private company the ability to make spammy email look legit. SpamAssassin has also had an occasional history of introducing negative-score rules that have been promptly exploited by spammers. So it was easy to jump to the conclusion that, somehow, spammers had learned how to appear in the Validity allowlist databases and were taking full advantage of that ability. But that conclusion was incorrect.

It seems that, beginning in March, Validity introduced a limit on the number of queries that any given site is allowed to make to its databases. Some investigation confirmed that LWN's email server had, indeed, exceeded that limit and been cut off. The message returned by the server was clear enough:

  28.88.79.45.sa-trusted.bondedsender.org. 10 IN TXT \
     "Excessive Number of Queries | \
     https://knowledge.validity.com/hc/en-us/articles/20961730681243"

SpamAssassin, however, did not read that explanation; instead, it took the presence of a reply as an indication that the site in question appeared in the list, and duly decremented the spam score of the message accordingly. That resulted in us having to clear out our inboxes with snow shovels.

There are a number of ways that anybody encountering this problem can address it, once it has been identified. Configuring SpamAssassin to use a local, caching DNS server will reduce blacklist queries and is the recommended path, but neither Debian nor RHEL derivatives ship with that configuration. Validity will allow more traffic from sites if an account has been set up for them. Or one can simply configure SpamAssassin to not query the Validity lists at all. That last option was taken, and our inboxes quickly returned to a more rational state. Though "rational" is a relative term for any inbox that is the target of a linux-kernel subscription.

Email in the 2020s

Running an email server requires configuring and maintaining several complex, security-sensitive pieces of software. In the absence of a full-time administrator, whoever handles that task has to get by with an imperfect understanding of how all of those components work. It is probably a fair guess that a large proportion of SpamAssassin users do not even know that the Validity lists exist, much less that their system is trusting those lists to make decisions about email delivery. SpamAssassin nicely hides the fact that its users are depending on an outside company's generosity and integrity to manage their email stream — and that works well most of the time.

In LWN's case, that configuration was introduced as part of a hurried email server replacement driven by the demise of CentOS 7. A suitably diligent administrator would have learned about the use of this service and made proper decisions about the configuration of the spam filter, but that is not what happened here. And everything quietly worked as expected, until it didn't. Even running our own system, we found ourselves having trusted a company we had never heard of, which had been given the ability to essentially bypass spam filtering.

That company chose to limit the availability of this service. That is within Validity's rights, even though it advertises that its lists are used in SpamAssassin's default rules; no promise was ever made that this free service would remain available. SpamAssassin's blocklist policy allows the inclusion of rules that use "free for some" services, since they work for a large subset of users. Anybody wanting to disable the use of blocklists entirely can do so with the skip_rbl_checks configuration parameter. One could argue with these policies, but they are at least clear.

What has always been less clear is the justification for negative-score rules in SpamAssassin in the first place. They are a tempting target for spammers, many of whom are certainly on the lookout for ways to get email past SpamAssassin, and they have created problems in the past. Decades of experience suggest that false-positive results from SpamAssassin are rare; it is not clear that there is a need for rules designed to reduce them. If those rules do exist, though, they should not fire in cases such as this one, where the expected condition is not present; a bug report has been duly filed.

There is arguably a case for the existence of reputation services for email; the fact that they are widely used attests to that. But we are essentially entrusting one of our important communication channels to unaccountable third parties with predictable results; anybody who has run an email server for a while knows the sinking feeling that comes with having to request removal from another blocklist — again. These services have been part of a dynamic that makes it increasingly difficult for small sites to get their email delivered at all.

All told, it not surprising that many people who would have once run their own mail systems have decided to throw in the towel and use one of the big providers instead. What was once a nice, federated service (in a much simpler time) has become a complex mess with high barriers to entry, pushing users into a small number of large, proprietary services. Email was not designed for these times, and we still lack something better to replace it. So those email administrators who stubbornly hang on to their own installations will continue to have to deal with glitches like the one described here; we will add another piece of duct tape to the system and continue to muddle along as well as we can.