Book review: Run Your Own Mail Server
The most common piece of advice given to users who ask about
running their own mail server is don't. Setting up
and securing a mail server in 2024 is not for the faint of heart, nor
for anyone without copious spare time. Spammers want to flood inboxes
with ads for questionable supplements, attackers want to abuse servers
to send spam (or worse), and getting the big providers to accept mail
from small servers is a constant uphill battle. Michael W. Lucas,
however, encourages users to thumb their nose at the "Email
Empire
", and declare email independence. His self-published book,
Run Your Own Mail
Server, provides a manual (and manifesto) for users who are
interested in the challenge.
![Cover: Run
Your Own Mail Server [Cover: Run Your Own Mail Server]](https://static.lwn.net/images/2024/ryoms-cover.png)
Run Your Own Mail Server came to my attention toward the end of its successful Kickstarter campaign. About 20 years ago I decided that hosting my own email was a hobby I no longer wished to pursue. I changed my MX records to an external provider and powered off the long-suffering Pentium Pro server for the last time. Regrets were few.
However, the general message of the campaign ("By running your
own email, you seize control of your communications.
") appealed to
me. Paying for email hosting has freed up some time, and it's
certainly cheaper for one or two users if one believes that their time
is worth money, but at the cost of giving up control. I was interested
in dipping my toes back in the email waters to see if it was time to
resume managing my own mail.
Lucas provided PDF and EPUB versions of the book for this review, and I worked from the PDF. The book is 350 pages and is currently available for about $15 for the digital editions, and about $35 for a paperback edition. Electronic editions are the cheaper and faster way to get one's hands on the title, but I would recommend picking up a print version for anyone serious about running their own mail server. This is the kind of tech book that one would like to annotate, plaster a few sticky notes and page markers in, and so forth. An early draft of the introduction is available on his site.
Who, why, and what
The introductory chapter establishes who should read the book, why they should be interested in running a mail server, and what platforms and tools Lucas will cover. The book is aimed solidly at system administrators, or at least those with substantial administration skills.
Lucas makes clear from the outset that readers are expected to be able to manage their own operating system, and extrapolate from examples rather than follow them blindly. For instance, he notes that he takes his examples from FreeBSD or Debian Linux—so the path to configuration files may differ for readers who are setting up these services on other systems.
Readers are expected to already have an understanding of things like DNS, networking, managing certificates with Let's Encrypt, and more. Or, at least, readers must be willing to become familiar—Lucas provides some recommendations for his own books on system administration as well as Cricket Liu and Paul Albitz's DNS and Bind. If dabbling with DNS records, logging, TLS certificates, databases (yes, plural), user management, and more sounds fun, then this is exactly the book to pick up.
The introduction provides a brief look at the history of email, covers things like Simple Mail Transport Protocol (SMTP), MX records, Sender Policy Framework (SPF), and his choices of software. The book covers using Postfix as the mail transfer agent (MTA), Dovecot as the Internet Message Access Protocol (IMAP) server, Roundcube for webmail, and Mutt as the mail user agent (MUA), or mail client. Often a book's introduction can be safely skipped over, but it's not recommended here, as it contains information and context that will be needed later.
Digging in
The next two chapters look at SMTP, Postfix, and Dovecot in more detail and start the reader on the journey to actually setting up a mail server to send and receive email. Two servers, actually: readers need to have two hosts connected to the internet to properly follow along with the book. This enables readers to test sending and receiving mail before trying to send mail to the world at large (and risk having a misconfigured server wrongly identified as a spam host).
It's easy to find guides online to configure Postfix or Dovecot with a set of explicit instructions one can buzz through in a few minutes. That is not the approach that Lucas takes. Instead, he walks through setting up Postfix and Dovecot while taking the time to explain the various configuration options and commands in a sort of lecture format. This method is generally enjoyable if one wants to know why, as well as how, to do things, but it's not always the most efficient way to convey setup instructions. Then again, if one is seeking efficiency, the most expedient thing to do is pay a provider to manage email and let it be somebody else's problem.
Scaling up
By the end of chapter 3, readers should have a pair of mail servers that can send, receive, and deliver messages to local users in the Maildir format. That is a useful foundation, but hardly what most would consider a functional mail setup—the mail servers are not ready, for example, to work with external clients—so users can only send mail while logged into the servers. Next Lucas turns to addressing virtual domains, IMAP, setting up authentication for users to send mail, and installing the web-based PostfixAdmin tool.
Chapter 3, "Postfix and Dovecot Setup", might have worked better as two chapters that focused on the respective programs. The coverage of Postfix seemed more thorough and organized than Dovecot's. For example, Lucas starts describing Dovecot's local delivery agent, (dovecot-lda) setup in Chapter 3 and then picks it up again in Chapter 4. The instructions, at least on my Debian 12 system, are somewhat incomplete and required further troubleshooting because the service did not have permissions to open the appropriate log files.
No book on running mail servers would be complete without addressing the topic of spam, and much of Run Your Own Mail Server is devoted to the topic in one way or another. Lucas devotes two chapters to his choice of spam filtering tool, Rspamd, which LWN covered in 2017. Given the complexity of Rspamd, compared to setting up Apache SpamAssassin, this seemed like a bit of overkill at first—but it seems to be an integral part of the way that Lucas likes to manage mail.
Chapter 7, "Rspamd Essentials", starts down the path of setting up Rspamd and requisite components. It spends a fair amount of time on the Universal Configuration Language (UCL) used by Rspamd as well as setting up separate Redis instances for Rspamd's configuration and Bayesian data, respectively. (He does note the Redis license change and acknowledges that users may need to migrate to another database down the road.)
The book picks up again with Rspamd in Chapter 14, "Managing Rspamd", after covering SPF records, DomainKeys Identified Mail (DKIM), DMARC, block lists, setting up web mail with Roundcube, and filtering mail using Sieve. The last time I managed my own mail, SPF was in its infancy and things like DMARC didn't exist yet. I found these chapters to be a helpful overview of those topics as well as useful in setting up SPF and DMARC records. The reason Rspamd setup is covered early on, and then set aside for several chapters, is that Lucas also recommends it for use in DKIM signing and verification rather than OpenDKIM. He does double back and cover OpenDKIM setup late in the book for instances where readers might want to set up mail servers to send, but not receive mail, saying that Rspamd is overkill for hosts that won't receive mail.
Chapter 9 covers protocol checks and block lists to reduce
spam that makes it to users' inboxes. It briefly touches on the robustness
principle, also known as Postel's Law after Jon Postel. Lucas
suggests that the principle, "be conservative in what you do, be
liberal in what you accept from others
", favors spammers. He
explains setting up postscreen
to look for telltale signs that incoming connections are from
spambots. This includes comparing against against Domain
Name System blocklists (DNSBLs) as well as DNSWL to identify trustworthy mail
servers. The chapter also discusses more "intrusive" checks such as
looking for non-SMTP commands or improper line feeds. The SMTP
protocol specifies carriage-return with line feed (CRLF), so if a
client sends bare line feeds (LFs) instead, there's a good chance it's
a poorly programmed spambot. Of course, not everything that's outdated
or poorly programmed is a spambot—so Lucas also discusses how to
wave through connections from legitimate organizations that have badly
behaved network devices or ancient mail servers.
Overall, I enjoyed and would recommend Run Your Own Mail Server for those comfortable with managing their own services, and who are willing to put in the work. It should be clear after reading the title that running a mail server is a hobby much like gardening: the work is never "done", it requires constant tending and eternal vigilance against pests. Currently, I'm on the fence about doing the long-term work to migrate back to self-hosting—but much more apt to do so after reading the book.
Readers who are looking for a book that provides a step-by-step set of instructions to be typed out will probably not be satisfied. The book is not well-suited for skimming over in a rush to set up a mail server. Lucas actually seeks to help readers understand what they are doing rather than running through a few steps by rote and hoping commands, syntax, and protocols never change.
The book is also meant to empower readers to be good communal email citizens and run an email server for a small organization, group, or just themselves. It does not cover every topic, or ready a reader to set up email for thousands of users. He touches on a few issues of scale, such as how to send email to many recipients at once (like a newsletter) while avoiding being consigned to spam purgatory by providers like Google, Microsoft, and Yahoo. But routinely handing massive email traffic, splitting out services among multiple servers, or dealing with hundreds or thousands of users is beyond the scope of the book.
Some books are like GPS navigation: set a destination and get what the author believes to be the shortest and fastest route with turn-by-turn instructions. Lucas occasionally takes the reader on the scenic route, sometimes navigating by landmarks instead of highway signs, and occasionally stopping at tourist attractions. One way gets there fast, but the other way builds skill and confidence in navigating solo.
Posted Nov 19, 2024 20:40 UTC (Tue)
by jkingweb (subscriber, #113039)
[Link] (13 responses)
Rspamd tried its best to convince me I'm insane, though. While I adore the results, configuring it was extremely painful, in no small part because the configuration format is never quite consistent with itself. For some things I had to resort to trial and error, which was a bit harrowing. Thus it sounds like it would be interesting to hear what the author has to say on the subject.
I wouldn't criticize people for saying one should leave mail to experts (it _is_ a lot of work to set up properly, and some work to maintain), but I'm glad there are those who go against the grain, and even try to empower others. I certainly would have appreciated something like this when I started out a decade ago.
Posted Nov 19, 2024 21:51 UTC (Tue)
by jasonjgw (subscriber, #52080)
[Link] (10 responses)
Posted Nov 19, 2024 21:58 UTC (Tue)
by dskoll (subscriber, #1630)
[Link] (1 responses)
I run (ugh) Sendmail on my MX host, but that's only because the commercial anti-spam software I spent 18 years of my life working on integrates with Sendmail.
It relays to my IMAP server, which is Postfix/Dovecot. If I were starting out again, I never would have picked Sendmail.
Posted Nov 20, 2024 15:45 UTC (Wed)
by smoogen (subscriber, #97)
[Link]
Posted Nov 19, 2024 22:47 UTC (Tue)
by jkingweb (subscriber, #113039)
[Link] (7 responses)
What I have satisfies me, though. It may be Frankenstein's monster, but it's already assembled and alive, which is the best kind of monster.
Posted Nov 20, 2024 0:05 UTC (Wed)
by willy (subscriber, #9762)
[Link] (6 responses)
Posted Nov 20, 2024 0:08 UTC (Wed)
by corbet (editor, #1)
[Link]
Posted Nov 20, 2024 0:33 UTC (Wed)
by jkingweb (subscriber, #113039)
[Link] (3 responses)
I think the selling point is that JMAP is easier for clients to implement than IMAP, but till such time as servers are common that's a rather illusory benefit.
Posted Nov 20, 2024 2:10 UTC (Wed)
by CChittleborough (subscriber, #60775)
[Link]
I read the RFCs a few years back. It seems to me to be a clever way to synchronize lists of lists between client and server over HTTP using JSON ... but I'm not an expert on this topic.
(Disclosure: I am a very satisfied customer of FastMail, and heartily recommend them to anyone who, like me, wants to never ever have to set up spam filtering again.)
Posted Nov 21, 2024 11:16 UTC (Thu)
by jond (subscriber, #37669)
[Link]
Posted Nov 22, 2024 1:38 UTC (Fri)
by dilinger (subscriber, #2867)
[Link]
Posted Nov 20, 2024 6:25 UTC (Wed)
by zdzichu (subscriber, #17118)
[Link]
Posted Nov 19, 2024 23:05 UTC (Tue)
by gerdesj (subscriber, #5446)
[Link]
The three levels of config looks a bit mad at first but it isn't. Get it right and upgrades are painless. There's the defaults, spelt out in config files that the packages manage. Then there is a tree of configs which will replace an entire component's config and finally another set of configs which only change those settings that are mentioned for a component.
Its really flexible and easy to work with and as I mentioned, makes upgrades a doddle.
Posted Nov 20, 2024 12:48 UTC (Wed)
by leromarinvit (subscriber, #56850)
[Link]
A long time ago, I used to be a happy user of dspam - but that was abandoned at some point IIRC. Looking for a replacement when the time came to move to a newer distro release, I installed Rspamd - but never took the time to understand how to configure it. So for the last 10 or so years, I've been deleting 1-10 spam emails a day. Not too terrible IMHO, some are even somewhat comically funny. That's a very small minority though - most are just the same old boring scams and gone as fast as I can hit delete.
What stops most spam though seems to be the very basic DNS requirements on sending hosts - the HELO name needs to resolve to the IP it's connecting from, and that IP must have a PTR record. Most zombied spam senders don't meet these requirements, while the vast majority of legitimate email servers does. (I remember having to manually allow exactly one host to be able to receive emails from a large, well-known company that shall remain unnamed.)
Posted Nov 19, 2024 22:13 UTC (Tue)
by dowdle (subscriber, #659)
[Link] (13 responses)
Posted Nov 19, 2024 22:25 UTC (Tue)
by jzb (editor, #7867)
[Link] (2 responses)
From the description, this book only has a smallish potential audience Oh, I dunno - I'd say that the potential audience is pretty large. In the tens of thousands, anyway. I think that NextCloud might include most, if not all, of the functionality you're looking for out of a communications system—though I haven't touched it in ages and ages. Not sure if there's a good book on NextCloud setup or not.
Posted Nov 19, 2024 23:28 UTC (Tue)
by Heretic_Blacksheep (guest, #169992)
[Link] (1 responses)
Posted Nov 20, 2024 0:07 UTC (Wed)
by raven667 (subscriber, #5198)
[Link]
Posted Nov 20, 2024 0:21 UTC (Wed)
by raven667 (subscriber, #5198)
[Link] (5 responses)
Posted Nov 20, 2024 4:29 UTC (Wed)
by buck (subscriber, #55985)
[Link] (4 responses)
Without getting too conspiracy-minded, if nobody is maintaining SMTP services as a least common denominator for password recovery, communications from financial services companies, pizza order delivery receipt sending, etc., aside from the big providers who make their profits elsewhere, then we are not far from reverting to the world of Prodigy, CompuServe, and AOL, with no links between our messaging providers and their online communities except what they maybe deign to allow us, perhaps for an interchange fee, and only as long as they can't more directly monetize our online time by keeping our packets in their networks.
That said, I myself am going to have to depend on other people keeping the vision alive, because I am not up to the challenge of not only getting a mail domain on-line, SPF and DKIM (or is it DMAC?) correctly configured, kept off IP block lists, etc., but then having to keep up with the continuing evolution of the tradecraft required to keep mail flowing with as much of the junk as possible being winnowed.
Though, for all the success that free SMTP exchange may have staying around and being vital longer term, i am not ultimately that hopeful. Not unless the administrators who continue in the trade band together like the network operators do and figure out how to keep settlement-free peering alive just because they feel it is the right thing to do. But I just don't see that being sustainable in this case and wonder what the cost of contacting each other that we currently take for granted as being marginally 0 is going to be once the email baseline is no longer there to undercut any provider's designs on charging us to communicate. Brings back memories of toll calling.
OK, maybe this is a wee bit conspiracy-minded.
Posted Nov 20, 2024 14:21 UTC (Wed)
by raven667 (subscriber, #5198)
[Link] (3 responses)
Posted Nov 20, 2024 19:30 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
email is one service that needs to work with all kinds of clients. Corporations that run in-house email servers will be pissed if you stop receiving their emails.
> where you can in theory host your own data but the indexing/search/feed/app services have a huge footprint
No, they're not. I'm self-hosting them on my system. Granted, the setup right now is a pain, but if you just want to follow people and let others follow you, then even an RPi can host it.
If you want to ingest the whole firehose, then it's a different story.
Posted Nov 22, 2024 15:08 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (1 responses)
> email is one service that needs to work with all kinds of clients. Corporations that run in-house email servers will be pissed if you stop receiving their emails.
I don't think a change like that would be instant, but just like the large providers deprecating username/passwords in favor of OAuth or insisting on TLS, if they published a new mail submission and mail transfer protocol the smaller services would update or outsource if they want to keep talking to their customers and suppliers on MS/Google email, and the incentive is for the hosting providers to make outsourcing the easier option, even if that is by making self-hosting more complicated and expensive (costs a larger provider can absorb but a smaller one may not).
>> where you can in theory host your own data [on Bluesky/AT] but the indexing/search/feed/app services have a huge footprint
> No, they're not. I'm self-hosting them on my system. Granted, the setup right now is a pain, but if you just want to follow people and let others follow you, then even an RPi can host it.
I think we are in agreement, you can self-host the data (bluesky-pds) on an RPi but you can't easily self-host the feed generator/relay/app/moderation provided by the Bluesky app, even with the code on the bluesky-social github you don't have a full independent service without relying on bsky.app infrastructure, it's designed by people whose relevant experience is managing a firehose of data processing using VC capital, it might not scale down in the way that personal email as described in RYOMS can. At least that's my understanding, I'm not an expert in AT, but I have read a few whitepapers about its design.
Posted Nov 22, 2024 22:02 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
This is not similar. Very few companies use OAuth exclusively, most support logins/passwords.
> I think we are in agreement, you can self-host the data (bluesky-pds) on an RPi but you can't easily self-host the feed generator/relay/app/moderation provided by the Bluesky app
Uhm. You can, it's not even complicated technically. The fundamental issue is that if you want to do moderation, then you actually have to do moderation.
Posted Nov 20, 2024 5:09 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Nov 20, 2024 12:47 UTC (Wed)
by jzb (editor, #7867)
[Link]
Posted Nov 20, 2024 13:32 UTC (Wed)
by rrolls (subscriber, #151126)
[Link]
I wish I'd had something like this 15 years ago!
Almost literally, too - that was about when I first would have set up various email-related things on my own servers, but it's always been a horribly complicated mess, mostly just does forwarding to Gmail (that I'd really like to get away from) or ISP mail accounts, and I've lost so many emails from having them miscategorised as spam or otherwise just not processed correctly for whatever reason. Trying to sort out any of this properly has always been such a demotivating nightmare, so hopefully this'll be the missing piece of guidance I needed.
E-mail really is the final boss of digital independence.
Posted Nov 21, 2024 22:17 UTC (Thu)
by dankamongmen (subscriber, #35141)
[Link]
Posted Nov 20, 2024 15:36 UTC (Wed)
by Kamiccolo (subscriber, #95159)
[Link] (2 responses)
Btw, the cover of the book (created by Eddie Sharam?) made me burst into laugh, I believe in this kind of work, right joke is a virtue :}
Btw, currently Lucas do seem to be running a Kikcstarter for his next book (or rather collection of FreeBSD Journal Letters Column Entries).
Posted Nov 20, 2024 15:40 UTC (Wed)
by daroc (editor, #160859)
[Link] (1 responses)
We don't do too many book reviews, but we also covered Practical Julia
back in April. So it's the kind of article we run sporadically.
Posted Nov 20, 2024 16:10 UTC (Wed)
by corbet (editor, #1)
[Link]
Posted Nov 21, 2024 4:34 UTC (Thu)
by ranger207 (subscriber, #134731)
[Link] (7 responses)
Posted Nov 21, 2024 10:22 UTC (Thu)
by NRArnot (subscriber, #3033)
[Link] (6 responses)
Are there any other such projects? Also who is maintaining docker-mailserver (because from bitter experience if there are any exploitable weaknesses in a mail server setup, some slimy spammer will exploit them). Hopefully some relatively large and permanent organisation, rather than "one guy working without reward from his bedroom in Nebraska" (XKCD, maybe mutated through my memory)
Bitter experience drove my employer into the arms of Google many moons ago, but I always wonder what we would do if Google turned evil?
Posted Nov 21, 2024 11:33 UTC (Thu)
by knewt (subscriber, #32124)
[Link]
Although I haven't tried it at this point (nor docker-mail server), there is also mailcow, which in complete coincidence I started investigating recently!
Posted Nov 21, 2024 12:18 UTC (Thu)
by intelfx (subscriber, #130118)
[Link]
Mail-in-a-box, for instance. Although it's deliberately designed to be non-customizable (and takes an entire host for itself, also by deliberate design).
Posted Nov 21, 2024 12:34 UTC (Thu)
by smoogen (subscriber, #97)
[Link] (3 responses)
At the moment I can only see email being done in the following forms:
The problem tends to be that a lot of companies were doing 1 but not staffing appropriately.. so they went to 2. Various companies which did email as a service also didn't staff appropriately and there was 'market' consolidation where 60-90% of the market ends up with 3 vendors.
that leaves people like me who were trying 3 for years but finally ended up being in 4.
Posted Nov 21, 2024 13:38 UTC (Thu)
by dskoll (subscriber, #1630)
[Link] (2 responses)
I think you overstate the difficulty. I ran self-hosted mail for my small company of 12 people for about 18 years, and it really wasn't much work once it was set up. I've been running my own personal email for about 8 years, and again... once the initial setup was done, tending it has been almost no work.
DKIM, DMARC, and SPF haven't changed in years, and neither has SMTP.
Posted Nov 21, 2024 14:07 UTC (Thu)
by smoogen (subscriber, #97)
[Link]
In the end, I expect that for people who have worked in email for a while, there is a lot of hidden knowledge which allows you to miss a lot of minefields other people seem to walk into over and over again.
Posted Nov 21, 2024 15:22 UTC (Thu)
by mbunkus (subscriber, #87248)
[Link]
The huge timesink is, unfortunately, the very start where you have to learn about everything & get everything right, otherwise you'll fight issues such as non-delivery, landing on anti-spam blocklists etc. for a long time. The initial barrier is huge, the ongoing maintenance very low.
Posted Nov 29, 2024 17:46 UTC (Fri)
by Curan (subscriber, #66186)
[Link] (19 responses)
We should rather point out bullshit and snakeoil requirements in the existing system. Has anybody ever received less spam after adding SPF and/or DKIM? I don't think so. It is just "bullshit compliance" required by big e-mail service providers to keep individuals out of the game (which rarely are the real issue). My personal best option to prevent spam, was blacklisting certain IP ranges. Ever since I did that, my spam level dropped to basically "no longer measurable". But that is, I have to admit, not a politically convenient option. And probably not something you can do like this all around the world, probably not even in most cases.
I can only recommend that everybody runs their own MTA (and makes sure it is secured, of course, but that was an issue from the beginning anyway).
Posted Nov 29, 2024 17:48 UTC (Fri)
by mpr22 (subscriber, #60784)
[Link] (6 responses)
Posted Nov 29, 2024 18:00 UTC (Fri)
by Curan (subscriber, #66186)
[Link] (5 responses)
But you are correct in the basic statement. Directly only my outbound e-mails would be affected. And to be honest: I never saw the reason why a MTA could accept my e-mails today, but not the next day, because I didn't have DKIM/SPF. It is just bullshit compliance theatre, imposed by organisations like CISA/BSI/ENISA/… (in my humble opinion).
Posted Nov 29, 2024 18:28 UTC (Fri)
by pizza (subscriber, #46)
[Link] (4 responses)
SPF/DKIM only attests that the sender is allowed to send on behalf of their domain. That by itself has *significantly* cut down on the amount of outright fradulent or malicious stuff landing in folks' inboxes -- think phising or worse, where the sender is actively trying to hide the origin of their messages.
The latter used to _heavily_ rely on spoofing legitimate domains via open relays or compromised systems; now those folks have to rely on stolen credentials, with a narrow window before the provider shuts it down.
Of course DKIM/SFP does nothing for "legitimate" [1] UCE, but then it's not supposed to.
[1] "unsolicited commercial email" where the sender is who they claim they are, aka what we traditionally referred to as "spam"
Posted Nov 29, 2024 18:45 UTC (Fri)
by Curan (subscriber, #66186)
[Link] (3 responses)
There is a (comparatively) small amount of e-mails that would be caught by SPF and/or DKIM.
Posted Nov 30, 2024 13:24 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
Might this, perhaps, be *because* they are enforced? That is, they've been required long enough that what they prevent has indeed been extinguished, but it is still cheap enough even with these being required to churn out junk email that it *appears* nothing has changed? Short of charging per email exchanged or much higher registration fees…what is actually going to increase costs for these operators?
Posted Nov 30, 2024 13:48 UTC (Sat)
by pizza (subscriber, #46)
[Link]
Yes, exactly this.
This discussion reminds me of Y2K, afterwards, laypeople (and many that should know better!) were going "what was the big deal, the world didn't end, we're not going to believe the next so-called panic" completely missing the fact that it was a non-event only because obscene amounts of effort went into fixing everything up (barely) in advance.
Posted Nov 30, 2024 16:53 UTC (Sat)
by Curan (subscriber, #66186)
[Link]
Posted Nov 29, 2024 18:16 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (6 responses)
My incoming mail volumes reduced by about 5% when I implemented both DKIM and SPF; I was facing a significant amount of backscatter (due to other people's badly configured MTAs accepting spam with my address as the "From", then sending a delivery report my way when they couldn't deliver it). On some days, backscatter was more than 50% of my total incoming mail.
Implementing DKIM and SPF removed virtually all of that; I can see from DMARC reports that people are still trying to forge my e-mail address as the reply target, but their mail is no longer being accepted. And the rate of forgery has nosedived; when I first implemented it, DMARC reports from the big players (Google, Outlook.com, mail.ru) showed that they were rejecting around 99% of mail that claimed to be from me. Now, they're not reporting any.
Posted Nov 29, 2024 18:41 UTC (Fri)
by Curan (subscriber, #66186)
[Link] (5 responses)
Posted Nov 29, 2024 19:05 UTC (Fri)
by farnz (subscriber, #17727)
[Link] (3 responses)
It wasn't my experience, until spammers started forging my address as the "From" address of their spam. At that point, backscatter went from "something I see once or twice a month" to 5% of my incoming mail. I implemented the entire set of DMARC measures more or less out of desperation, and it basically reduced backscatter to nil. It also, as far as I can tell from the DMARC reports, had a significant impact on the amount of spam that other people got with my address forged onto it.
Posted Nov 30, 2024 6:01 UTC (Sat)
by Curan (subscriber, #66186)
[Link] (2 responses)
I am really sorry, you saw an increase in spam, but I will contest, that SPF/DKIM actually helped you. I will agree, that those technologies will have helped you to paper over existing issues, but in the end it won't matter. Today my most common spam sending e-mail domains are `gmail.com` and `hotmail.com` (ie.: Google & Microsoft, who have been pushing the hardest for SPF/DKIM).
And, TBH, it is in their interest to have these accounts. It just is not in mine and there is no automated way to prevent that e-mail from being processed by my local filters. Those catch the spam and sort it accordingly, but that also means I have to expend far more resources than I want to.
Posted Nov 30, 2024 13:03 UTC (Sat)
by pizza (subscriber, #46)
[Link]
Contest it all you want, but you're going to need a more reasoned argument than "I don't believe you" and quoting wikipedia definitions.
(BTW: I've been self-hosting my own email for over 25 years. The combination of SPF, DKIM, and DMARC made a *huge* difference on the domains I control)
> Today my most common spam sending e-mail domains are `gmail.com` and `hotmail.com` (ie.: Google & Microsoft, who have been pushing the hardest for SPF/DKIM).
You sound like the person who points at snow on the ground claiming that as proof that global warming isn't real.
> It just is not in mine and there is no automated way to prevent that e-mail from being processed by my local filters
Uh, nobody has ever prevented you from running your own client-side filters. (Except perhaps your employer, but that's their problem, not yours)
Posted Nov 30, 2024 15:31 UTC (Sat)
by farnz (subscriber, #17727)
[Link]
OK then; if SPF/DKIM didn't help me, what exactly did reduce the load? I went from approximately 2 backscatter mails caused by spam with my address forged on it in a month, to typically 10,000 in a day, and the thing that caused it to go back down to approximately 2 in a month was implementing DKIM, SPF and a DMARC policy.
Further, having done this, I could see from the DMARC reports that for about 3 months after I set this up, big hosts like mail.ru, fastmail.com, Outlook.com, and Google were seeing typically 20,000 hosts not compliant with my DMARC policy in a day. That has now dropped to zero.
If it wasn't DKIM, SPF and a DMARC policy that helped, what was it?
Posted Nov 29, 2024 21:15 UTC (Fri)
by dskoll (subscriber, #1630)
[Link]
I expect that it's no longer the case [that SPF "pass" is a mild spam indicator] because the standard is old and most people have caught up.
Posted Nov 29, 2024 18:38 UTC (Fri)
by dskoll (subscriber, #1630)
[Link] (4 responses)
Neither SPF nor DKIM are anti-spam measures. They are designed to make it harder for someone to fake mail from your domain. Back when I ran an email security company (so 2000-2018) we found that an SPF "pass" was a slight spam indicator, because spammers were more diligent about maintaining correct SPF records than non-spammers.
Google et. al. demand SPF/DKIM/DMARC not to reduce spam, but to be able to hold senders accountable... if a piece of spam passes DKIM and SPF, we can generally know who (as in which domain or which MTA) was responsible for letting it out onto the Internet.
I agree that telling people not to run their own MTA is very harmful and is damaging to the integrity of the Internet. We cannot let an open and useful protocol slip out of the hands of the community and into the hands of a few powerful multinationals.
Posted Nov 29, 2024 18:56 UTC (Fri)
by Curan (subscriber, #66186)
[Link] (3 responses)
You should really tell this to various government bodies. Because they list this as requirements to combat spam...
Bud I do understand your position, it is just not, what is implemented in the world by big providers and required by government bodies.
> Back when I ran an email security company (so 2000-2018) we found that an SPF "pass" was a slight spam indicator, because spammers were more diligent about maintaining correct SPF records than non-spammers.
And just like that you told me, that SPF is pointless.
> Google et. al. demand SPF/DKIM/DMARC not to reduce spam, but to be able to hold senders accountable... if a piece of spam passes DKIM and SPF, we can generally know who (as in which domain or which MTA) was responsible for letting it out onto the Internet.
Oh come on, that is not helping anybody. Unless you try to tell me it is hard to get a domain. Which it is not, I must say. And if we assume getting a domain is easy: what does anybody gain by knowing "who" it was? The spammers will just change the domain.
Anyway, in my experience most spam comes from „legitimate“ domains like `gmail.com` and such. Making the whole endeavour circular at least.
Posted Nov 29, 2024 19:47 UTC (Fri)
by mbunkus (subscriber, #87248)
[Link] (2 responses)
You're wrong insofar as you're only thinking about fighting spam. But as Diane said, SPF & DKIM are supposed to establish authenticity. When I receive email from my customers (for whom my company has set up SPF, DKIM, DMARC in the past) I can rely on two things:
1. The email does indeed come from my customer & not from a malicious third party.
These particular technologies are not about fighting spam. They addressed different shortcomings of the SMTP protocol itself:
1. No way to verify that a sender (both in the sense of the program creating the mail & the server relaying the mail) is allowed to send mail for a certain domain. This is what SPF addresses.
Again, this is _not about spam_.
[1] Before DKIM there had only been a way to protect email bodies from malicious modifications by sending cryptographically signed emails (GPG, S/MIME). Those didn't do anything for the headers, though.
Posted Nov 30, 2024 5:33 UTC (Sat)
by Curan (subscriber, #66186)
[Link] (1 responses)
No you can't, because this still assumes honest players. Really, I am running MTAs for very large organisations and my experience is: a lot of legitimate e-mails would be filtered if I enforce SPF/ DKIM. On the other hand I have actual and obvious spam in troves, that meets SPF/DKIM requirements.
In my personal experience: the best way to verify a sender is OpenPGP. That being said: that is a negligible amount of traffic here.
But since you insist on the "not about spam" line: that is not how it is sold. And not how it is promoted. See eg. https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail where it says in the first line
> DomainKeys Identified Mail (DKIM) is an email authentication method designed to detect forged sender addresses in email (email spoofing), a technique often used in phishing and email spam.
Or in https://en.wikipedia.org/wiki/Sender_Policy_Framework
> Sender Policy Framework (SPF) is an email authentication method which ensures the sending mail server is authorized to originate mail from the email sender's domain.[1][2] [...] Forgery of this address is known as email spoofing,[3] and is often used in phishing and email spam.
And I know Wikipedia is not the best source, but if you are honest you will find, the RFCs and other industry sources agree.
---
That being said: I do not think us discussing this issue over text in a comment section is going to help us, because this is a very poor communication channel. If you are at FOSDEM, let me know and we can meet up in person and discuss this better. Probably over a beer or two. ;-)
Posted Nov 30, 2024 7:20 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link]
FWIW, spammers don't even bother with forging the 'from' field anymore because they assume that SPF/DKIM is everywhere.
Posted Aug 3, 2025 14:30 UTC (Sun)
by pschneider1968 (guest, #178654)
[Link]
I do some hosting of web and mail domains for myself, for a couple of friends, and for my local chess club. I have the domains with IONOS, and the Debian server VMs with Strato for some 6-11€/month. I do use Postfix, Dovecot and Roundcube on the servers.
To configure all this on a bare Debian installation is way too much work, and error-prone, too. You need some good hosting software to make this easier and manageable. From my point of view, there are basically two viable software packages ("Admin Panels") that do this: one is Plesk, which is commercial, very good, but not cheap if it's not included with your hosting plan. The other one, and this is the one I'd really reommend, is Virtualmin. It's even better than Plesk (I think) and it's FOSS and free (as in "free beer") to use, with a subscription-based paid variant with some additional but non-essential features available, too.
You can install it with either Apache or with Nginx as the webserver. I prefer Nginx, because it makes redirection to Docker containers really simple. With Virtualmin as the Admin panel, virtual domain hosting is really made easy. It does all the hard and complicated configuration work for you, but in a very transparent way. You can always override any setting manually by editing the individual components config file directly, but through the Web GUI.
The only things regarding the mail server I had to do fully manually was incorporating the free Spamhaus.org data query service (i.e. their free block list subscription) into the Spamassassin config, and the addition of OpenSRS (implementation of sender rewriting) to Postfix, because I do a lot of mail forwarding, as not all of my users want a mailbox; some prefer having their domain's mails forwarded to their private mail account.
Rspamd I looked into, but figured that at that point in time, it was serious overkill for what I needed. My setup was been working fine with Spamassassin for some years now. Virtualmin also allows for easy integration of mailbox virus scanning with ClamAV. And you can, albeit a lot more complicated than the basic setup, use LDAP for sender and mailbox accounts.
For securing your mail setup, the Email Hardening Guide is a very good read (short of reading THIS book) which can be found here:
https://www.mailhardener.com/kb/email-hardening-guide
It explains SPF, DKIM, DMarc and MTA-STS for mere mortals, and is helpful if you have some basic understanding of how DNS and SMTP work and relate to each other. You have to set this up correctly to get mails delivered to most of the big mail providers. Additionally, for some mail providers you have to go through some additional hops... For Google/Gmail, you have to have a Google account and register your mail server in the Google Postmaster tools, and then add a DNS TXT record to your domain with the Google verification code. In Germany, in order to be able to deliver to T-Online mail accounts, you have to contact the T-Online Mail admin team and ask that they add your mail server IP to their whitelist, which requires that you have to setup a web page for the mail domain with a legally required Imprint, satisfying German and EU law Imprint requirements.
With all setup correctly, I never had a problem delivering mails to Gmail, T-Online, web.de, gmx.de/org/net, yahoo, outlook.com/de etc.
And with the help of this great book, you can even gain some more deep understanding of these setup and security topics. But honestly, with a basic understanding AND the ability to read the Virtualmin documentation AND the Email Hardening Guide above, you are good to go!
Sounds familiar
Sounds familiar
Sounds familiar
Sounds familiar
Sounds familiar
Sounds familiar
JMAP is a competing email protocol; you can never have enough of those, after all.
JMAP
Sounds familiar
JMAP is good, IMHO
the aerc client (featured on LWN recently) supports JMAP, although I've not used it.
Sounds familiar
Sounds familiar
Sounds familiar
Sounds familiar
Sounds familiar
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
> If you want to ingest the whole firehose, then it's a different story.
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Who is this book for?
Cheers and (un)expectations
Cheers and (un)expectations
Occasional book reviews are a longstanding tradition here, though, going back to 1999.
Book reviews
Sounds useful
Sounds useful
Sounds useful
Sounds useful
Sounds useful
1. Full time job of 1 admin for every 50->100 mailboxes with lots of ways for the service to go down daily still.
2. Outsourced job to some company which deals with millions of mailboxes
3. A local gardening job to let someone who likes to dabble in things learn how the internet changes regularly.
4. One of the above but poorly resourced and over burdened.
Sounds useful
Sounds useful
Sounds useful
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
- barely secured domains (the spammers got a "legitimate" account, even though the operator would say "not an actual user")
- domains created for spamming
Invalid recommendations
Invalid recommendations
> Might this, perhaps, be *because* they are enforced?
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
Invalid recommendations
2. No malicious third party has modified the most important headers while in transit.
2. No way to validate that email headers haven't been modified after having been sent by the sender. This is what DKIM addresses.[1]
3. DMARC as the third part finally tells the recipient that both SPF & DKIM are actually supposed to be used by all mail originating by a domain, and what the domain's owner wants the recipient to do if one of both or them are invalid or missing.
Invalid recommendations
Invalid recommendations
Virtualmin