|
|
Subscribe / Log in / New account

Book review: Run Your Own Mail Server

By Joe Brockmeier
November 19, 2024

The most common piece of advice given to users who ask about running their own mail server is don't. Setting up and securing a mail server in 2024 is not for the faint of heart, nor for anyone without copious spare time. Spammers want to flood inboxes with ads for questionable supplements, attackers want to abuse servers to send spam (or worse), and getting the big providers to accept mail from small servers is a constant uphill battle. Michael W. Lucas, however, encourages users to thumb their nose at the "Email Empire", and declare email independence. His self-published book, Run Your Own Mail Server, provides a manual (and manifesto) for users who are interested in the challenge.

[Cover: Run Your Own Mail Server]

Run Your Own Mail Server came to my attention toward the end of its successful Kickstarter campaign. About 20 years ago I decided that hosting my own email was a hobby I no longer wished to pursue. I changed my MX records to an external provider and powered off the long-suffering Pentium Pro server for the last time. Regrets were few.

However, the general message of the campaign ("By running your own email, you seize control of your communications.") appealed to me. Paying for email hosting has freed up some time, and it's certainly cheaper for one or two users if one believes that their time is worth money, but at the cost of giving up control. I was interested in dipping my toes back in the email waters to see if it was time to resume managing my own mail.

Lucas provided PDF and EPUB versions of the book for this review, and I worked from the PDF. The book is 350 pages and is currently available for about $15 for the digital editions, and about $35 for a paperback edition. Electronic editions are the cheaper and faster way to get one's hands on the title, but I would recommend picking up a print version for anyone serious about running their own mail server. This is the kind of tech book that one would like to annotate, plaster a few sticky notes and page markers in, and so forth. An early draft of the introduction is available on his site.

Who, why, and what

The introductory chapter establishes who should read the book, why they should be interested in running a mail server, and what platforms and tools Lucas will cover. The book is aimed solidly at system administrators, or at least those with substantial administration skills.

Lucas makes clear from the outset that readers are expected to be able to manage their own operating system, and extrapolate from examples rather than follow them blindly. For instance, he notes that he takes his examples from FreeBSD or Debian Linux—so the path to configuration files may differ for readers who are setting up these services on other systems.

Readers are expected to already have an understanding of things like DNS, networking, managing certificates with Let's Encrypt, and more. Or, at least, readers must be willing to become familiar—Lucas provides some recommendations for his own books on system administration as well as Cricket Liu and Paul Albitz's DNS and Bind. If dabbling with DNS records, logging, TLS certificates, databases (yes, plural), user management, and more sounds fun, then this is exactly the book to pick up.

The introduction provides a brief look at the history of email, covers things like Simple Mail Transport Protocol (SMTP), MX records, Sender Policy Framework (SPF), and his choices of software. The book covers using Postfix as the mail transfer agent (MTA), Dovecot as the Internet Message Access Protocol (IMAP) server, Roundcube for webmail, and Mutt as the mail user agent (MUA), or mail client. Often a book's introduction can be safely skipped over, but it's not recommended here, as it contains information and context that will be needed later.

Digging in

The next two chapters look at SMTP, Postfix, and Dovecot in more detail and start the reader on the journey to actually setting up a mail server to send and receive email. Two servers, actually: readers need to have two hosts connected to the internet to properly follow along with the book. This enables readers to test sending and receiving mail before trying to send mail to the world at large (and risk having a misconfigured server wrongly identified as a spam host).

It's easy to find guides online to configure Postfix or Dovecot with a set of explicit instructions one can buzz through in a few minutes. That is not the approach that Lucas takes. Instead, he walks through setting up Postfix and Dovecot while taking the time to explain the various configuration options and commands in a sort of lecture format. This method is generally enjoyable if one wants to know why, as well as how, to do things, but it's not always the most efficient way to convey setup instructions. Then again, if one is seeking efficiency, the most expedient thing to do is pay a provider to manage email and let it be somebody else's problem.

Scaling up

By the end of chapter 3, readers should have a pair of mail servers that can send, receive, and deliver messages to local users in the Maildir format. That is a useful foundation, but hardly what most would consider a functional mail setup—the mail servers are not ready, for example, to work with external clients—so users can only send mail while logged into the servers. Next Lucas turns to addressing virtual domains, IMAP, setting up authentication for users to send mail, and installing the web-based PostfixAdmin tool.

Chapter 3, "Postfix and Dovecot Setup", might have worked better as two chapters that focused on the respective programs. The coverage of Postfix seemed more thorough and organized than Dovecot's. For example, Lucas starts describing Dovecot's local delivery agent, (dovecot-lda) setup in Chapter 3 and then picks it up again in Chapter 4. The instructions, at least on my Debian 12 system, are somewhat incomplete and required further troubleshooting because the service did not have permissions to open the appropriate log files.

No book on running mail servers would be complete without addressing the topic of spam, and much of Run Your Own Mail Server is devoted to the topic in one way or another. Lucas devotes two chapters to his choice of spam filtering tool, Rspamd, which LWN covered in 2017. Given the complexity of Rspamd, compared to setting up Apache SpamAssassin, this seemed like a bit of overkill at first—but it seems to be an integral part of the way that Lucas likes to manage mail.

Chapter 7, "Rspamd Essentials", starts down the path of setting up Rspamd and requisite components. It spends a fair amount of time on the Universal Configuration Language (UCL) used by Rspamd as well as setting up separate Redis instances for Rspamd's configuration and Bayesian data, respectively. (He does note the Redis license change and acknowledges that users may need to migrate to another database down the road.)

The book picks up again with Rspamd in Chapter 14, "Managing Rspamd", after covering SPF records, DomainKeys Identified Mail (DKIM), DMARC, block lists, setting up web mail with Roundcube, and filtering mail using Sieve. The last time I managed my own mail, SPF was in its infancy and things like DMARC didn't exist yet. I found these chapters to be a helpful overview of those topics as well as useful in setting up SPF and DMARC records. The reason Rspamd setup is covered early on, and then set aside for several chapters, is that Lucas also recommends it for use in DKIM signing and verification rather than OpenDKIM. He does double back and cover OpenDKIM setup late in the book for instances where readers might want to set up mail servers to send, but not receive mail, saying that Rspamd is overkill for hosts that won't receive mail.

Chapter 9 covers protocol checks and block lists to reduce spam that makes it to users' inboxes. It briefly touches on the robustness principle, also known as Postel's Law after Jon Postel. Lucas suggests that the principle, "be conservative in what you do, be liberal in what you accept from others", favors spammers. He explains setting up postscreen to look for telltale signs that incoming connections are from spambots. This includes comparing against against Domain Name System blocklists (DNSBLs) as well as DNSWL to identify trustworthy mail servers. The chapter also discusses more "intrusive" checks such as looking for non-SMTP commands or improper line feeds. The SMTP protocol specifies carriage-return with line feed (CRLF), so if a client sends bare line feeds (LFs) instead, there's a good chance it's a poorly programmed spambot. Of course, not everything that's outdated or poorly programmed is a spambot—so Lucas also discusses how to wave through connections from legitimate organizations that have badly behaved network devices or ancient mail servers.

Overall, I enjoyed and would recommend Run Your Own Mail Server for those comfortable with managing their own services, and who are willing to put in the work. It should be clear after reading the title that running a mail server is a hobby much like gardening: the work is never "done", it requires constant tending and eternal vigilance against pests. Currently, I'm on the fence about doing the long-term work to migrate back to self-hosting—but much more apt to do so after reading the book.

Readers who are looking for a book that provides a step-by-step set of instructions to be typed out will probably not be satisfied. The book is not well-suited for skimming over in a rush to set up a mail server. Lucas actually seeks to help readers understand what they are doing rather than running through a few steps by rote and hoping commands, syntax, and protocols never change.

The book is also meant to empower readers to be good communal email citizens and run an email server for a small organization, group, or just themselves. It does not cover every topic, or ready a reader to set up email for thousands of users. He touches on a few issues of scale, such as how to send email to many recipients at once (like a newsletter) while avoiding being consigned to spam purgatory by providers like Google, Microsoft, and Yahoo. But routinely handing massive email traffic, splitting out services among multiple servers, or dealing with hundreds or thousands of users is beyond the scope of the book.

Some books are like GPS navigation: set a destination and get what the author believes to be the shortest and fastest route with turn-by-turn instructions. Lucas occasionally takes the reader on the scenic route, sometimes navigating by landmarks instead of highway signs, and occasionally stopping at tourist attractions. One way gets there fast, but the other way builds skill and confidence in navigating solo.



to post comments

Sounds familiar

Posted Nov 19, 2024 20:40 UTC (Tue) by jkingweb (subscriber, #113039) [Link] (13 responses)

My own setup seems to be very close to Mr. Lucas' own—which perhaps isn't surprising, but at least it means I'm probably not crazy in my choices.

Rspamd tried its best to convince me I'm insane, though. While I adore the results, configuring it was extremely painful, in no small part because the configuration format is never quite consistent with itself. For some things I had to resort to trial and error, which was a bit harrowing. Thus it sounds like it would be interesting to hear what the author has to say on the subject.

I wouldn't criticize people for saying one should leave mail to experts (it _is_ a lot of work to set up properly, and some work to maintain), but I'm glad there are those who go against the grain, and even try to empower others. I certainly would have appreciated something like this when I started out a decade ago.

Sounds familiar

Posted Nov 19, 2024 21:51 UTC (Tue) by jasonjgw (subscriber, #52080) [Link] (10 responses)

Mine also comprises Postfix, Dovecot and Rspamd. I found the online test at internet.nl to be valuable in checking my SPF, DKIM, DMARC and TLSA records. Stalwart Mail Server seems interesting (and perhaps too new to be covered in the book), and I haven't examined it in enough detail to decide whether switching to it would be advantageous.

Sounds familiar

Posted Nov 19, 2024 21:58 UTC (Tue) by dskoll (subscriber, #1630) [Link] (1 responses)

I run (ugh) Sendmail on my MX host, but that's only because the commercial anti-spam software I spent 18 years of my life working on integrates with Sendmail.

It relays to my IMAP server, which is Postfix/Dovecot. If I were starting out again, I never would have picked Sendmail.

Sounds familiar

Posted Nov 20, 2024 15:45 UTC (Wed) by smoogen (subscriber, #97) [Link]

And thank you for that software.. I ran sendmail way too many years because it made it usable on the Internet in the 00's and early 10's.

Sounds familiar

Posted Nov 19, 2024 22:47 UTC (Tue) by jkingweb (subscriber, #113039) [Link] (7 responses)

A friend of mine recently switched from the same set-up as mine to Stalwart. I didn't hear too much gnashing of teeth from him, though he did have a bit of difficulty setting up DKIM signing, and he couldn't figure out JMAP (which is the only reason I'd be tempted to switch).

What I have satisfies me, though. It may be Frankenstein's monster, but it's already assembled and alive, which is the best kind of monster.

Sounds familiar

Posted Nov 20, 2024 0:05 UTC (Wed) by willy (subscriber, #9762) [Link] (6 responses)

Is JMAP a typo for IMAP or is this some protocol I'm not familiar with?

JMAP

Posted Nov 20, 2024 0:08 UTC (Wed) by corbet (editor, #1) [Link]

JMAP is a competing email protocol; you can never have enough of those, after all.

Sounds familiar

Posted Nov 20, 2024 0:33 UTC (Wed) by jkingweb (subscriber, #113039) [Link] (3 responses)

JMAP is a replacement for IMAP (and SMTP submission, and CalDAV/CardDAV, I think) which uses HTTP for transport and JSON for payload formatting. I haven't looked at it in too much detail, but it did go through the IETF standard track process, and was championed by FastMail. There are few servers and even fewer clients, though, so it remains a curiosity more than anything.

I think the selling point is that JMAP is easier for clients to implement than IMAP, but till such time as servers are common that's a rather illusory benefit.

JMAP is good, IMHO

Posted Nov 20, 2024 2:10 UTC (Wed) by CChittleborough (subscriber, #60775) [Link]

JMAP is not just championed by FastMail: they created it. Their webmail service uses it. (They also use and contribute to Cyrus; in a closely-related development, Cyrus supports JMAP.)

I read the RFCs a few years back. It seems to me to be a clever way to synchronize lists of lists between client and server over HTTP using JSON ... but I'm not an expert on this topic.

(Disclosure: I am a very satisfied customer of FastMail, and heartily recommend them to anyone who, like me, wants to never ever have to set up spam filtering again.)

Sounds familiar

Posted Nov 21, 2024 11:16 UTC (Thu) by jond (subscriber, #37669) [Link]

the aerc client (featured on LWN recently) supports JMAP, although I've not used it.

Sounds familiar

Posted Nov 22, 2024 1:38 UTC (Fri) by dilinger (subscriber, #2867) [Link]

A selling point of JMAP for me is the "over HTTP" bit. It's frustrating when you're at a coffee shop or a doctor's waiting room and their wifi is filtering various SMTP-related ports, so you can't actually send the email you just wrote. Same reason I want XMPP over HTTP..

Sounds familiar

Posted Nov 20, 2024 6:25 UTC (Wed) by zdzichu (subscriber, #17118) [Link]

It's a protocol we discussed here already: https://lwn.net/Articles/680057/

Sounds familiar

Posted Nov 19, 2024 23:05 UTC (Tue) by gerdesj (subscriber, #5446) [Link]

"Rspamd tried its best to convince me I'm insane"

The three levels of config looks a bit mad at first but it isn't. Get it right and upgrades are painless. There's the defaults, spelt out in config files that the packages manage. Then there is a tree of configs which will replace an entire component's config and finally another set of configs which only change those settings that are mentioned for a component.

Its really flexible and easy to work with and as I mentioned, makes upgrades a doddle.

Sounds familiar

Posted Nov 20, 2024 12:48 UTC (Wed) by leromarinvit (subscriber, #56850) [Link]

I also run a similar setup - Postfix and Dovecot mainly. I didn't find it all that difficult to set up for my single-user config (granted, it's of course more work than signing up for an email account somewhere).

A long time ago, I used to be a happy user of dspam - but that was abandoned at some point IIRC. Looking for a replacement when the time came to move to a newer distro release, I installed Rspamd - but never took the time to understand how to configure it. So for the last 10 or so years, I've been deleting 1-10 spam emails a day. Not too terrible IMHO, some are even somewhat comically funny. That's a very small minority though - most are just the same old boring scams and gone as fast as I can hit delete.

What stops most spam though seems to be the very basic DNS requirements on sending hosts - the HELO name needs to resolve to the IP it's connecting from, and that IP must have a PTR record. Most zombied spam senders don't meet these requirements, while the vast majority of legitimate email servers does. (I remember having to manually allow exactly one host to be able to receive emails from a large, well-known company that shall remain unnamed.)

Who is this book for?

Posted Nov 19, 2024 22:13 UTC (Tue) by dowdle (subscriber, #659) [Link] (13 responses)

From the description, this book only has a smallish potential audience... someone who wants to set up an email server... for mostly personal use... with a relatively small number of users. While I would personally likely find that interesting, the world needs another book that is aimed at a wider audience... of those who want to setup a larger system with more of the services included that most contemporary users expect in a "communications" system. Modern needs include a global address book, calendaring, advanced searches, document sharing, etc. While a lot of the mentioned functionality might be pawned off on the email client side, we are really lacking in feature-full email clients these days and so many users have just grown accustomed to web-based email systems. Think Zimbra or Microsoft's Office 365 OWA and not Roundcube. Maybe this book could have a big-brother book someday?!?

Who is this book for?

Posted Nov 19, 2024 22:25 UTC (Tue) by jzb (editor, #7867) [Link] (2 responses)

From the description, this book only has a smallish potential audience

Oh, I dunno - I'd say that the potential audience is pretty large. In the tens of thousands, anyway.

I think that NextCloud might include most, if not all, of the functionality you're looking for out of a communications system—though I haven't touched it in ages and ages. Not sure if there's a good book on NextCloud setup or not.

Who is this book for?

Posted Nov 19, 2024 23:28 UTC (Tue) by Heretic_Blacksheep (guest, #169992) [Link] (1 responses)

"Tens of thousands" is not a large market for any book, much less so when that's only the potential reach. Hence self publishing as no traditional publisher would get out of bed for that.

Who is this book for?

Posted Nov 20, 2024 0:07 UTC (Wed) by raven667 (subscriber, #5198) [Link]

The author seems to make a living writing and self-publishing technical books/documentation for small audiences, and that's fine, not everything needs to be targeted for hyper growth 10x mega returns to be successful, which is unfortunately a default criteria for too many people/projects that leads to unhappiness when it doesn't happen.

Who is this book for?

Posted Nov 20, 2024 0:21 UTC (Wed) by raven667 (subscriber, #5198) [Link] (5 responses)

I think this is a good book to help create a pipeline of competent admins outside of a few large corporations and service providers. Who is supposed to be hired as the next generation of admins for those corporations and service providers if very few people ever learn or have experience with email on a small scale, people whose only experience is the internal training at that organization, or similar peer institutions, which creates blind spots that become outside of the possible solution space as the system needs to change over time. I think there is a large cohort of technical experts who started by running small scale personal servers and desktops using Linux/Unix that gave them a breadth of experience that isn't had by people who only know Python and Kubernetes. Keeping a community of postmasters, hostmasters and webmasters alive, by showing technically minded people that they _can_ do this stuff themselves if they have the time and interest, its not something _only_ for Gmail/O365, is a good thing IMHO, not just because it might lead to cushy jobs but because users keep the software and standards alive too.

Who is this book for?

Posted Nov 20, 2024 4:29 UTC (Wed) by buck (subscriber, #55985) [Link] (4 responses)

Touché

Without getting too conspiracy-minded, if nobody is maintaining SMTP services as a least common denominator for password recovery, communications from financial services companies, pizza order delivery receipt sending, etc., aside from the big providers who make their profits elsewhere, then we are not far from reverting to the world of Prodigy, CompuServe, and AOL, with no links between our messaging providers and their online communities except what they maybe deign to allow us, perhaps for an interchange fee, and only as long as they can't more directly monetize our online time by keeping our packets in their networks.

That said, I myself am going to have to depend on other people keeping the vision alive, because I am not up to the challenge of not only getting a mail domain on-line, SPF and DKIM (or is it DMAC?) correctly configured, kept off IP block lists, etc., but then having to keep up with the continuing evolution of the tradecraft required to keep mail flowing with as much of the junk as possible being winnowed.

Though, for all the success that free SMTP exchange may have staying around and being vital longer term, i am not ultimately that hopeful. Not unless the administrators who continue in the trade band together like the network operators do and figure out how to keep settlement-free peering alive just because they feel it is the right thing to do. But I just don't see that being sustainable in this case and wonder what the cost of contacting each other that we currently take for granted as being marginally 0 is going to be once the email baseline is no longer there to undercut any provider's designs on charging us to communicate. Brings back memories of toll calling.

OK, maybe this is a wee bit conspiracy-minded.

Who is this book for?

Posted Nov 20, 2024 14:21 UTC (Wed) by raven667 (subscriber, #5198) [Link] (3 responses)

I had a half-formed thought before that keeping organizations invested in SMTP allows small independent services to exist, like a personal domain, but if MS and Google agreed and coordinated with the top 100 (maybe even top 20) email services they could define a new private protocol between themselves, that might have great technical and security benefits, which would be too difficult for small personal servers to effectively use, even if it is officially "open". In some ways that reminds me of Mastodon/ActivityPub vs Bluesky/AT where you can host your own personal Mastodon instance just like email (but then have to deal with substantial administrivia and performance issues such that paid hosting services exist to help automate server management) or Bluesky where you can in theory host your own data but the indexing/search/feed/app services have a huge footprint that can really only be provided by a big tech company, it's infeasible/impossible to host the app on your own. I could see email moving to an AT-like situation.

Who is this book for?

Posted Nov 20, 2024 19:30 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> I had a half-formed thought before that keeping organizations invested in SMTP allows small independent services to exist, like a personal domain, but if MS and Google agreed and coordinated with the top 100 (maybe even top 20) email services

email is one service that needs to work with all kinds of clients. Corporations that run in-house email servers will be pissed if you stop receiving their emails.

> where you can in theory host your own data but the indexing/search/feed/app services have a huge footprint

No, they're not. I'm self-hosting them on my system. Granted, the setup right now is a pain, but if you just want to follow people and let others follow you, then even an RPi can host it.

If you want to ingest the whole firehose, then it's a different story.

Who is this book for?

Posted Nov 22, 2024 15:08 UTC (Fri) by raven667 (subscriber, #5198) [Link] (1 responses)

>> but if MS and Google agreed and coordinated with the top 100 (maybe even top 20) email services

> email is one service that needs to work with all kinds of clients. Corporations that run in-house email servers will be pissed if you stop receiving their emails.

I don't think a change like that would be instant, but just like the large providers deprecating username/passwords in favor of OAuth or insisting on TLS, if they published a new mail submission and mail transfer protocol the smaller services would update or outsource if they want to keep talking to their customers and suppliers on MS/Google email, and the incentive is for the hosting providers to make outsourcing the easier option, even if that is by making self-hosting more complicated and expensive (costs a larger provider can absorb but a smaller one may not).

>> where you can in theory host your own data [on Bluesky/AT] but the indexing/search/feed/app services have a huge footprint

> No, they're not. I'm self-hosting them on my system. Granted, the setup right now is a pain, but if you just want to follow people and let others follow you, then even an RPi can host it.
> If you want to ingest the whole firehose, then it's a different story.

I think we are in agreement, you can self-host the data (bluesky-pds) on an RPi but you can't easily self-host the feed generator/relay/app/moderation provided by the Bluesky app, even with the code on the bluesky-social github you don't have a full independent service without relying on bsky.app infrastructure, it's designed by people whose relevant experience is managing a firehose of data processing using VC capital, it might not scale down in the way that personal email as described in RYOMS can. At least that's my understanding, I'm not an expert in AT, but I have read a few whitepapers about its design.

Who is this book for?

Posted Nov 22, 2024 22:02 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> I don't think a change like that would be instant, but just like the large providers deprecating username/passwords in favor of OAuth or insisting on TLS, if they published a new mail submission and mail transfer protocol the smaller services would update or outsource if they want to keep talking to their customers and suppliers on MS/Google email

This is not similar. Very few companies use OAuth exclusively, most support logins/passwords.

> I think we are in agreement, you can self-host the data (bluesky-pds) on an RPi but you can't easily self-host the feed generator/relay/app/moderation provided by the Bluesky app

Uhm. You can, it's not even complicated technically. The fundamental issue is that if you want to do moderation, then you actually have to do moderation.

Who is this book for?

Posted Nov 20, 2024 5:09 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

This book is like Linux From Scratch. It's a fun exercise that teaches you how to make a Linux system.

Who is this book for?

Posted Nov 20, 2024 12:47 UTC (Wed) by jzb (editor, #7867) [Link]

Good comparison! Wish I’d thought of that. :)

Who is this book for?

Posted Nov 20, 2024 13:32 UTC (Wed) by rrolls (subscriber, #151126) [Link]

I bought it straight away upon reading this LWN article.

I wish I'd had something like this 15 years ago!

Almost literally, too - that was about when I first would have set up various email-related things on my own servers, but it's always been a horribly complicated mess, mostly just does forwarding to Gmail (that I'd really like to get away from) or ISP mail accounts, and I've lost so many emails from having them miscategorised as spam or otherwise just not processed correctly for whatever reason. Trying to sort out any of this properly has always been such a demotivating nightmare, so hopefully this'll be the missing piece of guidance I needed.

E-mail really is the final boss of digital independence.

Who is this book for?

Posted Nov 21, 2024 22:17 UTC (Thu) by dankamongmen (subscriber, #35141) [Link]

and yet the kickstarter pulled in $76,883, a figure which made my jaw drop. it is not easy to sell books. it is still less easy to presell them, and doing so via kickstarter is hard indeed.

Cheers and (un)expectations

Posted Nov 20, 2024 15:36 UTC (Wed) by Kamiccolo (subscriber, #95159) [Link] (2 responses)

Woa, for me articles like book reviews weren't the thing I was ever expecting on LWN. But, I must admit, I've enjoyed this detailed review and run-through a lot! The consistency and the format in general makes it really pleasant to read and get familiar what to expect from the book (which do seem like just a book to get!). Thanks a lot!

Btw, the cover of the book (created by Eddie Sharam?) made me burst into laugh, I believe in this kind of work, right joke is a virtue :}

Btw, currently Lucas do seem to be running a Kikcstarter for his next book (or rather collection of FreeBSD Journal Letters Column Entries).

Cheers and (un)expectations

Posted Nov 20, 2024 15:40 UTC (Wed) by daroc (editor, #160859) [Link] (1 responses)

We don't do too many book reviews, but we also covered Practical Julia back in April. So it's the kind of article we run sporadically.

Book reviews

Posted Nov 20, 2024 16:10 UTC (Wed) by corbet (editor, #1) [Link]

Occasional book reviews are a longstanding tradition here, though, going back to 1999.

Sounds useful

Posted Nov 21, 2024 4:34 UTC (Thu) by ranger207 (subscriber, #134731) [Link] (7 responses)

This looks like a very useful book. I host my own email with docker-mailserver which is basically a prepackaged collection of Postfix etc, and it's good for being comprehensive enough to be usable immediately while open enough that you can learn more about standard email setups when you need or want to. This book looks like the perfect companion for me in that it's a overall look at why each part is the way it is, which is not readily provided by the RFCs

Sounds useful

Posted Nov 21, 2024 10:22 UTC (Thu) by NRArnot (subscriber, #3033) [Link] (6 responses)

I was going to comment that somebody ought to virtualize this stuff into a "black box" that was safe to use straight away while customizable later if the need arose. Looks as if somebody already has.

Are there any other such projects? Also who is maintaining docker-mailserver (because from bitter experience if there are any exploitable weaknesses in a mail server setup, some slimy spammer will exploit them). Hopefully some relatively large and permanent organisation, rather than "one guy working without reward from his bedroom in Nebraska" (XKCD, maybe mutated through my memory)

Bitter experience drove my employer into the arms of Google many moons ago, but I always wonder what we would do if Google turned evil?

Sounds useful

Posted Nov 21, 2024 11:33 UTC (Thu) by knewt (subscriber, #32124) [Link]

> Are there any other such projects?

Although I haven't tried it at this point (nor docker-mail server), there is also mailcow, which in complete coincidence I started investigating recently!

Sounds useful

Posted Nov 21, 2024 12:18 UTC (Thu) by intelfx (subscriber, #130118) [Link]

> Are there any other such projects?

Mail-in-a-box, for instance. Although it's deliberately designed to be non-customizable (and takes an entire host for itself, also by deliberate design).

Sounds useful

Posted Nov 21, 2024 12:34 UTC (Thu) by smoogen (subscriber, #97) [Link] (3 responses)

The main problem with any 'blackbox' is that it takes a lot of work to keep up. When JZB mentions that this is like a 'garden', it really is. It takes a lot of time and resources to keep up with the 'game' of stopping spam and malware. You think you got all the weeds only to come back tomorrow to find a whole new set come in.. and you have to get to them THEN because they have roots which will kill your crops if they get too big.

At the moment I can only see email being done in the following forms:
1. Full time job of 1 admin for every 50->100 mailboxes with lots of ways for the service to go down daily still.
2. Outsourced job to some company which deals with millions of mailboxes
3. A local gardening job to let someone who likes to dabble in things learn how the internet changes regularly.
4. One of the above but poorly resourced and over burdened.

The problem tends to be that a lot of companies were doing 1 but not staffing appropriately.. so they went to 2. Various companies which did email as a service also didn't staff appropriately and there was 'market' consolidation where 60-90% of the market ends up with 3 vendors.

that leaves people like me who were trying 3 for years but finally ended up being in 4.

Sounds useful

Posted Nov 21, 2024 13:38 UTC (Thu) by dskoll (subscriber, #1630) [Link] (2 responses)

I think you overstate the difficulty. I ran self-hosted mail for my small company of 12 people for about 18 years, and it really wasn't much work once it was set up. I've been running my own personal email for about 8 years, and again... once the initial setup was done, tending it has been almost no work.

DKIM, DMARC, and SPF haven't changed in years, and neither has SMTP.

Sounds useful

Posted Nov 21, 2024 14:07 UTC (Thu) by smoogen (subscriber, #97) [Link]

I was probably too harsh having spent too many times lately trying to help people with 'why isn't my email system getting/sending any email' and finding that it is DKIM, SPF, DMARC, SSL certs or some other item which was slightly broken for them... but wasn't easy to figure out. The ones who are using 'black boxes' find that they are usually not kept up or ghost ware. The usual 'howtos' people are pointing to seem to be written from the early 2000's at the latest when the most you had to deal with was procmail/mimedefang spamassassin type things to deal with garbage.

In the end, I expect that for people who have worked in email for a while, there is a lot of hidden knowledge which allows you to miss a lot of minefields other people seem to walk into over and over again.

Sounds useful

Posted Nov 21, 2024 15:22 UTC (Thu) by mbunkus (subscriber, #87248) [Link]

I second Diane's experience. The thing is that the amount of stuff to learn & get right is a lot, that's true, but once it's been set up it's definitely _not_ a full-time job per 50 mailboxes. I've been running my own mail server both for myself & my company of ~50 people for years now, and the amount of time I have to spend with mail-server related issues averages to single-digit minutes per day, if that.

The huge timesink is, unfortunately, the very start where you have to learn about everything & get everything right, otherwise you'll fight issues such as non-delivery, landing on anti-spam blocklists etc. for a long time. The initial barrier is huge, the ongoing maintenance very low.

Invalid recommendations

Posted Nov 29, 2024 17:46 UTC (Fri) by Curan (subscriber, #66186) [Link] (19 responses)

Having a recommendation of "don't run your own MTA" is so very invalid on so very many levels, that I am wondering, why even LWN is repeating such half-thought-out statements. Giving up our infrastructure is never going to be an option.

We should rather point out bullshit and snakeoil requirements in the existing system. Has anybody ever received less spam after adding SPF and/or DKIM? I don't think so. It is just "bullshit compliance" required by big e-mail service providers to keep individuals out of the game (which rarely are the real issue). My personal best option to prevent spam, was blacklisting certain IP ranges. Ever since I did that, my spam level dropped to basically "no longer measurable". But that is, I have to admit, not a politically convenient option. And probably not something you can do like this all around the world, probably not even in most cases.

I can only recommend that everybody runs their own MTA (and makes sure it is secured, of course, but that was an issue from the beginning anyway).

Invalid recommendations

Posted Nov 29, 2024 17:48 UTC (Fri) by mpr22 (subscriber, #60784) [Link] (6 responses)

I thought the point of SPF and DKIM was to reduce the amount of spam other people receive in their mailboxes whose headers allege that it's from you, not the amount of spam received by you.

Invalid recommendations

Posted Nov 29, 2024 18:00 UTC (Fri) by Curan (subscriber, #66186) [Link] (5 responses)

Well, in the end it is about me, isn't it? If I follow the example of big e-mail providers, I will reject anybody who doesn't have SPF/DKIM. It's just, that in my experience: that doesn't help. Actual spam e-mails have those valid headers most of the time. So my improvement is in the point zero-zero-x range, once I add SPF/DKIM in- and outbound.

But you are correct in the basic statement. Directly only my outbound e-mails would be affected. And to be honest: I never saw the reason why a MTA could accept my e-mails today, but not the next day, because I didn't have DKIM/SPF. It is just bullshit compliance theatre, imposed by organisations like CISA/BSI/ENISA/… (in my humble opinion).

Invalid recommendations

Posted Nov 29, 2024 18:28 UTC (Fri) by pizza (subscriber, #46) [Link] (4 responses)

> It is just bullshit compliance theatre, imposed by organisations like CISA/BSI/ENISA/… (in my humble opinion).

SPF/DKIM only attests that the sender is allowed to send on behalf of their domain. That by itself has *significantly* cut down on the amount of outright fradulent or malicious stuff landing in folks' inboxes -- think phising or worse, where the sender is actively trying to hide the origin of their messages.

The latter used to _heavily_ rely on spoofing legitimate domains via open relays or compromised systems; now those folks have to rely on stolen credentials, with a narrow window before the provider shuts it down.

Of course DKIM/SFP does nothing for "legitimate" [1] UCE, but then it's not supposed to.

[1] "unsolicited commercial email" where the sender is who they claim they are, aka what we traditionally referred to as "spam"

Invalid recommendations

Posted Nov 29, 2024 18:45 UTC (Fri) by Curan (subscriber, #66186) [Link] (3 responses)

Your response relies on the premise, that „modern“ spammers are still rely on faked sending addresses. In my experience spam comes from two sources:
- barely secured domains (the spammers got a "legitimate" account, even though the operator would say "not an actual user")
- domains created for spamming

There is a (comparatively) small amount of e-mails that would be caught by SPF and/or DKIM.

Invalid recommendations

Posted Nov 30, 2024 13:24 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (2 responses)

> There is a (comparatively) small amount of e-mails that would be caught by SPF and/or DKIM.

Might this, perhaps, be *because* they are enforced? That is, they've been required long enough that what they prevent has indeed been extinguished, but it is still cheap enough even with these being required to churn out junk email that it *appears* nothing has changed? Short of charging per email exchanged or much higher registration fees…what is actually going to increase costs for these operators?

Invalid recommendations

Posted Nov 30, 2024 13:48 UTC (Sat) by pizza (subscriber, #46) [Link]

>> There is a (comparatively) small amount of e-mails that would be caught by SPF and/or DKIM.
> Might this, perhaps, be *because* they are enforced?

Yes, exactly this.

This discussion reminds me of Y2K, afterwards, laypeople (and many that should know better!) were going "what was the big deal, the world didn't end, we're not going to believe the next so-called panic" completely missing the fact that it was a non-event only because obscene amounts of effort went into fixing everything up (barely) in advance.

Invalid recommendations

Posted Nov 30, 2024 16:53 UTC (Sat) by Curan (subscriber, #66186) [Link]

That is probably correct, but it doesn't help me, does it? My spam levels haven't dropped with SPF/DKIM, so it is, from my POV, just pointless. But we can of course argue semantics here. And obviously I would never be able to prove, that the situation wouldn't be worse, if we didn't have these.

Invalid recommendations

Posted Nov 29, 2024 18:16 UTC (Fri) by farnz (subscriber, #17727) [Link] (6 responses)

My incoming mail volumes reduced by about 5% when I implemented both DKIM and SPF; I was facing a significant amount of backscatter (due to other people's badly configured MTAs accepting spam with my address as the "From", then sending a delivery report my way when they couldn't deliver it). On some days, backscatter was more than 50% of my total incoming mail.

Implementing DKIM and SPF removed virtually all of that; I can see from DMARC reports that people are still trying to forge my e-mail address as the reply target, but their mail is no longer being accepted. And the rate of forgery has nosedived; when I first implemented it, DMARC reports from the big players (Google, Outlook.com, mail.ru) showed that they were rejecting around 99% of mail that claimed to be from me. Now, they're not reporting any.

Invalid recommendations

Posted Nov 29, 2024 18:41 UTC (Fri) by Curan (subscriber, #66186) [Link] (5 responses)

OK, that is not my experience on my rather high-volume MTAs. But if it helps you, then my experience might be skewed.

Invalid recommendations

Posted Nov 29, 2024 19:05 UTC (Fri) by farnz (subscriber, #17727) [Link] (3 responses)

It wasn't my experience, until spammers started forging my address as the "From" address of their spam. At that point, backscatter went from "something I see once or twice a month" to 5% of my incoming mail. I implemented the entire set of DMARC measures more or less out of desperation, and it basically reduced backscatter to nil. It also, as far as I can tell from the DMARC reports, had a significant impact on the amount of spam that other people got with my address forged onto it.

Invalid recommendations

Posted Nov 30, 2024 6:01 UTC (Sat) by Curan (subscriber, #66186) [Link] (2 responses)

I will assume, that you run a private MTA, because your numbers make no sense on a larger base.

I am really sorry, you saw an increase in spam, but I will contest, that SPF/DKIM actually helped you. I will agree, that those technologies will have helped you to paper over existing issues, but in the end it won't matter. Today my most common spam sending e-mail domains are `gmail.com` and `hotmail.com` (ie.: Google & Microsoft, who have been pushing the hardest for SPF/DKIM).

And, TBH, it is in their interest to have these accounts. It just is not in mine and there is no automated way to prevent that e-mail from being processed by my local filters. Those catch the spam and sort it accordingly, but that also means I have to expend far more resources than I want to.

Invalid recommendations

Posted Nov 30, 2024 13:03 UTC (Sat) by pizza (subscriber, #46) [Link]

> I am really sorry, you saw an increase in spam, but I will contest, that SPF/DKIM actually helped you

Contest it all you want, but you're going to need a more reasoned argument than "I don't believe you" and quoting wikipedia definitions.

(BTW: I've been self-hosting my own email for over 25 years. The combination of SPF, DKIM, and DMARC made a *huge* difference on the domains I control)

> Today my most common spam sending e-mail domains are `gmail.com` and `hotmail.com` (ie.: Google & Microsoft, who have been pushing the hardest for SPF/DKIM).

You sound like the person who points at snow on the ground claiming that as proof that global warming isn't real.

> It just is not in mine and there is no automated way to prevent that e-mail from being processed by my local filters

Uh, nobody has ever prevented you from running your own client-side filters. (Except perhaps your employer, but that's their problem, not yours)

Invalid recommendations

Posted Nov 30, 2024 15:31 UTC (Sat) by farnz (subscriber, #17727) [Link]

OK then; if SPF/DKIM didn't help me, what exactly did reduce the load? I went from approximately 2 backscatter mails caused by spam with my address forged on it in a month, to typically 10,000 in a day, and the thing that caused it to go back down to approximately 2 in a month was implementing DKIM, SPF and a DMARC policy.

Further, having done this, I could see from the DMARC reports that for about 3 months after I set this up, big hosts like mail.ru, fastmail.com, Outlook.com, and Google were seeing typically 20,000 hosts not compliant with my DMARC policy in a day. That has now dropped to zero.

If it wasn't DKIM, SPF and a DMARC policy that helped, what was it?

Invalid recommendations

Posted Nov 29, 2024 21:15 UTC (Fri) by dskoll (subscriber, #1630) [Link]

I expect that it's no longer the case [that SPF "pass" is a mild spam indicator] because the standard is old and most people have caught up.

Invalid recommendations

Posted Nov 29, 2024 18:38 UTC (Fri) by dskoll (subscriber, #1630) [Link] (4 responses)

Neither SPF nor DKIM are anti-spam measures. They are designed to make it harder for someone to fake mail from your domain. Back when I ran an email security company (so 2000-2018) we found that an SPF "pass" was a slight spam indicator, because spammers were more diligent about maintaining correct SPF records than non-spammers.

Google et. al. demand SPF/DKIM/DMARC not to reduce spam, but to be able to hold senders accountable... if a piece of spam passes DKIM and SPF, we can generally know who (as in which domain or which MTA) was responsible for letting it out onto the Internet.

I agree that telling people not to run their own MTA is very harmful and is damaging to the integrity of the Internet. We cannot let an open and useful protocol slip out of the hands of the community and into the hands of a few powerful multinationals.

Invalid recommendations

Posted Nov 29, 2024 18:56 UTC (Fri) by Curan (subscriber, #66186) [Link] (3 responses)

> Neither SPF nor DKIM are anti-spam measures.

You should really tell this to various government bodies. Because they list this as requirements to combat spam...

Bud I do understand your position, it is just not, what is implemented in the world by big providers and required by government bodies.

> Back when I ran an email security company (so 2000-2018) we found that an SPF "pass" was a slight spam indicator, because spammers were more diligent about maintaining correct SPF records than non-spammers.

And just like that you told me, that SPF is pointless.

> Google et. al. demand SPF/DKIM/DMARC not to reduce spam, but to be able to hold senders accountable... if a piece of spam passes DKIM and SPF, we can generally know who (as in which domain or which MTA) was responsible for letting it out onto the Internet.

Oh come on, that is not helping anybody. Unless you try to tell me it is hard to get a domain. Which it is not, I must say. And if we assume getting a domain is easy: what does anybody gain by knowing "who" it was? The spammers will just change the domain.

Anyway, in my experience most spam comes from „legitimate“ domains like `gmail.com` and such. Making the whole endeavour circular at least.

Invalid recommendations

Posted Nov 29, 2024 19:47 UTC (Fri) by mbunkus (subscriber, #87248) [Link] (2 responses)

> And just like that you told me, that SPF is pointless.

You're wrong insofar as you're only thinking about fighting spam. But as Diane said, SPF & DKIM are supposed to establish authenticity. When I receive email from my customers (for whom my company has set up SPF, DKIM, DMARC in the past) I can rely on two things:

1. The email does indeed come from my customer & not from a malicious third party.
2. No malicious third party has modified the most important headers while in transit.

These particular technologies are not about fighting spam. They addressed different shortcomings of the SMTP protocol itself:

1. No way to verify that a sender (both in the sense of the program creating the mail & the server relaying the mail) is allowed to send mail for a certain domain. This is what SPF addresses.
2. No way to validate that email headers haven't been modified after having been sent by the sender. This is what DKIM addresses.[1]
3. DMARC as the third part finally tells the recipient that both SPF & DKIM are actually supposed to be used by all mail originating by a domain, and what the domain's owner wants the recipient to do if one of both or them are invalid or missing.

Again, this is _not about spam_.

[1] Before DKIM there had only been a way to protect email bodies from malicious modifications by sending cryptographically signed emails (GPG, S/MIME). Those didn't do anything for the headers, though.

Invalid recommendations

Posted Nov 30, 2024 5:33 UTC (Sat) by Curan (subscriber, #66186) [Link] (1 responses)

> When I receive email from my customers (for whom my company has set up SPF, DKIM, DMARC in the past) I can rely on two things

No you can't, because this still assumes honest players. Really, I am running MTAs for very large organisations and my experience is: a lot of legitimate e-mails would be filtered if I enforce SPF/ DKIM. On the other hand I have actual and obvious spam in troves, that meets SPF/DKIM requirements.

In my personal experience: the best way to verify a sender is OpenPGP. That being said: that is a negligible amount of traffic here.

But since you insist on the "not about spam" line: that is not how it is sold. And not how it is promoted. See eg. https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail where it says in the first line

> DomainKeys Identified Mail (DKIM) is an email authentication method designed to detect forged sender addresses in email (email spoofing), a technique often used in phishing and email spam.

Or in https://en.wikipedia.org/wiki/Sender_Policy_Framework

> Sender Policy Framework (SPF) is an email authentication method which ensures the sending mail server is authorized to originate mail from the email sender's domain.[1][2] [...] Forgery of this address is known as email spoofing,[3] and is often used in phishing and email spam.

And I know Wikipedia is not the best source, but if you are honest you will find, the RFCs and other industry sources agree.

---

That being said: I do not think us discussing this issue over text in a comment section is going to help us, because this is a very poor communication channel. If you are at FOSDEM, let me know and we can meet up in person and discuss this better. Probably over a beer or two. ;-)

Invalid recommendations

Posted Nov 30, 2024 7:20 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

> On the other hand I have actual and obvious spam in troves, that meets SPF/DKIM requirements.

FWIW, spammers don't even bother with forging the 'from' field anymore because they assume that SPF/DKIM is everywhere.

Virtualmin

Posted Aug 3, 2025 14:30 UTC (Sun) by pschneider1968 (guest, #178654) [Link]

I realize I'm a bit late to chime in here, but I want to do it nevertheless.

I do some hosting of web and mail domains for myself, for a couple of friends, and for my local chess club. I have the domains with IONOS, and the Debian server VMs with Strato for some 6-11€/month. I do use Postfix, Dovecot and Roundcube on the servers.

To configure all this on a bare Debian installation is way too much work, and error-prone, too. You need some good hosting software to make this easier and manageable. From my point of view, there are basically two viable software packages ("Admin Panels") that do this: one is Plesk, which is commercial, very good, but not cheap if it's not included with your hosting plan. The other one, and this is the one I'd really reommend, is Virtualmin. It's even better than Plesk (I think) and it's FOSS and free (as in "free beer") to use, with a subscription-based paid variant with some additional but non-essential features available, too.

https://www.virtualmin.com/

You can install it with either Apache or with Nginx as the webserver. I prefer Nginx, because it makes redirection to Docker containers really simple. With Virtualmin as the Admin panel, virtual domain hosting is really made easy. It does all the hard and complicated configuration work for you, but in a very transparent way. You can always override any setting manually by editing the individual components config file directly, but through the Web GUI.

The only things regarding the mail server I had to do fully manually was incorporating the free Spamhaus.org data query service (i.e. their free block list subscription) into the Spamassassin config, and the addition of OpenSRS (implementation of sender rewriting) to Postfix, because I do a lot of mail forwarding, as not all of my users want a mailbox; some prefer having their domain's mails forwarded to their private mail account.

Rspamd I looked into, but figured that at that point in time, it was serious overkill for what I needed. My setup was been working fine with Spamassassin for some years now. Virtualmin also allows for easy integration of mailbox virus scanning with ClamAV. And you can, albeit a lot more complicated than the basic setup, use LDAP for sender and mailbox accounts.

For securing your mail setup, the Email Hardening Guide is a very good read (short of reading THIS book) which can be found here:

https://www.mailhardener.com/kb/email-hardening-guide

It explains SPF, DKIM, DMarc and MTA-STS for mere mortals, and is helpful if you have some basic understanding of how DNS and SMTP work and relate to each other. You have to set this up correctly to get mails delivered to most of the big mail providers. Additionally, for some mail providers you have to go through some additional hops... For Google/Gmail, you have to have a Google account and register your mail server in the Google Postmaster tools, and then add a DNS TXT record to your domain with the Google verification code. In Germany, in order to be able to deliver to T-Online mail accounts, you have to contact the T-Online Mail admin team and ask that they add your mail server IP to their whitelist, which requires that you have to setup a web page for the mail domain with a legally required Imprint, satisfying German and EU law Imprint requirements.

With all setup correctly, I never had a problem delivering mails to Gmail, T-Online, web.de, gmx.de/org/net, yahoo, outlook.com/de etc.

And with the help of this great book, you can even gain some more deep understanding of these setup and security topics. But honestly, with a basic understanding AND the ability to read the Virtualmin documentation AND the Email Hardening Guide above, you are good to go!


Copyright © 2024, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds