Mailman 3.0 to modernize mailing lists

More than a decade after its last major rewrite, the GNU Mailman mailing list manager project aims to release its 3.0 suite in April, during the sprints following PyCon North America. Mailman 3 is a major rewrite that includes a new user membership system, a REST API, an archiver replacement for Pipermail, and a better web interface for subscriptions and settings — but it carries with it a few new dependencies as well. Brave system administrators can try out the fifth beta version now.

Mailman has been a ubiquitous if slow-moving part of the social infrastructure for many open-source projects for decades. More than fourteen years have passed since the Mailman 2.0 release in November 2000, twelve years since the first release of the last stable version (2.1), and seven years since first alpha of the 3.0 branch. However, Mailman has stayed under continuous development during that time; new 2.0 branch releases have come out about twice annually, with new features in point releases (e.g., the DMARC mitigations added in 2.1.16 to keeps lists from running afoul of an anti-spam measure). It supports moderation, list digests, bounce detection, spam protection, multilingual lists, archiving, and list administration via email or web interfaces, among other features. The new release aims to get rid of some annoyances and to refresh Mailman for administrators' and users' current needs.

What's new

The architecture and user interfaces of previous versions of Mailman reflect a different era of the web, and of application interoperability. Mailman 2 was a single codebase, written in Python 2, encompassing a rudimentary web application for subscription and list management (and incorporating the Pipermail web archiver) as well as the engine for receiving, moderating, and propagating messages. Lead developer Barry Warsaw explained in the overview he wrote for The Architecture of Open Source Applications that, beyond the browser-based interface, Mailman 2 also offered a dedicated command-line interface, and a Python internal API that system administrators could integrate with by writing Python code.

In contrast, Mailman 3 is a suite of five connected projects, each of which can run independently:

• The Mailman core provides the backend engine that interacts with Mail Transfer Agents (e.g., Exim, Postfix, sendmail), manages users, messages, and lists, and processes mail based on the lists' rules (e.g., filtering out spam, queueing messages for moderation, distributing them to a list's membership). Mailman core also exposes a RESTful API to provide easier integration for other applications or scripts. Mailman core is written entirely in Python 3, thus requiring system administrators to have Python 3 installed.
• mailman.client is the official Python bindings for the REST API, and requires Python 2.6 or newer.

• HyperKitty is a Django-based archiver application that replaces Pipermail, which we previewed last April. It provides a modern web interface to browse and search archived messages and threads (see this live demo); interestingly, it also allows users to post to discussions, so that by default users can interact with any Mailman mailing list as though it were a web forum, if they prefer. HyperKitty is written in Python 2. It uses mailman.client to interact with the core engine for administrative purposes, and uses Mailman core's Python-based IArchiver API to access the mailing list posts that have been successfully posted and thus should be archived.
• Postorius is a Django-based web application for administering lists and managing user preferences. The refreshed appearance (see screenshots) and theming capabilities should make it easier for communities to integrate Postorius into their websites without a jarring contrast. Postorius is written in Python 2 and, like HyperKitty, uses mailman.client to interact with Mailman's core engine.
• The bundler is a set of small Python scripts to bundle Mailman components together for easy installation.

Mailman 3 also provides improved abstractions for dealing with lists and users. Instead of using pickled files, Mailman 3 stores configuration data in a SQL database (by default, SQLite). This and other new features make it easier to (for instance) manage multiple mailing lists with the same name in different domains (e.g., dev@example.com and dev@example.org), give someone administrative power over all the lists within a domain, and provide a unified user interface to a person who subscribes to multiple lists with different email addresses.

To answer the question that at least a few dozen of you are looking for: Mailman 3 will not, by default, send you a monthly reminder that includes your password in plain text. In fact, it can't; Mailman 3 hashes passwords before storing them.

Implications

Developers are aiming for Mailman 3.0 to be at feature parity with Mailman 2.0, so users should not experience much degradation in functionality, but there are a few newer features in 2.1 that have not yet been ported forward into 3.0; this is still being finalized. Localization is still up in the air. The Mailman community is committed to improving the currently labor-intensive process it uses for managing string translations, but the developers have not yet decided on a platform for translators.

The new release includes several usability updates, such as the HyperKitty interface that allows subscribers to interact with lists via the web, and list subscription management improvements on the backend. However, at first, system administrators will probably be hesitant to adopt the new release. In order to run all the components of the Mailman 3.0 suite, they will have to be willing to install more dependencies (such as Django) and install and run two different versions of Python. Mailman 2.1 may seem serviceable enough; for core infrastructure like mailing list software, user interface improvements and better scriptability may not make this upgrade feel urgent.

The new features may change many open-source citizens' habits, such as subscribing and auto-archiving a list just to be able to search it easily, and using and linking to third-party sites like Nabble and gossamer-threads for convenience in discussing threads. And the web-native feel of HyperKitty (whose interaction design was led by Máirín Duffy and Karen Tang) may help Mailman compete with proprietary hosted list-style competitors, such as Google Groups, and with open-source forums that feel web-native, such as phpMyBB, Discourse, and Community.

Mailman's developers have gone with current trends in the Python community by using Django, but have made a bold move in converting the core engine to a Python 3 codebase. For many system administrators, Mailman 3.0 will be the first application that requires Python 3; in fact, Warsaw wrote some core Python code in order to support Mailman's use cases. It will be worth watching to see how this influences the overall adoption of Python 3, for which the rollout has been gradual.

The new release's REST API provides a more modular and flexible way for administrators and developers to reuse the Mailman core engine in other applications, replace Postorius or HyperKitty with custom web applications, or otherwise script and hook other actions into mailing list traffic. Mailman's API also goes further with its REST interface than many other web APIs, as it takes a particularly hypermedia-centric approach to resource representation and retrieval. It (unusually) even uses the HTTP verbs PUT and PATCH, which will exercise some of the less full-featured HTTP interaction libraries.

With its large suite of unit tests and a dedicated, active, and friendly community of creators and users, Mailman seems poised to continue improving, and to stay a premier competitor among open-source mailing list managers. Its new architecture and dependencies also point to continuing trends in server-side applications in general. For open-source communities already using Mailman 2.x, this is a welcome update, and its rollout will make open source as a whole a bit easier on the eyes.