A preview of HyperKitty's reimagined mailing list interface

The first beta for version 3 of the GNU Mailman suite was released on April 24. The upcoming version of the mailing-list management program will bring a host of new features, among them a completely new web-based front end called HyperKitty that is designed to bring list archiving up to modern user-interaction standards. HyperKitty looks more like a web discussion forum than Mailman 2's list-of-links archive pages, and it integrates better with other web applications, all while retaining the features expected of mailing lists.

Although it is difficult to calculate the precise deployment numbers, GNU Mailman is so widely used as a mailing-list manager that it scarcely has any competition, particularly among free-software projects. Years ago, mailing-list management was dominated by the proprietary LISTSERV and Majordomo. Mailman quickly won over a lot of converts after its debut in the late 1990s, although it has not had a major update in several years. Recently, Mailman has seen stiff competition from a different angle: web-based discussion forums. Between bulletin-board forum packages like phpBB and self-contained sites like Reddit and StackExchange, email-based discussions have lost noticeable ground in recent years in many places.

The reason for such a shift is up for debate; perhaps some users loathe the email-driven sign-up process required to join a discussion, perhaps others prefer to intentionally visit a site rather than be bombarded by replies delivered straight to the inbox. But many people, including those in the Mailman project, seem to agree that the web front-end to Mailman 2 offers an experience far less comfy than that of most web forums: plain-looking and hard-to-navigate archives, less-than-smooth handling of threads, the inability to reply from the web interface, and a number of missing features that—for better or worse—users have come to expect (such as user profiles and rating messages up or down).

Mailman's current archiver component, Pipermail, also suffers from some serious technical limitations. It does not integrate easily with other web frameworks, it offers weak protection against email-address harvesting, it does not scale well, and archived messages do not have stable URLs. Lack of stable URLs means that if the archive needs to be regenerated (or indeed if some other administrative actions like message deletion need to be taken), the existing URLs break, causing consternation to search engines, bookmark users, and third-party applications.

More modular mail machines

The architecture of Mailman 3 splits the application into a core module that communicates with other applications through a REST API. The two applications bundled with the core are the list-administration interface Postorius and HyperKitty, the new archiver front-end. However, either component could be replaced in a specific Mailman deployment, so that admin or archiving functionality could be built directly into some other web application or framework. Despite such flexibility, of course, the Mailman project has set out create the best archiver that it can, which can be seen in HyperKitty's new feature set and its user interface.

Work on HyperKitty started in early 2012. Much of the front-end implementation stems from design work spearheaded by Máirín Duffy, while backend development has come from developers at Red Hat and several Google Summer of Code students. HyperKitty includes a Django application with a Bootstrap-based user interface, plus a storage backend called KittyStore (which can run on PostgreSQL, SQLite, or MySQL).

KittyStore provides stable URLs for archived message pages, since Mailman 3 generates the URL for each message based on the RFC 2822 Message-ID header, which is designed to be globally unique. Unique, stable URLs offer some advantages beyond simply not breaking every now and then. For example, they can be pre-computed based on the Message-ID header, so Mailman can insert a permanent link into the message footer when it is sent out to the list recipients. For that purpose, Mailman 3 uses the Archived-At header format described in RFC 5064.

In addition to the message archives themselves, KittyStore tracks basic user profiles for subscribers, including a simple "reputation" that counts up and down votes registered by the community on the author's posts. It also provides full-text search capabilities (which Pipermail did not) via the Whoosh library. That not only removes the need for lists to bolt on a site-specific Google search widget, but it allows private lists to be searchable by their subscribers.

The hyper experience

The visual makeover updates the aging basic-HTML look of Pipermail, but it, too, adds functionality. A demo server is running on the Fedora infrastructure with mirrors of the distribution's mailing lists; it is a good place to see the new interface in action. The server overview page allows searching and filtering through the hosted lists and displays a sparklines-style graph of each list's recent activity.

The bigger changes, however, are found beginning in the individual list archive pages. Statistics are provided for each list, including the most recent and most popular threads and the most active posters. By default, HyperKitty gives each thread its own page (as opposed to one page per message in Pipermail), though there is a permalink for every message. The thread rendering is reminiscent of Gmail or the commenting systems in WordPress and other blogs; it shows the sender's name and an avatar image at the top of every message (as opposed to the sender's email address), indicates the number of up and down votes, shows attachments with a paperclip icon, and includes a "Reply" button at the bottom.

Obviously, list subscribers can already reply via their email client, but HyperKitty allows them to respond through the web interface as well, first authenticating them against the Mailman server. But HyperKitty can also let new users reply to messages in the web interface: unsubscribed users are prompted to subscribe to the list when they click on the "Reply" button. Naturally, it is important for list administrators to clearly explain the implications of subscribing to the list; to users not proficient with mailing lists, however, the sign-up process closely resembles that used by most other forum software.

HyperKitty also attempts to provide navigation elements on each thread page that are easy to find but do not overpower the message content. There is a breadcrumb-style header on the top of each page (leading to the list home page, monthly archives, and back up to the overview of all lists), plus "Newer thread" and "Older thread" links that include a preview of each thread subject. There is also work underway to tag or categorize threads, although how useful that feature is remains unclear on a demo server. HyperKitty also tracks which threads a browser has visited, so users can easily see read and unread threads.

Future cats

Improving mailing list archives is a multi-faceted problem—even the major features discussed above can sometimes be in conflict. For example, the stable URLs of a HyperKitty archive include a lengthy Base32 hash derived from the message headers. Though it does not change, it is more difficult to remember than Pipermail's simple "1234.html"-style URLs. It is also certainly possible that some list administrators and subscribers will have no interest in web-oriented features like user-profile pages and up/down voting.

On the whole, though, HyperKitty looks far better than Pipermail, and not just in the aesthetics of the theme. Small touches make it more readable, like highlighting just the sender's name on each message, rather than showing multiple email headers and several lines of links back into the archive pages. Better default settings, such as one-page-per-thread, reduce the amount of time that a user has to spend clicking and waiting for pages to load. Statistics are not the most popular things in the universe, but by keeping them unobtrusive on the page, they manage to be helpful. And in the long run, improving the look of the web archiving component is not merely a cosmetic alteration: it makes the software—and the mailing list—more useful to the users.

Perhaps the ultimate test of HyperKitty's usability would be whether it actually makes Mailman seem friendly enough that users who prefer web discussion forums are comfortable reading and replying through it. After all, the free-software community has long grappled with the segmenting of discussions into separate silos; removing some of those obstacles would be a feat. Then again, what makes HyperKitty possible is Mailman 3's REST API; it the long run, perhaps Drupal, Joomla, MediaWiki, and other popular web applications will simply develop their own Mailman integration and the day will come when email-only discussions are a historical novelty.

There are other free-software list efforts, such as Groupserver, intent on modernizing the usability of mailing lists and there are other discussion-forum projects, such as Discourse, intended to provide a alternative to the current integrate-with-Facebook (or Google) option so many web sites seem to be opting for. If luring site maintainers away from proprietary web services for their discussion threads is the goal, there are several options.

If revamping a mailing list's user experience is the goal, though, Mailman 3 with HyperKitty is clearly a big step in the right direction that many list administrators will welcome. Mailman 3 has been in development for years; although there is no set timetable for when it will be declared stable, it looks like an update worth talking about.