Consolidating server messages with fedmsg

One of the recurring themes at Flock 2014 in Prague was the importance of fedmsg to the Fedora distribution—and, increasingly, to other projects. Fedmsg is a project-wide messaging bus that shuttles notifications from numerous services in the Fedora infrastructure to interested listeners. It replaces a long list of heterogeneous message-generating packages from individual servers and provides uniformly formatted message output that can be consumed by a wide array of applications.

Fedmsg was the subject of just one Flock session (Ralph Bean's workshop for developers wanting to add fedmsg support to their applications), but it came up in the course of many others. The most notable example, perhaps, was Kevin Fenzi's talk about the current status and future plans for Fedora's infrastructure—which is where fedmsg has made the biggest impact.

Like any major distribution, Fedora requires a lot of servers to keep things functioning smoothly. Fenzi put the current number at 460, divided up between seven data centers in various locations. The servers handle source code management, bug tracking, package repositories, build systems, QA, mailing lists, wikis, web pages, and more—as well as the behind-the-scenes duties that keep the other services running (such as monitoring and mirroring). Although there are tools available to make wrangling that many server instances a task mere mortals can manage (and Fedora takes advantage of them), several years ago the project realized that there was a missing piece that could simplify day-to-day operations for Fedora developers and packagers.

The missing piece was a way for one server to issue a notification to another. Many of the services, such as the Koji build system or Bodhi packaging server, could already generate messages of one type or another. Some of the possibilities included general-purpose output like RSS or email, but many services offered only an API for which custom code are required if one wants to consume updates in another application. Thus, connecting any two services generally required adding glue code to parse the other service's messages and often required running a cron script to execute the appropriate action in response to recurring message traffic.

Fedmsg was developed to unify these various message-passing connections between different services, using a well-defined message format and a shared bus. Each participating service publishes messages to the bus under a particular "topic," the syntax of which is reminiscent of D-Bus connection names. For example, org.fedoraproject.service.fedpkg.post-update is the topic for messages generated by post-update hooks on the fedpkg Git server. Other services (as well as applications) can subscribe to topics of interest or simply listen to every message—which is referred to as drinking from the firehose.

The messages themselves are JSON encoded, with a payload that usually consists of a serialized Python object dictionary. An example from the documentation says, for instance, that a message noting that a user has edited his or her Fedora Account System (FAS) profile would include (in its dictionary) the username and the names of the profile fields that have changed. In that case, the message body alone would not make much sense, but the topic (org.fedoraproject.service.fas.user.update) provides listeners with the correct context. Messages can also be signed, either by OpenPGP keys or with X.509 certificates.

Although the initial plan for fedmsg involved a central message broker, the design today is entirely decentralized. Messages are published with ZeroMQ. The decentralized topology, naturally, requires that there be some form of service discovery so that participants joining the bus can find other services to listen to. Because Fedora's instance is not open to any and all random participants, the project instead distributes a plain text file describing the bus configuration to each service.

Fedmsg was rolled out (slowly at first) in 2012. As of today, the topology in use by the project includes Koji, Bodhi, Trac, and a number of other infrastructure services, plus a separate relay to connect servers behind the project's firewall (namely package repositories). There are also several listen-only services, such as an IRC channel, an RSS feed of the entire firehose, and a hub that connects to various web applications.

Although the original use case for fedmsg focused on service-to-service features (such as a Git commit triggering a Koji build, which would in turn trigger QA tests), the web-application hub enables another class of uses: notifications to users and developers. The Fedora Message Notifications (FMN) system is designed to publish messages to individuals over email, private IRC messages, and even desktop pop-ups.

There is a status page detailing what Fedora services are fedmsg-enabled. At present it is quite a long list, although there are some noticeable absences, such as the project's Bugzilla and the Taskotron test automation system.

As fedmsg's recurring presence in Flock talks and workshops indicates, the system has already had quite a positive impact on the day-to-day workings of the machinations that keep the Fedora distribution running. But an even better testament to its value may be that other projects have started to deploy their own fedmsg systems. The most notable example is Debian, which began a fedmsg project in 2013. The Debian system is not nearly as fleshed out as Fedora's, but interested observers can join the #debian-fedmsg channel on irc.debian.org to see the firehose. Package uploads and the Debian bug tracking system (BTS) are working already; more services are in the works.

As one might guess, the name fedmsg originally stood for "Fedora messaging." Several people at Flock, however, pointed out that since Debian has started making use of fedmsg, too, the "official" name for the project has been altered to mean "federated messaging." Those same people also pointed out that fedmsg is not yet actually federated—Fedora's system and Debian's cannot communicate with one another—although perhaps such a feature will come in time. Until then, the framework has still proven its value for the job of unifying and connecting the heretofore cacophonous glut of messages, updates, and notifications that seem to be constantly generated by the service infrastructure required to keep a modern Linux distribution running smoothly.

[The author would like to thank the Fedora project for travel assistance to attend Flock 2014.]