Why Debian returned to FFmpeg

Slightly less than one year ago, the Debian community had an extended discussion on whether the FFmpeg multimedia library should return to the distribution. Debian had followed the contentious libav fork when it happened in 2011, but some community members were starting to have second thoughts about that move. At the time, the discussion died out without any changes being made, but the seeds had evidently been planted; on July 8, the project's multimedia developers announced that not only was FFmpeg returning to Debian, but it would be replacing libav.

Chances are that many Debian (and Ubuntu) users are not more than peripherally aware of which multimedia library is running on their system. Libav has been in use for some years and has generally filled the bill, so it is natural to wonder what drove the project to make a change that will require a lot of work and which seems certain to prove disruptive while it is underway. Getting to the answers is an interesting study in how distributions try to ensure that the code they ship comes from healthy upstream projects.

Security and more

The Debian project is not normally known for being afraid to ship multiple packages providing the same function, so one might wonder why it can't just ship both FFmpeg and libav, letting users decide which one they want to use. The big sticking point in 2014 was security support. Both projects have had more than their share of security issues, and the Debian security team didn't think it could keep up with patching both of them. At that time, FFmpeg seemed to have a better record of responding to vulnerabilities than libav, but that still was not enough to convince the security team to support both libraries.

One year later, security issues remained at the top of the list, but it would appear that FFmpeg has pulled well ahead of libav in this regard. Debian security team member Moritz Muehlenhoff made it clear that he saw FFmpeg as being more responsive to security reports. A rather stronger argument came from Mateusz “j00ru” Jurczyk who, in his security-oriented role at Google, has been doing extensive fuzz testing of both projects and reporting the problems that come up:

The notion of hundreds of open security issues is generally unappealing. FFmpeg does not appear to lead on just security updates, though; by all accounts, it now supports a far wider variety of codecs and containers than libav does. There is an increasing range of formats that FFmpeg can play, but libav cannot. As the feature gap grows, the project's desire to stay with libav wanes.

The libav maintainer's perspective

When Alessio Treglia restarted the discussion at the end of April, the above points were quickly expressed. Even so, the conversation did not appear to be heading toward any sort of consensus. Arguably, the turning point was when Debian libav maintainer Reinhard Tartler entered the discussion. Reinhard argued forcefully for the advantages he saw in libav, but, in the end, could not bring himself to say that he was sure libav was the better choice.

With regard to security issues, Reinhard attributed the difference in fix rates to a difference in how the two projects approach development ("Michael" is Michael Niedermayer, the lead developer of FFmpeg):

Reinhard initially asserted that, even so, libav had parity with FFmpeg when it came to fixing security-related bugs, but he later backed down on that.

In Reinhard's view, the two projects are managed differently, with different goals; that difference makes libav appealing in a number of ways. Libav, he said, is trying to improve the state of the code and come up with something better than the "horrible" APIs it inherited from FFmpeg. He summarized the differences between the project this way:

Even through he seems to like the libav approach more, Reinhard, in the end, was not able to argue against the change; his position came down to: "I still have some concerns with this move, but I can't claim Libav to be superior to FFmpeg at this point". With the project's libav maintainer taking that position (and also, importantly, saying that he no longer has the time to maintain libav at the same level as he has in the past), the decision seemed to settle out fairly quickly.

Other concerns

A desire that was expressed more than once in this discussion was that the two projects would stop fighting and join back into a single, well-supported effort. There is, however, no real indication that any such reconciliation is in the cards. There is another way that the community could go back to having a single project, though: if one of them were to simply fail. Dmitry Smirnov suggested that a switch to FFmpeg by Debian could maybe bring that about:

Opinions vary on how much "life support" Debian actually provides to libav, but the loss of Debian and Ubuntu seems certain not to do the project any good. There aren't a lot of distributions out there that carry libav anymore; without Debian, that list will be short indeed. It might just be that libav is not sustainable without Debian.

That said, there are some concerns about the sustainability of FFmpeg as well. By all accounts, Michael is a highly productive developer; he accounts for, by far, the largest share of the patches going into FFmpeg. Reinhard asked whether FFmpeg is a one-developer project that would find itself in trouble should Michael stop working on it. "To me, this constitutes a serious bus-factor: Without Michael, (probably) nobody is able to replace him." He went on to suggest, though, that Michael's departure could do a lot to bring an end to the fork.

As an argument against the "one-man show" concern, Andreas Cadhalpun posted some commit statistics for both projects, covering the period since September 2014:

libav		FFmpeg
Commits	Developer	Commits	Developer
294	Vittorio Giovara	1831	Michael Niedermayer
253	Martin Storsjö	294	Vittorio Giovara
206	Anton Khirnov	252	Martin Storsjö
131	Luca Barbato	197	Anton Khirnov
72	Diego Biurrun	179	Clément Bœsch
46	Michael Niedermayer	155	James Almer
32	Rémi Denis-Courmont	150	Carl Eugen Hoyos
21	Andreas Cadhalpun	114	Andreas Cadhalpun
17	Hendrik Leppkes	113	Luca Barbato
16	Gabriel Dume	98	Lukasz Marek
16	Himangi Saraogi	93	Paul B Mahol
16	wm4	85	Ronald S. Bultje
14	Federico Tomassetti	83	wm4
12	Peter Meerwald	66	Christophe Gisquet
11	Janne Grunau	48	Benoit Fouet

At a first glance, the table shows that (1) FFmpeg appears to have a much higher commit traffic than libav, and (2) Michael, while being the largest contributor, is certainly not the only contributor. But, as Reinhard pointed out, there is a bit more to this story. Changes to libav are routinely merged into FFmpeg, but the flow of patches in the other direction is quite low. If the libav changes are subtracted out of the FFmpeg numbers, the result is that Michael very much stands alone; no other developer is even close.

The Debian multimedia developers decided to make the switch to FFmpeg even though nobody really had an answer to Reinhard's concern. For now, FFmpeg appears to be going strong, but there is a single-developer risk there that could come to the fore in the future. Given that nearly the entire distribution ecosystem now depends on FFmpeg, chances are that a way would be found to keep the project going if Michael were to decide he had better things to do. But the process of getting there might prove to be a little rough.

The Debian project was faced with a difficult choice: given that it was not possible to support both libraries in the distribution, which one offers the most to Debian's users while presenting the least long-term sustainability and security risk? The developers involved chose to move away from a project that many of them see as lacking the resources needed to be truly healthy. That choice will result in a lot of work, but, assuming the choice was the correct one, Debian users should benefit in the long term.