Mozilla reconsiders H.264

Mozilla raised eyebrows in mid-March when a patch materialized that would allow Gecko to fall back on operating system or hardware media decoders for multimedia content — in particular for patent- and royalty-encumbered codecs like H.264, which are not supported natively in Gecko. The project had fought hard to promote the adoption of unencumbered alternatives (such as Ogg Theora or Google's WebM), so many on the mozilla.dev.platform discussion group saw enabling any support for H.264 as a violation of principle. Mozilla's Chief Technology Officer Brendan Eich argued, however, that the decision is an improvement over the existing Flash-fallback method, that Mozilla has more important fights to focus on, and that the blame for WebM's snail-paced adoption lies squarely at the feet of Google.

History of the WebM, part 1

Eich posted his take on the situation in a round-up at his own blog (which was then syndicated at the official Mozilla Hacks blog), which started with a history lesson on H.264, WebM, and the HTML5 <video> element. As far back as 2007, Eich and Mozilla had argued for the standardization of the <video> and <audio> elements to include "unencumbered" baseline codecs — at the time, Ogg Vorbis for audio and Ogg Theora for video. Eich argued that the term "unencumbered" most accurately describes the state of the codecs needed to ensure that the open web remains open; the issue is not about open source (for there are open source implementations of encumbered codecs), nor is it about patents (for standards bodies can and will accepted a patented codec if the patent holders agree to license it under royalty-free terms).

In 2007's <video> element fight, the main opponent of Theora was H.264, which was being pushed by a royalty-collecting consortium of companies. The protracted battle ultimately resulted in a non-decision, with the default codec language being removed from the draft standard. But the situation appeared to take a sudden shift in favor of unencumbered codecs in 2009, when Google purchased codec-maker On2 and released the WebM codec under unencumbered terms.

WebM was far newer than Theora and offered quality roughly on par with H.264; Theora's relative performance was a big reason Google did not advocate it as the default codec.. Google subsequently announced its intention to transcode YouTube videos to WebM, and in 2010 Adobe announced that it would support WebM in its Flash products (meaning not just the browser plug-ins, but the content-creation tools and media-delivery servers as well). In January 2011, Google went a step further, and publicly announced that it would drop support for H.264 from its Chrome browser.

But that change never happened. 14 months later, Chrome still supports H.264, and Eich and other Mozilla employees report that Google has remained silent about the decision when asked. Adobe didn't implement the WebM support that it promised either.

Meanwhile, Eich said, H.264 adoption has continued to spread, which has hampered Firefox's growth. For starters, Google's oft-cited promise to trancode YouTube's content to WebM is not all it is cracked up to be: to date Google has only transcoded half of the site's videos, and more importantly, YouTube only delivers WebM content to desktops, and only for those videos that serve no ads (which Eich said makes up a shrinking portion of the total). No other major sites have rejected H.264 in favor of WebM delivery either, while the consumer electronics industry builds more and more H.264 encoding support into video cameras.

On the desktop, Firefox falls back on the Flash plug-in's H.264 support, but on mobile devices, there is no such option. Mozilla believes that mobile browsers are clearly the battlefield deserving the most attention, and on top of that, the project's Boot to Gecko (B2G) effort stands no chance of making it into device-makers' products without H.264 support, Eich said.

Patches and a new API

The combination of Google not pushing WebM and content creators adopting H.264 puts Mozilla in an untenable position, Eich said. Yet the majority of phone and tablet designs ship with a pre-authorized (meaning the royalty fees have been paid) H.264 decoder in silicon, which is what led developer Andreas Gal to propose letting Gecko hand H.264 decoding (and perhaps other encumbered formats like MP3 and AAC) down to the OS or hardware level. Although other option have been discussed, Eich endorses Gal's solution. The upshot is that Firefox and B2G users will be able to see H.264 content on platforms that support it, Mozilla will not have to pay royalty fees (or pass them on to users), and the project can turn its attention to fighting for unencumbered codecs in the next round of standardization battles.

Those battles are not far off, Eich said, starting with the WebRTC real-time chat standard — which is already in-progress and looks poised to recommend unencumbered codecs. There will be other fights, he said, and other generations of video streaming codecs. Continuing to ignore H.264, particularly on mobile devices, would ultimately risk making Mozilla irrelevant further down the line, if not nonexistent altogether:

Gal's patch is attached to bug 714408. In essence, it creates a new API (which Gal dubbed the Media Player API, or MPAPI) for use by OS or hardware decoders. On desktop OSes like Linux, MPAPI would likely be tied in to a media framework like GStreamer (which can use encumbered codecs). It is also possible that the Mozilla Flash plug-in could continue to serve as the H.264 decoding chain on desktop systems, which would require work to expose MPAPI to the Gecko's main plug-in API, NPAPI. Of course, Adobe has also said that it will stop developing its NPAPI Flash plug-in for Linux, which means Flash-fallback will not remain a solution for free OSes in the long term.

As of now, the patch itself is the main source of information on MPAPI, which is still very much a work in progress. MPAPI would hand audio and video frames back to the main Gecko rendering toolchain, however, meaning the content would be fully integrated with the page's JavaScript, HTML, and CSS like any other <video> or <audio> element. The vast majority of the energy expended about the move has not been with its technical points, but with whether or not Mozilla's decision itself is wise, foolish, too pessimistic, or long overdue.

Reaction

The mozilla.dev.platform discussion thread about Gal's proposal is rife with critics arguing that Mozilla is making the wrong move, and with defenders inside and outside the project. The criticisms fall into three basic camps: those that think the H.264 situation is not as bad as described, those that feel Mozilla has not tried hard enough to advocate WebM (or that it should try Just One More Time), and those that object to enabling any form of H.264 playback on principle alone.

Critics who say that WebM has not lost to H.264 tend to point to Google's YouTube transcoding effort (which Eich countered in his blog post), or hold out hope that Google will indeed drop H.264 support from Chrome (often based on the number of WebM "supporters" listed). On the latter point, Eich argues that Chrome's H.264 support is moot, because the desktop browser would simply fall back on the Flash plug-in like Firefox does today. In fact, he said, Chrome's heavily optimized Flash plug-in amounts to a "practically custom" Flash offering "best-of-breed fallback."

Christopher Blizzard added that ultimately, the content-delivery sites simply are not interested in WebM:

Justin Lebar typified the position that Mozilla is giving up too quickly, calling it "going down without a fight," and saying that he would:

Side-stepping the fact that Lebar's public deadline idea paradoxically threatens to increase support for H.264 if it is not abandoned, Mozilla's Robert O'Callahan calls it "grossly unfair" to suggest that Mozilla has not fought hard enough or long enough against H.264 adoption:

Likewise, Asa Dotzler confirmed that Mozilla has spent months trying to get a response from Google on the H.264 support question, only to be met with silence. O'Callahan also observed that there is already mobile hardware capable of decoding WebM video, but that Google does not enable it on Android devices. In the absence of support from the format's owners, Mozilla says, it alone cannot move WebM forward.

The objection on principle is trickier. Some in the discussion thread expressed personal hurt that Mozilla was not standing its ground against H.264 playback support, but more were concerned that relenting would make it harder for the organization to lobby against encumbered formats in the future. Eich argued that anyone who ignores the fact that Firefox users watch H.264 video via the Flash plug-in is "hiding behind Mother Adobe's skirts" and is not taking a "realistic view of the entire fallback logic chain, and of Firefox's current acute dependency on Flash," which is not different in kind from falling back on an OS decoder. Gal concurred, noting that the MPAPI proposal is only "using existing accelerated decoders that already are licensed and available on the system."

Regarding Mozilla's ability to advocate for open and unencumbered formats in the future, Doztler said that the project has backed down on other lost battles in past, such as the document.all DOM feature, but has maintained it credibility. Mozilla can influence the web, and from time to time kill a bad idea, he said in another message, but "sometimes the Web decides to go where we don't want it;" not supporting it only costs the project developers and users.

Ultimately, though, Dotzler argues that even in light of H.264's popularity, Mozilla's WebM advocacy does not constitute a wasted effort:

A continuing story

It is worth remembering that however one feels about the H.264 lobby and its royalty-collecting schemes, the presence of dedicated video decoding chips is hardly an isolated situation. There are currently a great many components in our computers which are covered by patents, and many chips for which we do not have source code. Yet enabling software access to those components is not perceived as a violation of principle. Furthermore, it is hard to argue that the Flash plug-in that Firefox currently falls back on is turf worth fighting for — it has a history of bugs and security holes unrelated to the video decoders it ships.

The debate over enabling H.264 playback via MPAPI shows little sign of calming down. Eich, Dotzler, Gal, and the other project members continue to argue that enabling the format is a purely pragmatic move, and that it would be a better use of Mozilla's energy to combat encumbered codecs on the still-in-development format battles.

They got a boost on March 18 when Mozilla chief Mitchell Baker posted her own blog entry in support of the proposed change. Baker said that "giving our users a great experience" is both one of the project's key values, and a demanding goal that drives realistic product development.

The comment thread on Baker's blog follows much the same pattern as the discussion group. There are supporters, commiserators, and vocal critics. But wherever H.264 itself ends up on future versions of Firefox and B2G, one thing is for sure: H.264-vs-WebM is not the last codec fight the software world will see. As several in the thread pointed out, progress on H.265 is already well underway, and the players involved are similar — there can be little doubt that the battle will be similar, too.