HTML5 video element codec debate reignited

On January 20, YouTube publicly unveiled a video player that allows site visitors to watch videos embedded directly into each page as HTML 5 video elements, replacing the plugin-based Flash player — and second-tier video sharing site Vimeo quickly followed suit. But both sites serve up HTML 5 video files only in the patented and royalty-collecting H.264 format. By sheer coincidence, the announcement neatly overlapped with the release of Firefox 3.6, and was followed days later with Apple's press event showcasing its iPad gadget, which lack H.264 and Flash support, respectively. What followed was a furious multi-way debate all about Flash, licensing, web video, and H.264 versus Ogg Theora. For the open source community, there is nothing to celebrate yet, but the high profile of the argument has opened the door for discussion of the real underlying issue: patented web standards.

Rewind

The root of the entire controversy is HTML 5's video element, which allows a web developer to include video content in a web page in any file format, obviating the need to wrap such content in a Flash player useful only because of the Flash plugin's ubiquity. But it is up to the browser to include support for the formats it chooses in its built-in video player. The HTML 5 standard does not mandate that support be included for any particular format in order to qualify as compliant, however, so a public war is underway between format proponents for de-facto dominance.

On one side is the ISO Moving Picture Experts Group (MPEG), pushing for adoption of its H.264 format. The H.264 codec is part of the broader MPEG-4 family, is patented, and all parties wishing to include support for it are required to pay licensing fees to the patent holders through a consortium called the MPEG-LA — the licensing requirement applies to encoders and decoders, hardware and software, and includes both original manufacturers and downstream redistributors.

Many on the other side are supporters of the free Theora format, which requires no royalties to implement in hardware or in software, thanks to irrevocable free licenses on the original patents granted by its original creator. The reference encoder and decoder are developed by Xiph.org and are available under a BSD-style license.

Theora proponents emphasize the need for HTML 5 to include a free-to-implement format, insulating the next decade of web development from the nightmare caused by the GIF patent enforcement debacle. H.264 supporters claim that Theora's quality-per-bitrate performance is behind H.264's, and that some unknown third-party might hold secret patents on one or more techniques used in Theora, and subsequently sue implementers for patent infringement if the format is made part of the standard (the so-called "submarine" patent threat).

The major web browsers are divided on format support. Apple's Safari ships with H.264 support only, Google's Chrome supports both H.264 and Theora, Firefox and Opera support only Theora. Microsoft's Internet Explorer does not support HTML 5 video at all. Confusing the mix slightly is the fact that both Safari and Chrome implement H.264 playback because their parent companies pay licensing fees to MPEG-LA; consequently the open source browser projects WebKit and Chromium do not support H.264, because the license fees paid do not cover these downstream derivatives.

Players

That, then, was the situation when YouTube and Vimeo announced their H.264 HTML 5 video player support. What should have been a red-letter day for open web standards instead resulted in complaints to Mozilla from users (and pundits) that Firefox 3.6 "did not support HTML 5." In fact, Firefox has supported HTML 5 video since version 3.5, but it does not include an H.264 decoder.

Video expert Silvia Pfeiffer traced the problem back to numbers. According to Statcounter's market share statistics, Firefox accounts for 22.57% of the browsers in the world, with Chrome and Safari totaling 8.53%. Thus, of all the HTML 5-capable browsers in the field, Firefox makes up nearly 73% — and that 73% could not watch any of the YouTube or Vimeo video. It should be no surprise, then, that some of those users complained.

Mozilla's Christopher Blizzard responded to the news with a detailed analysis of the H.264 ownership and patent problem. The situation is precisely the same as the GIF disaster of a decade earlier, and as the MP3 situation from the early 2000's — but with considerably higher stakes. H.264 is patented, pure and simple, and the patent owners charge royalties today and will continue to do so until their patents expire. If H.264 becomes a de facto standard, the patent owners will have the freedom to hike the price of licenses, and they will no doubt do so.

Blizzard goes on to examine the terms of H.264 licensing and its effects on corporate and independent producers of web content. To include an H.264 decoder in Firefox, Mozilla would have to pay a license fee (perhaps $5 million per year), but such a move would also undermine Mozilla's founding principles of supporting and promoting free formats and standards.

Flash, we hardly knew ye

The other big news from the last week of January was Apple's iPad launch party. The iPad, like its diminutive siblings the iPhone and iPod Touch, uses a Safari-based web browser, and includes Apple's licensed H.264 decoder for HTML 5 video. But also like the smaller devices, the iPad does not include Flash support.

Coming from Apple, that decision was hailed by some in the media as a death knell for Flash. Once the preferred format for incorporating animation and interactive page elements into web content, in recent years its usage has shrunk to the point where it is used almost exclusively as a platform to deliver online video (and for irritating advertising, of course, although strictly speaking that would not be considered "content" by most).

No one seems to lament the possibility of Flash's demise. Apple has suggested that Flash is the cause of most of the Safari crashes reported through its OS X Crash Reporter utility. Mozilla said in October of 2009 that third-party plugins cause at least 30% of all Firefox crashes, a statistic supported by the popularity of Flash-blocking add-ons.

Apple's Steve Jobs even went so far as to publicly call Flash too buggy for use in a town hall meeting last week, declaring HTML 5 the way of the future.

What's a site owner to do?

Flash may indeed have no fans remaining outside of Adobe, a fact that magnifies the importance of HTML 5 video codec battle. The plugin has survived as long as it has for one reason alone: its availability on almost every browser on almost every operating system. Long after AJAX became popular for interactive content functionality, a web developer could implement video playback in a Flash element and feel secure that it would work on virtually every browser that would encounter it.

The same cannot be said of HTML 5 video, and certainly not of HTML 5 video with H.264 content. If Theora becomes the dominant format (or officially sanctioned in the HTML 5 specification), it will be possible again, but that is simply not true of H.264. Both encoders and decoders require licensing; a fact often overlooked in the debate about browser support, but one which Blizzard addresses in his blog entry. Anyone can set up a site delivering CSS, HTML, and even Theora using free, legal tools, and without asking or signing for permission; H.264 would change that.

The only question is whether or not the web development community will recognize that and rally behind Theora or another free alternative. The H.264 patent owners' attacks on Theora are not substantive; the quality comparison is highly subjective (and, in fact, comparing video encoding quality is inherently subjective), and as Xiph.org points out, submarine patents are an equal threat to free and non-free codecs alike. The original patents on Theora technology are known and licensed freely; if a patent owner possessed sufficient evidence to kneecap Theora with an infringement lawsuit regarding other patents, it surely would have happened already.

Moreover, the HTML 5 video element includes support for multiple source files, so content providers can offer each video in multiple formats; the fight is only the H.264 patent holders trying to prevent a rival format from being blessed as part of the standard. Those patent holders would take the same tactics with any other video format.

Some critics have suggested that another free video codec is needed, and Theora is certainly not the only option. Sun has been developing its own patent-avoiding video codec through the Open Media Commons project for several years, although the project is rather quiet. Blizzard suggests that Google may have a video patent play of its own in mind with its recent attempts to acquire On2, the company that developed the VP3 codec from which Theora descended. Dan Glidden, formerly of the Open Media Commons project, is a proponent of the MPEG-RF movement to change MPEG policy to establish a royalty-free option as a "baseline" codec for MPEG-4.

The debate is far from over. YouTube and Vimeo may have changed one aspect of it, however — unlike in years past when the fight took place almost entirely within World Wide Web Consortium working groups, this time it is being fought in public. Consequently, more people are getting a look at what HTML 5 video is in practice, and can better understand the difference between the HTML element and video format delivered, which can only be a good thing.

In the meantime, small web developers who want to serve up HTML 5 video content still have choices. The simplest option is to include multiple video source files, but a better alternative is to use the Cortado applet from Xiph.org; a streaming media Java applet that decodes Theora. It is open source, works transparently on any platform that includes Java support, and does not require encoding multiple source files — so there is no inadvertent spreading of unnecessary H.264 content required. But no one should hold their breath waiting for YouTube to implement it, of course.