Specifying codecs for the web

Audio and video content are increasingly important components of the World Wide Web, which some of us remember, initially, as a text-only experience. Users of free software need not be told that the multimedia aspect of the net can be hard to access without recourse to proprietary tools. So the decisions which are made regarding multimedia support in the next version of the HTML specification are of more than passing interest. A current dispute over the recommended codecs for HTML5 shows just how hard maintaining an interoperable web may be.

In particular, several big players have complained about the inclusion of Ogg Vorbis and Theora into the standard, causing a predictable uproar in the free software community. To many, it looks like a classic free-versus-proprietary standards showdown. In truth, the issue is not clear cut; there are nuances that are difficult to turn into a banner headline. The heart of the problem is patents, but, unexpectedly, it is the Ogg codecs that are claimed to be at risk.

Nokia fired a very public shot at the Ogg family with a position paper [PDF], calling it "proprietary". It is unclear what Nokia hoped to gain with this statement, other than inflaming the community, as Ogg Vorbis and Theora are clearly open codecs, with free reference implementations – just the opposite of proprietary. In addition, unlike most (or all) other codecs, a patent search was done to look for relevant patents for Vorbis and Theora, with the Xiph.Org Foundation claiming that none could be found. Some contend that an exhaustive patent search is essentially impossible, but most codecs (MP3, H.264, etc.) are known to be patent-encumbered, which would seem to make them a poor choice for HTML5.

The draft of an HTML5 specification under construction by the Web Hypertext Application Working Group (WHATWG) contained, up until this week, a recommendation for the Ogg codecs. Ogg was not required, only listed as something that SHOULD (i.e. not MUST) be implemented by conforming browsers. That recommendation was dropped from the draft this week, replaced with the following:

Some of the big browser makers, notably Microsoft and Apple, have said that they will not support Ogg Theora – Vorbis is less of an issue – out of a concern for patents, particularly submarine patents. Ian Hickson, WHATWG spokesperson points to the Eolas and MP3 patent attacks against Microsoft (with damages in excess of a billion dollars) as examples of what the large, deep-pocketed companies are concerned about. If there is a patent covering (or appearing to cover) any of the techniques used in Theora, it is the large companies that are going to be on the hook.

Some in the community believe this move is part of a proprietary lock-in play:

There may be some truth to that, but there are some legitimate problems with Theora as well. The technical complaints tend to compare it to H.264 (the most popular MPEG-4 codec), but that is something of a red herring. Neither the WHATWG, nor the World Wide Web Consortium (W3C) are going to allow a technology known to be licensed only on a royalty basis into HTML5. W3C, which will eventually make the final decision on what goes into HTML5, has a policy of requiring technology to be licensed in a royalty-free (RF) mode before it can be approved for inclusion into a standard.

All members of a particular W3C working group are required to disclose patents they believe to be relevant and to provide them to implementors on an RF basis. There may be relevant patent holders who are not members of the working group, thus not subject to that requirement, but if they have enforced their patent on a particular technology, the W3C will try to find an alternative. There may also be patent trolls waiting for someone with deep pockets to implement something covered by a patent they hold – this is the submarine patent threat.

Apple, Nokia, Microsoft and others have already implemented (and licensed) MPEG-4, so there would be no additional risk to them if that were used as the baseline video codec for the web. Using Theora as an alternative is seen by the larger players as a huge increase in their risk, with no benefit to their customers because there is, for all intents and purposes, no Theora content out there. For free software and smaller companies, the situation is clearly quite different.

The lack of Theora-encoded content is the crux of the matter. There might be lots of whining, but big companies would be forced by their customers to support Theora, patent suit risk or no, if there were interesting content available in only that form. This has led to a call for more Theora content:

The WHATWG folks seem to have the needs of free software firmly in mind; certainly the W3C RF policy makes it abundantly clear that a proprietary solution will not be required, or even recommended, for HTML5. The participants on the mailing list, and Hickson, in particular, have been very patient with the onslaught of flamers screaming about the change. The whole HTML5 effort is centered around interoperability for the web, so any technology that will not be implemented by Microsoft and Apple runs directly counter to that goal. WHATWG seems to be between the proverbial rock and hard place.

Several potential solutions are being considered. Possibilities include leaving a video codec recommendation out of HTML5 – not a particularly interoperable solution – or finding a codec that is old enough that any patents covering it must have expired. Another alternative would be to get some other current codec (MPEG-4 for instance) licensed on an RF basis. This issue will undoubtedly be discussed at the W3C Video on the Web Workshop currently being held in San Jose and Brussels. Stay tuned.

How come that codec quality is never discussed when Theora comes up in discussions?  That must
be because Theora is outclassed by a long shot when compared to more modern codecs like MPEG-4
ASP or MPEG-4 AVC (H.264)...

Theora is technically inferior, period.  Centering discussions on patent issues is not going
to get us anywhere.  Any alternatives we wish to pursue must deliver comparable quality.

You can choose between Theora and something else only when there is 'something else'. The
current situation looks like there is pretty much nothing to choose from, except for Theora,
considering royalty-free licensing requirements.

While I haven't looked at H.264 specifically many of the other MPEG pool patents have been
granted in Europe. 

Try getting the companies arguing against Theora to say that the encumbered formats are not
encumbered in Europe. Haha.

While software patents may not be directly enforceable in Europe, devices (including software)
implementing certain techniques can probably infringe.  Though this is outside of my scope of
expertise so I can't say much on that point. Whatever benefits the European system currently
has can not be guaranteed to last.

Beyond that, infringing US patents is unwise even if they can't be enforced in Europe... Since
a judgment against you in the US would be a problem if you ever need to do business in the US.
Perhaps this may not matter to a person, but single people aren't currently the subject of
patent litigation. Companies are, and this matters a lot.

This codec licensing stuff can largely be seen as a fight where there are a few LARGE and
deeply invested companies against a huge number of smaller companies.  The public suffers as a
result but they are just collateral damage.

This is nothing to do with what formats *can* be supported; any user is free to add extra
codecs to their browser. This is about what codecs will ship with browsers, and therefore what
codecs publishers can assume are present.

The question that needs to be asked is not whether Theora is better than H264, but is it
better than nothing? And the answer is clearly yes: it is quite a reasonable default to
provide to users. Those who pay for bandwidth, or have limited bandwidth, and can legally use
an H264 implementation can install one later if they wish.

Flash video sucks, but it is very widely used, just because it is convenient.  And because it
is widely used, publishers offer their video in that format. If publishers knew that Theora
would be available in their users browser, they would provide that format.

Nokia was actually pushing H264/AAC. There seems to be a demand, so some form of supply should
better be decided upon, rather than leaving every player on the market to push their own
mutually incompatible de-facto standards. WHATWG tries to help everyone by finding some common
denominators here, but that proves not to be an easy task.

Concerning the opinion of yours on the matter, why not settle with some 'inferior' codec until
there is a better alternative?

Correct, Nokia was pushing for a high-quality video codec and when offered crap as alternative
they went for the "nothing at all" option instead.  When you say "some form of supply should
better be decided upon" you assume that any supplied solution will be good enough and
acceptable to all players.  This is not the case.

My opinion on the matter is completely irrelevant.  Settling on an inferior codec just will
not work, no matter how much wishful thinking we apply.  Better alternatives without patent
encumbrances will not fall from the sky.  So it is going to be something patent-encumbered or
nothing at all.  It may be a sad fact, but nonetheless it is a fact.

To put things in slightly more graphic terms: When you are looking for an automobile and
somebody offers you a tricycle instead, will you settle for it until there is a better
alternative or will you keep demanding a car and keep looking for car providers elsewhere?

I still see "crap" coming only from the horse's ass, not the mouth it referenced.

Anyway, suit yourself, but flaming on about crap is rather trollish of you. Nobody is or would
have been stopping you or anyone from using any codec you like regardless of the baseline
recommendation whose sole purpose would be to ensure achievable compatibility across all
browsers from everyone without active threat from outside.

Well, actually, somebody is stopping you. Not us or the W3C, though, but rather the MPEG LA
gang of Nokia and friends. If you're worth their time.

Of course, some of us are hardly worth anyone's time anymore. How lucky.

<xiphmont_> Vorbis still stands up nicely.  Theora, OTOH, is a a bit embarrassing.
<xiphmont_> rather, it's a bit embarrassing until you look at the code, then it's alot embarrassing.
<xiphmont_> and that's 70% 'really fucking stupid encoder, really On2, be ashamed' and 40% 'format design flaws'.  It's so bad it adds up to 110%.
<xiphmont_> I plan to help Theora limp along not too embarrassingly until it can be replaced for real-- possibly 2-4 years.
<xiphmont_> Theora is actually fixable tho.  The amount of low-hanging fruit is staggering.
<xiphmont_> I mean, an entropy backend that results in *more* bits being written than went in?  It's just... wow.

the horse itself - displaying considerable wisdom - is *not* sugarcoating anything. rather,
technical issues are being discussed frankly, shortcomings included.

for those who are used to marketing blabber and will take any internal criticism of a *work in
progress* as a sign of total obsolescence, the horse kindly spells it out for you in the
preamble:

>  "This document does *not* say [...] Theora is doomed or hopelessly obsolete. It says the
current encoder is lacking compared to the very very best."

>  "Don't forget kids, this isn't a fight about *technology*. It's a fight about *control*."

nobody is doubting that your favourite codec has a bigger dick than mine. that is not the
point.

agreed, and my apologies. it's just that i've been working with theora pretty much since first
alpha (because it's the only available codec that allows free video streams worldwide), and i
know many people who put great efforts into it, creating a free platform for those who care
about such things. the derogatory nature (and lack of clue) of some comments here hit a
nerve... 
taking your advice, i shall withdraw from the discussion now.

Sorry for getting a bit carried away and thanks for pulling us all back in line.  However, I
firmly believe that there is a point to be made here.  Unfortunately the people disliking that
point have started to kill the messenger...

I shall try to make all further comments as respectful as possible.

Theora is not a tricycle. RLE compression is :)

In my current project, I use IMA ADPCM to compress sounds. The sound quality is not the best,
the compression ratio is not the best either, but it's very fast, and that's the absolute
requirement there. Before it was Vorbis, and I had to throw it away, no matter how much I
liked the compression quality. Same here.

Another example is bzip/bzip2. The author had to throw the superior arithmetic coding used in
bzip away and replace it with Huffman for bzip2 because of the patents.

You talk about royalties and how MPEG is so expensive, but did you
really have a look at the license terms? The h.264 are the cheapest
license terms for such a huge patent portfolio i've ever seen.
First 100'000 units per year are for free, and those over 100'000 are
a mere 0.2 USD per unit. I don't know how you calculate production costs,
but even administrative costs are higher than that. And please note that
only _paid_ products are included, anything that is given away for free
(like OSS) is royality free.

So, over all, h.264 has the same royalties like theora for OSS,
with the simple exception that you really know that you don't have
to pay for h.264 while it is always possible for someone to pop out
of nowhere demanding royalties for a patent violated by theora.
For commercial products the fees are negligible for h.264 and knowing
exactly what you have to pay makes it a much safer choice than theora.

Summa summarum, i have to say that the patent situation favors h.264 rather
than theora.

Theora is indeed not the most advanced video codec there is. However, if you had actually read
the article or understood the issues, these patent-encumbered codecs are not simply even in
the running for a W3C recommendation. Therefore technical comparisons to them are, indeed,
quite irrelevant.

In fact, barring a miracle and the MPEG LA granting a royalty-free license to some MPEG
profile, the base codec recommended by HTML5 is likely to be an extremely old and crappy one
whose possible patents have expired already. That is if there is going to be a recommended
base codec; a consensus is not overly likely, what with MPEG LA protection racket members
having a strong say in the matter.

> What's worse, there are no hard facts that prove Theora to not infringe any patents. It's
just a popular belief in certain circles.

as a certain member of such circles let me tell you in no uncertain terms that this remark is
not particularly brilliant.
some facts to clear the FUD:

* theora is based on a patented codec (VP3 by On2) donated to the open-source community. a
license for unlimited use has been granted, and a lot of legal babble ensures that this stays
so.
* therefore it has been scrutinized by patent law to some extent and found sufficiently
original that a patent was granted.
* there can never be absolute proof that something is entirely unencumbered by patents. this
is not an argument against theora, but against software patents.

http://theora.org/faq/#24

i think nobody who has worked with theora will dispute that there are better codecs out there.
*but*: in terms of interoperability and accessibility (for people/organisations on very tight
budgets), there is no alternative to theora today.
its toolchain needs more polishing, but that is likely going to happen within weeks of its
adoption for a major standard such as html5.

with broadband becoming a commodity in many countries, the trade-off between freedom/cost and
compression efficiency is not an issue any more. and in the developing world, where bandwidth
is scarce, the money to license proprietary codecs is in even shorter supply.

that said, the an implementation formerly bbc-funded dirac codec has recently gone beta, so
there is hope for a free alternative down the road. it should be able to compete with
proprietary state-of-the-art codecs in terms of compression vs. quality.

thanks for letting me know that US != world. being a german citizen, that is quite a
reassuring fact. :-D
bickering aside, you have misread the web page you were linking to. the implementation itself
is free (GNU GPL), but there are patents involved:
http://en.wikipedia.org/wiki/H264#Patent_licensing
which is an issue in the US, specifically, because of their software patent laws.
i guess you will find that your money is involved either way... 
but i'd rather consumers had to pay a little more to access content, than see independent
content creators threatened by bullying corporations or lobby organizations over patent
issues.

Yes, and what you need to remember is that Europe != world. Many of the major developers in 
question happen to be located in the US (Apple, Mozilla, Microsoft, etc), and so they are
liable if 
they infringe on people's patents. The W3C is also in the US, and has a very strong policy
against 
standardizing on anything that requires royalties or has patents that might require royalties.

Furthermore, there has been a lot of pressure to allow software patents in the EU, so it's
possible 
that at any time, a legislative decision will mean that even in Europe, x264 will be illegal. 

A central concept of the Internet is that content is independent of the location of the end
user (as a general rule).  What you are suggesting is to have the video codec dependent on the
location of the viewer.  That is setting a very bad precedent.

I think people need to sit up and take a good long hard look at the global patent system.  It
is in a mess.  It is too easy to patent processes that didn't require any significant R&D to
develop in the first place.  The patent system is expensive and so naturally favours large
corporations over small corporations or individuals.  There are so many patents that it is now
very difficult to be sure that no patent infringement is occuring.  I believe the patent
system is near collapse.  Throught these discussions it is important to keep in mind that the
patent system is not an immovable rock, which is how  many seem to view it.  A lot of "patent
reform" has occured in recent decades .

As mp3 proved, having a widely available codec is effectively game over for competitors even
if they are technically superior (and in this particular case only theora is a bit weak
technically).

So the original wording had the potential to vaporise the huge investment companies like Apple
and Nokia made in software or hardware implementations of proprietary codecs. It does not
matter if those codecs are better technically (and for sound they are not). Making ogg
vorbis/theora a public default (even if it's optional) would marginalize them at once. All the
carefully built closed gardens and associated revenue streams would suddenly lose their
consumer appeal.

Companies like Apple and Nokia bet on codecs that could only win over similarly legaly
encumbered alternatives, now that WHATWG threatens to call the race end with Ogg Theora/Vorbis
in they are frantically trying to preserve the status-quo.

Submarine patents are a problem but they affect every codec and forestalling standardization
because of them is only good for entities with no interest in standards.

I compare codecs to codecs, you compare codecs to hardware.

If you want to go into hardware-land look how bad clearly technically superior formats like
blue-ray and hd-dvd fare against simple encumbrance-free DVD.

On the hardware-side for all practical matters it is. I agree it's not a perfect analogy.

VHS vs Beta would be pretty close

You have pushed x264 all over the comments on this page--but you have not explained how to get
around the simple fact that no one in the United States can use x264 without leaving
themselves liable to patent infringement lawsuits. Like it or not, the fact that software is
patentable in the US does matter--you can't just bury your head in the sand!

How can you be sure you have payed all the existing patent owners for the specific
x264-related patents?

Microsoft has learned it the hard way recently with MP3.

So that risk still applies for all others. At least for Theora there are no known patents.

Microsoft surely can't. At least not if they participate in the standartization of HTML5.

Video cards?

File formats exhibit considerable network effect:  I install XYZ tools to read the XYZ files
you produce, causing me to produce XYZ files which ...

The same isn't true for Video cards.

It's also the case that unencumbered software is ZERO per unit cost software. The same would
not be true for 'free' video cards. 

The dynamic is just different.

You are right that the level of inferiority is material. But even today Theora is simply 'not
quite as good', rather than 'so much worse that it would be unsuitable for the application',
like MJPEG or H.120 would be for web video.

Theora is good enough today that, if widely adopted, it would put price pressure on vendors of
H.264.  The payoff math is particularly simple for web use: Both Theora and H.264 can fit
acceptable quality streaming video into typical broadband connections. If compatibility is
equal, H.264 must be enough better to pay its way in reduced bandwidth consumption. Between
decreasing bandwidth costs and improvements to Theora, this will become increasingly difficult
over time.  ...

Frankly, the above seems like a much simpler an obvious explanation from companies with
patents in the MPEG patent pool's than the handwaving argument that they would suffer a
material increase in patent liability because of an optional recommendation by the W3C.

The video card network effect is seen most in games.  The current Nvidia dominance of the
performance video card has resulting in games optimized to Nvidia cards.  This translates to
review benchmarks showing Nvidia running games faster than ATI/AMD.  More gamers buy the
Nvidia card, leading to game writers optimizing for it ...

It also applies to game consoles.  When one console offers a superior game experience, the
game authors target it, the game buyers buy more of them, etc.

I'm disappointed to see LWN go to press on this without getting Xiph's position on the
subject.

I think the content available question is itself a red-herring:

While it certainly is true that Theora is not as widely adopted as some contemporary
encumbered formats, it's dishonest to claim that it isn't used at all. For example, Ogg/Theora
is the exclusive video format supported on Wikipedia, one of the top ten most visited websites
in the world.

Ultimately it takes much less effort to OFFER content in a new format then it does to upgrade
clients to support that format. Many content providers are simply waiting for the client
support to exist, once the client support reaches an acceptable level they will be able to
easily migrate thus avoiding the MPEG-LA per-user webcasting fees which begin in 2009.

On the quality front, the Theora codec offers quality/bitrate performance orders of magnitude
better than other video formats, such as the H.120 standard from the early 80s or MJPEG, which
are believe to equally free of patent risk.  The difference in performance of megabits per
second vs hundreds of kilobits per second is utterly critical for the success of web based
video. 

The currently available Theora code achieves a quality/bitrate ratio somewhat worse than H.264
(MPEG-4 part 10/AVC), an expensive, heavily encumbered, and computationally costly codec. Xiph
believes that with further enhancements to the codecs Theora can achieve and maintain
generally competitive performance with H.264. (Critical commentary on Theora from its
maintaining engineers: http://web.mit.edu/xiphmont/Public/theora/demo.html)

I'm not sure where the Microsoft comment came from as MSFT is not a member of WHATWG or the
W3C, and they already ship Xiph codecs in various products.

Finally, I think it's a little unfair to fail to mention that the vocally opposed parties all
profit from the patent licensing for the encumbered formats that they prefer.

Ah. My error. I asked someone involved with the W3C and I must have misunderstood his answer.

While I made an error in that detail the important message I was trying to convey, that MSFT
has not publicly protested the inclusion of the Ogg codecs as baselines for HTML5 and that the
article was incorrect to claim otherwise, still stands.

Microsoft already ships Ogg container codecs in a number of products and has for a long time.
I think they are too wrapped up with the grim realities of using known patented codecs right
now to worry about Theora.

Ultimately this is an area where the free world and Microsoft have interests in common, and
this time Microsoft might actually realize it, if not quite well enough to actually give their
public support.

Huh, really? Whatever happened to WMV?

I have not heard about MS using Theora anywhere; however I believe that DirectPlay uses Vorbis
and Speex for voice communications.

some more clarification can be found in conrad parker's blog (a developer working on
ogg/annodex and related media formats):
http://blog.kfish.org/2007/12/html5-for-free-media-today-...

looks like this issue is not as bad as it looks...

Perhaps this kind of situation could be resolved by inventing a legal way of flushing out
submarine patents.  Someone wanting to develop or standardize something would make a
declaration to the relevant patent offices, and there would then be a period during which
possible patent holders would have to declare themselves, or else hold their peace forever
after.

What objections to this are there?

Those with submarine patents will surely object.

True, but that doesn't count, as they have no justification - either moral, or in terms of the
usual reasoning for the existence of patents - for that "business" model.

This could easily backfire.  Big companies that wanted to avoid a small guy's patent would put
out a continuous stream of declarations to keep him so busy that he doesn't notice the one
that actually applies to his patent.

Good point; that's a more tricky one.  I'll report back if I can think of a solution.

This article is so even-handed!  I miss the Corbet-snark.

Ogg vorbis is much better position than theora. vorbis has accepable quality and some nice
features for web, such as peeling (dropping bitrate without recompressing), has seen patent
review and is in widespread use by community and industry players like samsung and software
houses wanting to avoid mp3 tax. The main disadvantage is that vorbis isn't really
dsp-friendly, which makes it hard to implement bugfree in low-powered portable players. This
is not a problem for the html5 video tag really.

Ogg theora on the other hand is technically inferior, and rarely used. Even the <a
href="http://www.effi.org/tapahtumat/bb-seminaari-07-videot.html">EFF finland site</a>
provides videos only in H.264 and mpeg2 format! I don't think the community is very honest
when demanding *theora* support from W3C when they are not ready it eat their own dogfood.

An interesting review would be the visual quality of H.261 (which is supposedly old enough
that all patents have expired) vs Ogg theora. With selected improvements, such as allowing
more video sizes, H.261 could become an acceptable baseline.

Oh, also I'll note that Fluendo/Cortado streaming has seen mainstream use here in Finland as
well at least in sports event streaming - perhaps they were less confident than EFFI in
asserting that software patents are illegal here. (Don't know which event(s) or how many, as
I'm not interested in sports - merely noted their use of free technology). As for more global
use, at least Wikipedia policy is to use spesifically Theora for video. They're only one of
the ten or so most popular web sites around, and while there's not overmuch video content at
this point, the amount is only going to increase.

Theora may not be as widely used as some of the encumbered options, but that doesn't mean it's
doesn't get around already.

But none of the major vendors are likely to obtain HTML5 compliance, unless its by watering
down the standard to contain mostly SHOULDs that they can decide not to implement.

HTML is a very qualified success. Tag soup has won out for the most part. Things have improved
in the last ten years, but not very quickly. Can you think of a web browser people use that's
HTML 4 compliant? Not "nearly" or "sort-of" or "the important bits" but the HTML 4 standard as
published ?

Last time I looked Mozilla.org still had bugs open against rarely used HTML features dating
back to the start of the project in the 1990s.

I'm not completely sure I understand how this happened, but one thing that definitely had an
effect was the encouragement to "edit in Notepad" without tools that could verify the
resulting document meant anything, let alone what was intended by the author. Modern tools
have much reduced this tendency (most people now create content in Wikis or blogs where they
can type an ampersand without making the site fail validation).

Assuming Theora gets a bit of momentum (such as, being approved).. I bet everyone will
suddenly jump on the wagon saying they have got a competetive advantage (because the
competitors do not (yet)) -- *even though* having complained beforehand that it will hurt
competition.

I'm not sure if you could call separate compilation of templates a `small 
corner of the language', really. It's just so hard to implement that 
everyone has got used to doing without it, and there *are* ways to get 
along without it.