Theora does not produce quality comparable to H.264 [LWN.net]

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 18:40 UTC (Tue) by patrick_g (subscriber, #44470) [Link]

From the article :

"This is a test on anime, nothing else; it inherently biases towards video formats with smaller transform size (like H.264), so the results are not generalizable to non-animated material"

I must add that I never said that Theora produces quality better than H.264. Your comparison with evolution vs creationists is insulting. I know very well that H.264 is better than Theora...even the author of my link know that :

"Theora isn't the most efficient video codec available right now.But it is by no means bad, and it is substantially better than many other widely used options. By conventional criteria Theora is competitive. It also has the substantial advantage of being unencumbered, reasonable in computational complexity, and entirely open source".

The conclusion is that Theora is "good enough".

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 18:52 UTC (Tue) by DonDiego (subscriber, #24141) [Link]

No. The comparison is flawed and thus must not be employed to draw any sort of conclusion. Any conclusion drawn from flawed assumptions is flawed itself.

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 21:12 UTC (Tue) by Trelane (subscriber, #56877) [Link]

Maybe you can start by telling us how it's flawed?

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 21:47 UTC (Tue) by DonDiego (subscriber, #24141) [Link]

First off, video and audio are combined. This skews any sort of sensible comparison because Theora receives higher bitrate than H.264. Also, at least at that moment YouTube encoded just H.264 baseline profile while the Theora encoding was tweaked to be optimal. Basically H.264 has all the odds stacked against it. Still the resulting quality is better and the file smaller...

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 22:11 UTC (Tue) by Trelane (subscriber, #56877) [Link]

As discussed in the source, video and audio are in practice seldom divorced. It was intended as a real-world test, and real-world it was. As mentioned, the sound quality in the Ogg version was better than in the YouTube version. You're good with lower sound quality in exchange no/little change in video performance? (Usually, audio is the thing people are more sensitive to!)

"YouTube encoded just H.264 baseline profile while the Theora encoding was tweaked to be optimal."

Yes, it was in answer to "YouTube can't use Theora." It naturally used YouTube to compare against.

"Still the resulting quality is better and the file smaller..."

Actually, the Ogg bitrates were slightly lower (486 vs 490kbps, 325 vs 327kbps).

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 22:38 UTC (Tue) by DonDiego (subscriber, #24141) [Link]

This test gets referenced all the time in an attempt to prove that Theora can compete with H.264 in raw quality or quality per bit. But no matter how often this falsehood gets repeated, it remains untrue.

The Vorbis audio does not sound better to my ears, on the contrary. It would also be a very big surprise if 64kbit Vorbis could beat 106.5kbit AAC.

So if I apply your argument that audio is more important, H.264 wins two times, because it allows dedicating more bits to the audio while preserving a slightly higher video quality.

> Actually, the Ogg bitrates were slightly lower (486 vs 490kbps,
> 325 vs 327kbps).

False:

-rw-rw-r-- 1 diego staff 13871183 Jan 26 22:41 bbb.h264
-rw-rw-r-- 1 diego staff 15009926 Jan 26 22:41 bbb.theora

The clip is 284 seconds long, so the bitrate is 423kbps for Theora and 390kbps for H.264.

This test also says nothing about the suitability of Theora for YouTube. Judging from this sample, YouTube still has ample room for improvement in the H.264 encoding pipeline. You can rest assured that they know about this and will eventually tweak their system to produce better results.

Theora does not produce quality comparable to H.264

Posted Jan 26, 2010 22:50 UTC (Tue) by Trelane (subscriber, #56877) [Link]

"This test gets referenced all the time in an attempt to prove that Theora can compete with H.264 in raw quality or quality per bit. But no matter how often this falsehood gets repeated, it remains untrue."

So far, that's all tangential accusation and precious little fact.

"The Vorbis audio does not sound better to my ears, on the contrary. It would also be a very big surprise if 64kbit Vorbis could beat 106.5kbit AAC."

Umm, ok. Didn't know your ears were the benchmark here.

"So if I apply your argument that audio is more important, H.264 wins two times, because it allows dedicating more bits to the audio while preserving a slightly higher video quality."

Except that you're wrong?

"False:

-rw-rw-r-- 1 diego staff 13871183 Jan 26 22:41 bbb.h264
-rw-rw-r-- 1 diego staff 15009926 Jan 26 22:41 bbb.theora"

du -bs bbb_*
17307153 bbb_theora_486kbit.ogv
17753616 bbb_youtube_h264_499kbit.mp4

dir bbb_*
-rw-r--r-- 1 auser agroup 17307153 2010-01-26 16:40 bbb_theora_486kbit.ogv
-rw-r--r-- 1 auser agroup 17753616 2010-01-26 16:40 bbb_youtube_h264_499kbit.mp4

Sure you downloaded the right files? Oddly, your file names aren't the ones from the site. (http://people.xiph.org/~greg/video/ytcompare/bbb_theora_4... and http://myrandomnode.dyndns.org:8080/~gmaxwell/ytcompare/b...) What's going on here?

"The clip is 284 seconds long, so the bitrate is 423kbps for Theora and 390kbps for H.264."

The bitrate was included in the captions. Bitrate isn't filesize divided by time either.

"This test also says nothing about the suitability of Theora for YouTube. Judging from this sample, YouTube still has ample room for improvement in the H.264 encoding pipeline. You can rest assured that they know about this and will eventually tweak their system to produce better results."

Except that YouTube is shipping with it now, and the results are at worst comparable, one codec without costing anyone a patent penny. So yes, YouTube can use Theora now.

In addition, if we count future tweaking in, Theora doens't lose again. From your link at http://www-cs-faculty.stanford.edu/~nick/theora-soccer/:
"With this 1.1alpha version, Theora requires somewhere around 60% more bandwidth than Mpeg-4, but Theora is being actively improved, so we'll see. The Mpeg-4 we have today is the result of years of tuning on the encoders."

So basically, it's comparable to what's currently in use, while at the same time being royalty-free. I don't see a loss there, friend.

Theora does not produce quality comparable to H.264

Posted Jan 27, 2010 0:25 UTC (Wed) by DonDiego (subscriber, #24141) [Link]

> The Vorbis audio does not sound better to my ears, on the contrary.
> It would also be a very big surprise if 64kbit Vorbis could beat
> 106.5kbit AAC."

> Umm, ok. Didn't know your ears were the benchmark here.

The AAC stream captures the water sound better than the Vorbis stream to my ears. Anyway, bit by bit AAC and Vorbis are roughly comparable codecs. But you seem to expect that Vorbis can beat AAC with 60% of the bitrate. This is not at all realistic, Vorbis is an excellent audio codec, but not fairy dust.

> Sure you downloaded the right files? Oddly, your file names aren't
> the ones from the site. What's going on here?

I extracted the raw video streams from the containers in order to be able to compare video sizes directly.

> > The clip is 284 seconds long, so the bitrate is 423kbps for Theora
> > and 390kbps for H.264.

> The bitrate was included in the captions.

The bitrate in the captions refers to the complete file together with audio and container overhead. That number is thus not suitable for video bitrate comparison.

> Bitrate isn't filesize divided by time either.

What we care about here is the total/average bitrate for the whole clip, i.e. file size divided by time.

> In addition, if we count future tweaking in, Theora doens't lose again.
> From your link at http://www-cs-faculty.stanford.edu/~nick/theora-soccer/:
> With this 1.1alpha version, Theora requires somewhere around 60% more
> bandwidth than Mpeg-4, but Theora is being actively improved, so we'll
> see. The Mpeg-4 we have today is the result of years of tuning on the
> encoders."

The Theora we have today is also the result of years of tuning on the encoders. The question is how far tweaking can get it. There is a limit to the quality you can coax out of the Theora bitstream format. The 60% bitrate difference quoted above is a *lot* of ground to cover.

Note that H.264 decoders are also being improved. Much more work is being dedicated to H.264 encoders than to Theora encoders. If you don't believe me, compare the x264 development activity to libtheora and keep in mind that libtheora is the only Theora encoder, but x264 is not the only H.264 encoder.

This is not a good vs. evil fight. It's a technical question. Compression formats are not created equal. Theora technology is two generations behind H.264. You cannot expect Theora to improve beyond H.264, much less since all patented techniques are out of bounds. I'm sorry that it's not a fair race, but facts remain facts.

Theora does not produce quality comparable to H.264

Posted Jan 27, 2010 3:48 UTC (Wed) by paulj (subscriber, #341) [Link]

Note that H.264 decoders are also being improved. Much more work is being dedicated to H.264

E.g. the BBC apparently have been able to reduce the bitrate of their High-Def H.264 broadcasts by 16% without loss of quality, thanks to newer, better encoding technology.

You've missed the point.

Posted Feb 3, 2010 21:56 UTC (Wed) by gmaxwell (subscriber, #30048) [Link]

I spent time in the article explaining the role of audio. I'd appreciate if you wouldn't make it out as though I were attempting to mislead anyone.

I'm also completely baffled regarding the mention of "raw quality" in your comment. Are you really attempting to say that there is some issue with raw quality? If you crank the bitrate sufficiently pretty much any codec should be transparent.

...and of course the comparison is saying something about the suitability of Theora for Youtube, the fact that they could possibly improve their H.264 processing chain doesn't change the fact that Theora was working OKAY compared to what they actually were using. If you're nitpicking about a couple of kbit/sec here or there, you're missing the point. It's intended to be an order of magnitude comparison to give some perspective. When people here codec ricers blather on with hyperbolic language about which codec is superior, they tend to think of enormous differences. But the differences are usually not enormous in a comparison unless you run it right up against the quality/bitrate knee.

Personally, I think it is insane to compare the files without container overhead. What else do you care about when transferring a file than the whole system? If one container has higher overhead than another, then thats a cost you're going to have to deal with in the real world.

I did also put up a strict buffer constrained rate controlled and lower bitrate file for people who emailed asking about that but since the screenshot looked the close enough to me I didn't think it was worth adding links to it and trying to explain the benefits that strict rate control has for streaming all post-publication.

You comment that "The Theora we have today is also the result of years of tuning on the encoders." is laughably, but sadly, wrong. The Theora encoder has had no more than one person working on it part-time at a time for the past two years, and had a number of years gulf of no active development. Go look at the SVN history. There simply haven't been the resources available to go work on it, and the overwhelming majority of the people working on "open source" video encoding have been contributing to the patent encumbered formats... they are more exciting because they are ahead and avoiding patents is nasty boring work.

With a bit of available funding and attention it made enormous progress in a short span of time and the ptalarbvorm development branch is already obviously improved vs 1.1 (about 2dB on SSIM on average, and casual subjective testing seems to suggest that much or somewhat more). There will be no miracles: After all, from a format perspective H.264 is nearly a superset of Theora, but there is clearly a decent amount of room for improvement.

No, I made the point I wanted to make.

Posted Feb 4, 2010 15:36 UTC (Thu) by DonDiego (subscriber, #24141) [Link]

> I spent time in the article explaining the role of audio. I'd appreciate if you wouldn't make it out as though I were attempting to mislead anyone.

Correct. You were not trying to mislead anyone. However, people are misreading your comparison and drawing the wrong conclusions.

> I'm also completely baffled regarding the mention of "raw quality" in your comment. Are you really attempting to say that there is some issue with raw quality? If you crank the bitrate sufficiently pretty much any codec should be transparent.

I spoke imprecisely. I meant bit-by-bit quality.

> ...and of course the comparison is saying something about the suitability of Theora for Youtube, the fact that they could possibly improve their H.264 processing chain doesn't change the fact that Theora was working OKAY compared to what they actually were using.

This assumes that YouTube is not planning ahead for a future in which they may wish to stream 1080p video to desktops worldwide. Do you think this is a realistic assumption?

> If you're nitpicking about a couple of kbit/sec here or there, you're missing the point. It's intended to be an order of magnitude comparison to give some perspective.

I'm not nitpicking, I'm putting your comparison in perspective. You sacrifice audio quality to achieve a not-quite-on-par video quality.

If you can live with that, fine. But your comparison is being touted as proof that Theora produces bit-by-bit quality comparable to H.264. As you very well know, this is simply not true.

> Personally, I think it is insane to compare the files without container overhead. What else do you care about when transferring a file than the whole system? If one container has higher overhead than another, then thats a cost you're going to have to deal with in the real world.

I agree, but the container overhead is higher for Ogg, so that's another drawback for Theora. What are you trying to say?

I know that Theora encoders still have room for improvement. I never disputed that fact. However, the same applies to H.264 encoders. Due to the technical superiority of the format and the different amount of work being dedicated to both, the gap is more likely to widen further than it is to shrink.

Also, I take issue with your putting open source video encoding in quotes. These are the people that are keeping Linux viable as a general-purpose desktop platform. They deserve thanks, not criticism.

No, I made the point I wanted to make.

Posted Feb 4, 2010 16:34 UTC (Thu) by gmaxwell (subscriber, #30048) [Link]

I agree, but the container overhead is higher for Ogg, so that's another drawback for Theora. What are you trying to say?

I'm not in much of a position to speak about MP4/QT container overhead, but the container overhead of Ogg is fairly low when Ogg is used in the typical way. Ffmpeg does a rather brain-dead thing and only ever places one packet per Ogg page which enormously increases the overhead at lower bitrates, but as far as I'm aware every other tool won't do that. If you are interested in low overhead, Ogg can be as low as ~0.433% (presuming big packets, like video). Is a streamable, seekable, MP4 file in that ballpark? Better seems ... unlikely... to me, or inconsequential if it's the case.

I'm putting your comparison in perspective. You sacrifice audio quality to achieve a not-quite-on-par video quality.

If you want to argue it out, I can arrange a blind listening test on the audio. My audio will win. :) AAC-LC isn't all that hot. Really.

But I don't we need to argue that take a gander at This Theora vs the YT H.264, or for the lazy the Theora still vs the YT h.264 still.

So there you have, lower bitrate pure video than the H.264, same keyframe interval, same input, etc. How do you think it looks?

Would sir like his hat medium or rare?

Obviously we don't know anything about the YT cpu consumption, so you can still argue that. The theora encoder is pretty fast, I've seen others compare it to the fast mode in x264, even though its not deeply optimized at all. But we have no idea how fast YT's tools are. I concede the encoder cpu consumption is usually relevant, and that other H264 encoders are better, ... and I never claimed otherwise. Can you concede that, for the purpose of this test, that the quality/bitrate is basically there?

Also, I take issue with your putting open source video encoding in quotes. These are the people that are keeping Linux viable as a general-purpose desktop platform. They deserve thanks, not criticism.

It's in quotes because people are widely ignorant about the implications, it isn't intended as an insult. Open source in the context of the tools for encumbered media formats doesn't quite mean the same thing as it does for most other pieces of open source software.

Obviously I'm mostly discussing high end encoders when I made that point.

There is a fine line between embracing the encumbered world enough to capture their users and outright embracing it. When the resulting tools are widening the gap between unencumbered and encumbered formats, its doing the format owners a much bigger favour than its doing everyone else. ::shrugs::

It's also a little frustrating that some of the popular tools like ffmpeg don't offer world-class support for the things like Ogg/Theora while a few of their developers are spending a lot of time and effort slinging unadulterated FUD. Can you sense my finger wagging? Come on, at least look at the freeking priority dates, dude.

No, I made the point I wanted to make.

Posted Feb 4, 2010 16:51 UTC (Thu) by gmaxwell (subscriber, #30048) [Link]

Erp. And it seems that VLC, of at least some versions, is performing a bizarre post-decode blurring on Theora and not H.264. Doesn't appear to be the libtheora post-processing filter... Only discovered this today, but you ought not view in VLC if you're comparing the videos.

Cheers.

No, I made the point I wanted to make.

Posted Feb 5, 2010 0:53 UTC (Fri) by DonDiego (subscriber, #24141) [Link]

I'm afraid your intuitions about container overhead are quite far off from reality. Let's see:

17307153 bbb_theora_486kbit.ogv
15009926 bbb_theora_486kbit.theora
 2107404 bbb_theora_486kbit.vorbis
--------
  189823

17753616 bbb_youtube_h264_499kbit.mp4
13898515 bbb_youtube_h264_499kbit.h264
 3796188 bbb_youtube_h264_499kbit.aac
--------
   58913

Ogg has more than 300% the overhead of MP4 here. You wanted to talk about the Youtube case, so there is no escaping that number here.

I made a listening test of both audio streams myself. To my ears the waterfall in the beginning really sounds like water for the AAC stream, the Vorbis stream is much more washed out.

I'm happy to see your latest encoder branch make further progress. I'll be even more happy to see it perform in a straight comparison with x264 and various FFmpeg encoders. If you want to have credibility, perform a fair test.

Can you concede that, for the purpose of this test, that the quality/bitrate is basically there?

No. My whole point is that this test is worthless because it avoids the most sticky points. You should pitch Theora against the best H.264 encoders, not against an encoding pipeline which obviously has huge room for improvement. You should also take future developments into account. 1080p should be the benchmark, not the horrible quality Youtube currently offers.

FFmpeg's Ogg muxer produces high overhead because Ogg is a deeply flawed container format and so far nobody has been willing to go out of their way to work around its deficiencies. We continue to have the fastest Vorbis decoder around and our Theora decoder has seen considerable speeups recently. David Conrad has a branch with some serious tweaks that is on its way to getting merged. It should at least match libtheora 1.1 in speed and eventually surpass it when further tweaks get applied. We clearly have no world-class support yet, but it's slowly improving. Also, we accept patches, yours are welcome as well.

There is the "encumbered" world, the "unencumbered" world and the real world. In the real world people try to listen to music and watch movies. FFmpeg, VLC, MPlayer, x264 and the rest of the bunch enable people to do just that. Without them free software desktop market share would be orders of magnitude smaller and decreasing. You seem to think this is worth sacrificing in order to avoid a patent threat that some day might affect some person in some part of the world. I wholeheartedly disagree.

I'll stop wondering about submarine patents the moment you guys earn my trust by publishing your findings. And yes, I'm wagging my finger at you. You act exactly as if you had something to hide. "There is no toxic waste drop here. We have made a study and come to the conclusion that this water is safe. We cannot show you the study, but you can *trust* us!" Trust is not something that is handed out for free, you have to earn it through integrity and transparency. I just wanted to say something about Iraq and weapons of mass destruction, but I'll shut up before I dig myself in deeper ;-)

No, I made the point I wanted to make.

Posted Feb 5, 2010 14:13 UTC (Fri) by gmaxwell (subscriber, #30048) [Link]

I wasn't giving you an intuition, I was giving you a format maximum: You can put about 64k of payload in an Ogg page or up to 255 packets. For that file that using maximum size pages would give an ogg overhead of 82887 bytes. Had I used larger page sizes in my comparison I would have no longer been using default settings, which was one of my goals. I'm not sure how you're writing out those files for your mp4 comparison, I'm guessing they still have some container headers, but reading the samplesizes directly from the mp4 dump brings me to 86245 bytes of container overhead, which is pretty similar.

not the horrible quality Youtube currently offers

Well, what can I say? I wrote a clearly described specific comparison about YT because of an outrageous goggler quote. Now you're running around the internet calling my statements fraudulent because they weren't a comparison with something else.

Without them free software desktop market share would be orders of magnitude smaller and decreasing.

I fail to see how advancing the state of the art in the best encumbered format encoders does much of anything to preserve free desktops by helping people "listen to music and watch movies". But to each his own. There are perfectly viable ways to get legally licensed encoders for your desktop even if tools like ffmpeg didn't exist, but its existence wasn't something that I was protesting.

I'll stop wondering about submarine patents the moment you guys earn my trust by publishing your findings.

You could at least get the terminology right. A "submarine patent" isn't what you should be worried about, submarine patents are an old technique for making effectively secret patents that could last for decades, scary indeed. This loophole was closed in the mid-90s. All the submarines were forced to surface.

The way the patent system is setup you are very strongly discouraged from disclosing your research: The parties suing for infringement have the stronger position and making them aware of your position allows them to avoid arguing particular things. For example "my own work is prior art for that claim" and "I don't practice that technique" are mutually exclusive defences. If the attacker knows how you will defend they can make the legal battle much longer and more expensive to win. Don't hate the player, hate the game.

The whole purpose of Xiph is to produce/promote unencumbered formats. We expend a fair amount of additional effort, and accept considerable limits because of what techniques we're not permitted to practice. Knowingly allowing something encumbered would be self-defeating in the most enormous way. Of course, mistakes could be made.

What I can give point to you is a decade of shipment, by a large number of parties (some with decently deep pockets) with no publicly known litigation related the formats, which is better than you can say for many mpeg standards. I can also point you to shipments by large organizations who had their own legal teams look at it. I'm sorry if thats not enough for you, but as far as I can tell nothing would be.

What you were doing by slinging a bunch of irrelevant patent numbers was pure FUDing, in a particularly vile way. I think it speaks a lot to your motivation and character, and it's a warning about the futility of engaging with you which I should have heeded.

Cheers.

No, I made the point I wanted to make.

Posted Feb 7, 2010 21:21 UTC (Sun) by DonDiego (subscriber, #24141) [Link]

I used 'mplayer -dumpaudio' and 'mplayer -dumpvideo' to extract the audio and video streams.

> Well, what can I say? I wrote a clearly described specific comparison about YT because of an outrageous goggler quote. Now you're running around the internet calling my statements fraudulent because they weren't a comparison with something else.

Your comparison is constantly being presented to me and others as proof that Theora is as good as H.264. If that was not your intention, fine. It does not change the fact that your comparison is being misread all the time. And sorry, but you are not exactly going out of your way to prevent this.

> > Without them free software desktop market share would be orders of magnitude smaller and decreasing.

> I fail to see how advancing the state of the art in the best encumbered format encoders does much of anything to preserve free desktops by helping people "listen to music and watch movies".

Simple. People use their computers to listen to music and watch movies. There are popular formats to make music and movies available in digital form. If no free software is available to handle these formats, people will use something else to get at their movies and songs. Free software loses, badly.

> But to each his own. There are perfectly viable ways to get legally'licensed encoders for your desktop even if tools like ffmpeg didn't exist, but its existence wasn't something that I was protesting.

Fluendo is much more recent than FFmpeg or MPlayer. But if Fluendo is an alternative option for you, then you don't value free software. I'm not such a person.

Container conundrum

Posted Mar 9, 2010 20:59 UTC (Tue) by gmaxwell (subscriber, #30048) [Link]

So, I'm reasonably confident that your claimed mp4 overhead is simply incorrect. I don't know why.

I measured it as 86245 using one method (segment sizes from mp4dump). A friend who has implemented the mp4 demux checked the file and also calculated 86245 using his own method. My guess was that mplayer must be including packet length data in the dump output, but I just repeated your method and got 13871183 bytes video, 3796188 bytes audio. Resulting in 86245. Exactly the same figure calculated by two other distinct methods.

Perhaps you could explain why I'm getting a different figure using exactly the same method as you? I'm using Mplayer SVN-r29800-4.4.2.

I remuxed the above Theora file we were discussing for minimum overhead using a copy of libogg with the default 4k page size target removed:

http://people.xiph.org/~greg/video/ytcompare/bbb_theora_4...

This file has 84604 bytes of container overhead.

This is somewhat lower overhead than your mp4 file. Now that I understand a bit more about the mp4 format I find it actually somewhat surprising Ogg provides a lot of properties that MP4 does not. Like being incrementally written in a complete form.

I hope this update convinces you to discontinue spreading these mistaken claims.

Cheers.

No, I made the point I wanted to make.

Posted Feb 7, 2010 21:34 UTC (Sun) by DonDiego (subscriber, #24141) [Link]

> > I'll stop wondering about submarine patents the moment you guys earn my trust by publishing your findings.

> You could at least get the terminology right. A "submarine patent" isn't what you should be worried about, submarine patents are an old technique for making effectively secret patents that could last for decades, scary indeed.

I'm not a native speaker and was indeed unfamiliar with the meaning you connect to the term "submarine patent". Note however, that this very same confusion exists between the German and English Wikipedia entries.