GStreamer 1.26.0 released

For prerecorded content, this is plenty good enough - you can cover all the bitrate points you want to cover, and use buffering to overcome variability in consumer connection speeds. But for live content, you start having problems with common connection types like WiFi and 3GPP cellular (LTE, 5G NR etc), because you've got more constraints on you:

Take someone using mobile data to watch live sports; you want to keep the latency from action being filmed to viewer seeing it low (so that they can keep up with a group chat including friends in the stadium), so your buffer size is on the order of half a second - much more than that, and messages from the friends in the stadium will trigger a notification on your phone before you've seen the corresponding action in the stream. For efficiency reasons, you want the stream you're viewing to have "switch points" (where you can switch to a different bitrate seamlessly) as infrequently as possible - Netflix uses every 10 seconds, for example - because that allows you to use inter-frame redundancy efficiently, and new viewers of the stream start at a lower bitrate so that they can predecode and discard up to 9.5 seconds before joining the live stream.

To make this work, you have to choose a bitrate for the base layer such that the lowest throughput over a half second period in any ten seconds is higher than the stream bitrate. But this then gives you a quality problem; if I'm in a train, going through a city, my lowest throughput over a half second period might be 500 kilobits per half second (1 Mb/s), with an average over a ten second period of 4,000 megabits delivered (400 Mb/s). You now have three options to exploit my high throughput mobile data link:

1. Increase latency. This gets you more buffering, and thus increases the averaging time for my throughput from half a second to (say) 10 seconds, and is what Netflix does for prerecorded video. But, for live sports and similar, it's not workable because of the group chat problem.
2. Let it glitch. When my mobile network drops to a slow speed for a short period, I miss the action. Again, not acceptable for live sports, since if I miss the key events completely because the stream glitched for a few seconds, I'm going to be upset.
3. Keep the quality and thus bitrate low. This is the current tradeoff for live sports - if I can only reliably receive 1 Mb/s, that's the stream speed I get, even though most of the time, I can receive 400 times faster. Here, though, you have a PR problem; how do you explain to customers that they're getting poor quality because their network blips occasionally as they're on the move, when every test they know how to run shows that they're getting far higher speeds?

That's where the enhancement layers idea comes from; you need a very different encoder to "normal" video (since the base layer has covered all the low frequency content) to do this efficiently, but you can then send the enhancement layers separately to the base layer, with different congestion behaviours; if the network blips in speed, you set things up so that the base layer gets through (quality reduces, but you can still see the important parts of the action), but the enhancement layers don't. Then, you can use enhancement layers to keep the quality up most of the time, and when there's a blip, the quality falls back to the base layer.

This gets you out of the hole; the enhancement layers can be turned off when they blip, resulting in a quality fallback, but you can now (e.g.) have a base layer at 1 Mb/s, with a 5 Mb/s and a 15 Mb/s enhancement layer. I get 1 Mb/s quality all the time, but whenever my network is good, I get the quality I'd expect from 19 Mb/s of base layer codec; if the network blips, instead of a glitch in the content where I simply miss 1 to 10 seconds of action outright (depending on what gets lost), I go down to 1 Mb/s quality for 1 to 10 seconds instead.

And that's also where I expect to see the patent minefield; the enhancement layers are going to need different codec design to the base layer - they're encoding a residual with very little (or no) low frequency content, they've always got the option of outputting the base layer's decoded output with no changes, they can have extra reference pictures (base layer, and combination of base layer with enhancement layer), and I'm sure there's less obvious things, too.