GStreamer: Past, present, and future

Longtime GStreamer hacker Wim Taymans opened the first-ever GStreamer conference with a look at where the multimedia framework came from, where it stands, and where it will be going in the future. The framework is a bit over 11 years old and Taymans has been working on it for ten of those years, as conference organizer Christian Schaller noted in his introduction. From a simple project that was started by Eric Walthinsen on an airplane flight, GStreamer has grown into a very capable framework that is heading toward its 1.0 release—promised by Taymans by the end of 2011.

Starting off with the "one slide about what GStreamer is", Taymans described the framework as a library for making multimedia applications. The core of the framework, which provides the plugin system for inputs, codecs, network devices, and so on, is the interesting part to him. The actual implementations of the plugins are contained in separate plugin libraries with a core-provided "pipeline that allows you to connect them together".

Some history

When GStreamer was started, the state of Linux multimedia was "very poor". XAnim was the utility for playing multimedia formats on Linux, but it was fairly painful to use. Besides GStreamer, various other multimedia projects (e.g. VLC, Ogle, MPlayer, FFmpeg, etc.) started in the 1999/2000 timeframe, which was something of an indication of where things were. The competitors were well advanced as QuickTime had appeared in 1991 and DirectShow in 1996. Linux was "way behind", Taymans said.

GStreamer's architecture came out of an Oregon Graduate Institute research project with some ideas from DirectShow (but not the bad parts) when the project was started in 1999. Originally, GStreamer was not necessarily targeted at multimedia, he said.

The use cases for GStreamer are quite varied, with music players topping the list. Those were "one of the first things that actually worked" using GStreamer. Now there are also video players (which are moving into web browsers), streaming servers, audio and video editors, and transcoding applications. One of the more recent uses for GStreamer, which was "unpredicted from my point of view", is for voice-over-IP (VoIP) and both the Empathy messaging application and Tandberg video conferencing application are using it.

After the plane flight, Walthinsen released version 0.0.1 in June 1999. By July 2002, 0.4.0 was released with GNOME support, though it was "very rough". In February 2003, 0.6.0 was released as the first version where audio worked well. After a major redesign to support multi-threading, 0.10.0 was released in December 2005. That is still the most recent major version, though there have been 30 minor releases, and 0.10.31 is coming soon. 0.10.x has been working very well, he said, which raises the question about when there will be a 1.0.

To try to get a sense for the size of the community and how it is growing, Taymans collected some statistics. There are more than 30 core developers in the project along with more than 200 contributors for a codebase that is roughly 205K lines of code. He also showed various graphs of the commits per month for the project and pointed a spike around the time of the redesign for 0.10. There was also a trough at the point of the Git conversion. As expected, the trend of the number of commits per month rises over the life of the project.

In order to confirm a suspicion that he had, Taymans made the same graph for just the core, without the plugins, and found that commits per month has trailed off over the last year or so. The project has not been doing much in the way of new things in the core recently and this is reflected in the commit rate. He quoted Andy Wingo as an explanation for that: "We are in 'a state of decadence'".

When looking at a graph in the number of lines of code, you can see different growth rates between the core and plugins as well. The core trend line is a flat, linear growth rate. In contrast, the trend line for the plugins shows exponential growth. This reflects the growing number of plugins, many of which are also adding new features, while the core just gets incremental improvements and features.

The current state

Taymans then spent some time describing the features of GStreamer. It is fully multi-threaded now; that code is stable and works well. The advanced trick mode playback is also a high point, and it allows easy seeking within audio and video streams. The video editing support is coming along, while the RTP and streaming support are "top notch". The plugins are extensive and well-tested because they are out there and being used by lots of people. GStreamer is used by GNOME's Totem video player, which puts it in more hands. "Being in GNOME helps", he said.

The framework has advanced auto-plugging features that allow for dynamic pipeline changes to support a wide variety of application types. It is also very "binding friendly" as it has bindings for most languages that a developer might want to use. Developers will also find that it is "very debuggable".

There are many good points with the 0.10 codebase, and he is very happy with it, which is one of the reasons it has taken so long to get to a 1.0 release. The design of 0.10 was quite extensible, and allowed many more features to be added to it. Structures were padded so that additional elements could be added for new features, without breaking the API or ABI. For example, the state changes and clocks handling code was rewritten during the 0.10 lifetime. The developers were also able to add new features like navigation, quality of service, stepping, and buffering in 0.10.

Another thing that GStreamer did well was to add higher-level objects. GStreamer itself is fairly low-level, but for someone who just wants to play a file, there are a set of higher-level constructs to make that easy—like playbin2, for playing video and audio content, and tagreadbin to extract media metadata. The base classes that were implemented for 0.10, including those that have been added over the last five years, are also a highlight of the framework. Those classes handle things like sinks, sources, transforms, decoders, encoders, and so on.

There are also a number of bad points in the current GStreamer. The current negotiation of formats, codecs, and various other variable properties is too slow. The initial idea was to have a easy and comprehensible way to ask an object what it can do. That query will return the capabilities of the object, as well as the capabilities of everything that it is connected to, so the framework spends a lot of time generating a huge list of capabilities. Those capabilities are expressed in too verbose of a format in Taymans's opinion. Reducing the verbosity and rethinking the negotiation API would result in major performance gains.

The "biggest mistake of all" in GStreamer is that there is no extensible buffer metadata. Buffers are passed between the GStreamer elements, and there is no way to attach new information, like pointers to multiple video planes or information to handle non-standard strides, to those buffers. There also need to be generic ways to map the buffer data to support GPUs and DSPs, especially in embedded hardware. It is very difficult to handle that with GStreamer currently and is important for embedded performance.

While dynamic pipeline modifications work in the current code, "the moment you try it, you will suffer the curse of new segments", Taymans said. Those can cause the application to lose its timing and synchronization, and it is not easy to influence the timing of a stream, so it is difficult for an application to recover from. The original idea was that applications would create objects that encapsulated dynamic modifications, but that turned out not to be the case. There are also a handful of minor problems with 0.10, including an accumulation of deprecated APIs, running out of padding in some structures, and it becoming harder to add new features without breaking the API/ABI.

A look to the future

To address those problems, Taymans laid out the plans for GStreamer for the next year or so. In the short term, there will be a focus on speeding up the core, while still continuing to improve the plugins. There are more applications trying to do realtime manipulation of GStreamer pipelines, so it is important to make the core faster to support them. Reducing overhead by removing locks in shared data structures will be one of the ways used to increase performance.

In the medium term, over the next 2 or 3 months, Taymans will be collecting requirements for the next major version. The project will be looking at how to fix the problems that have been identified, so if anyone "has other problems that need fixing, please tell me". There will also be some experimentation in Git branches for features like adding extensible buffer metadata.

Starting in January, there will be a 0.11 branch and code will be merged there. Porting plugins to 0.11 will then start, with an eye toward having them all done by the end of 2011. Once the plugins have started being ported, applications will be as well. Then there will a 1.0 release near the end of 2011, not 2010 as was announced in the past. "This time we'll do it, promise". Taymans concluded his talk with a joking promise that "world domination" would then be the result of a GStreamer 1.0 release.