Where to next for GStreamer

The 2012 release of GStreamer 1.0 marked a significant achievement for the project: the multimedia framework was now a reliable "plumbing" layer used in the majority of Linux distributions and in a wide variety of multimedia products, and it had an API and ABI that it could confidently declare stable. It had even fixed most of the outstanding issues that users had complained about. But that success raises the question of where GStreamer as a project should go next. The topic came up in a number of talks at this year's GStreamer Conference, and there appears to be rough consensus that the next major set of challenges for the GStreamer team includes making development easier for the framework's downstream users.

Tim-Philipp Müller opened the conference on Tuesday with a look at the current status of the project and some ideas for taking things "to the next level" post-1.0. The 1.0 release, he said, allowed the project to fix and overhaul quite a few things that required breaking APIs or ABIs from the 0.x series. In particular, GStreamer largely fixed its memory management problems: it is more efficient, allowing zero-copy buffer management in many cases, and it is more flexible, abstracting away the low-level details of earlier releases. It also implemented the long-requested feature of dynamic pipeline construction and reconfiguration. This had always been a sore point in the past he said; it was sometimes possible to reconfigure a GStreamer pipeline in the 0.10 release, but one could never be sure if it would break. These changes also lay the groundwork for transparent support of hardware acceleration, which he said should land in GStreamer six to twelve months from now. After the 1.0 release came the 1.2 release, which did not add major features, but did include support for additional protocols and was accompanied by binary releases for Windows, OS X, Android, and iOS.

When asking where GStreamer should go next, he said, it is simple to ask "are we done?" and "will we ever be 'done'?" Those questions may not have answers, but there are less abstract questions that are more meaningful, such as "what is needed to keep GStreamer relevant and interesting to developers?" There are a few technical features that are still missing, he said, but most of the challenges are not of a technical nature. On the technical front, for example, there is still work underway to allow sandboxing of GStreamer pipelines—which is good for security and will help to isolate crashes. There is also interest in allowing a mechanism to connect GStreamer to a DRM system; "we don't like DRM," he said, "but manufacturers need it" and it is perceived by them as a major gap between GStreamer and other frameworks. There are also 3D features still to come (including stereoscopic video and handling depth information like the Microsoft Kinect camera captures), as well as Blu-Ray disc support and improvements to subtitle handling.

That may sound like a long list of technical challenges, but none of them are really unsolved problems or tricky challenges (with the possible exception of transparent access to hardware acceleration, which can be quite tricky to implement across the variety of embedded boards). Most of them will be handled by development work that is already underway. The bigger challenges are working more effectively with the developer community. This includes several things, he said: quality assurance (QA), better development tools, engagement with the GStreamer community, and routine maintenance.

The QA challenge is one of the biggest, Müller said. It is not so much that GStreamer lacks quality, he said, as it is that the project needs to work at preventing regressions. In "the good old days," he pointed out, Nokia did the project a major service by coordinating QA. Since Nokia dropped GStreamer, however, no one has really taken up that slack. Consequently, most bugs are discovered by application developers when their own code breaks, or they are discovered by outside companies making GStreamer-based products—which means, of course, that the bug report sits in a private issue tracker and may not be reported upstream for quite some time. Some bugs are also found by Linux distribution build-bots, he said, but even those are mostly just automated unit tests. While make check is useful, he said, it does not reveal bugs found with real media files or interactive usage.

GStreamer needs a proper test suite, he said, one that comes with installable tests (including tests that can be run remotely for when GStreamer is cross-compiled for embedded devices). It also needs nightly builds and tests for extreme conditions, such as running a video capture job for 36 hours (at the end, will the audio and video still be synchronized?). These things require a massive effort to be done correctly, but they also require a lot of "social" buy-in from GStreamer companies: QA tools cannot be allowed to bit-rot, and the companies involved must also care when a bug is uncovered by QA testing, or else they will eventually lose interest in the process.

Developer tools needed by GStreamer start with debugging tools, he said. Certainly GStreamer applications can be debugged today, but only "if you know where to look." Developers need tools that will allow them to take snapshots of pipelines, to trace pipelines (such as to discover where it is in a pipeline that a media frame gets dropped), and to hook their debugging tools into embedded devices. The project also needs to work on tools for Windows developers and other on non-Linux OSes, since those developers will probably not be using GDB. There is some progress on this task, he said; a lot of individuals write their own tools and GStreamer has started to collect them in a repository, but more is still needed.

Engaging with the GStreamer-using community takes several forms, he said. The project needs to encourage more people to work upstream. Too many companies (including system-on-chip (SoC) vendors) write their own plugins in order to check the "GStreamer support" box on their packaging, but they do not actively cooperate with the project. As a result, many of them end up "solving" the same problems over and over again. The project also needs far more documentation: there is a shortage of high-level application design documentation, and there is a long-planned "GStreamer book" that has never materialized. Müller also noted that a lot of people seem confused by the GStreamer SDK effort, which is not a direct product of the GStreamer project (although it was put together by people who are project participants).

Finally, he said, the project needs to take on maintenance in a more deliberate way. It cannot be simply an "after hours" task. Currently, according to the Ohloh statistics, 50% of the project's maintenance fixes are done by three individuals. It is good that the right people are handling maintenance, but more people would be better (as well as more robust against vacation interruptions).

Several of these non-technical challenges have partial solutions already, Müller said, so they are more a matter of focus and execution. One recurring issue in all of them is that it is currently difficult for an interested company to contribute back to GStreamer—outside of assigning an employee to work on the code, or sponsoring the annual GStreamer Conference. Perhaps forming a non-profit organization would help, he said, but that is a big question deserving careful consideration. Ultimately, he concluded, GStreamer is much better technically than it was one year ago, its ecosystem is growing, and there are a lot of companies and projects that have a long-term interest in the project's success—the project just needs to make better use of its strengths in those areas.

Developers, developers, developers

The theme of providing better tooling and support for downstream developers was picked up in the second day's keynote by Wim Taymans. Taymans talk addressed common pitfalls and best practices for developing GStreamer-based products, and many of his observations related to how companies using GStreamer either did or did not engage with the development community. For example, his first observation was that companies often develop their own plugins "on an island" rather than working directly with the project. There are valid reasons for this approach, but it causes complications. Even when a company develops its plugin as an open source component, he said, if they do it in their own GitHub repository lots of people clone and fork it before it is ready. After the fact, it becomes very difficult to locate the original, canonical version of the plugin. If GStreamer hosted a repository for downstream plugin developers, Taymans said (which it is considering), that might alleviate the headache.

Other companies get into trouble by starting their projects off writing their own multimedia layer from scratch ("or writing several of them," he added), then trying to shoehorn the existing codebase into GStreamer when they give up on their own framework. Often the mapping between the existing code and GStreamer is imperfect, or the code does not expose everything that GStreamer needs to operate (such as assuming only one particular media format). The GStreamer community can help if it is asked, he said, but it can only help if it knows about the problem. Similarly, other companies make the mistake of selecting a hardware platform first (such as a SoC), then have to do lots of extra work to make GStreamer run on it when they discover that the only GStreamer plugins for the board are untested or limited in features. Here again, the community would be happy to help, but it often does not find out about these difficulties until late in the process.

The community could help overcome some of these pitfalls with better documentation, he said: insufficient knowledge of GStreamer leads a lot of companies to create inefficient pipelines or build ugly hacks because they are unaware that a feature already exists. There are also a lot of cases where companies fork GStreamer in order to implement their own features, but this ultimately leaves them trapped on an old release when they do not know how to migrate to the latest version of GStreamer. Forks are encouraged, he said, but companies should at least open a bug in the GStreamer issue tracker when they encounter a problem; the community might implement a fix. Better testing tools would also relieve some of the pain of migrating between releases.

Taymans also observed that the non-technical challenges have become GStreamer's next big target. Companies frequently say that GStreamer is hard to debug and test. In particular, developers often test a format decoder only with a small handful of test files, when what they really need is a test harness that can automatically test all of the features of a proper decoder. He also mentioned the documentation problem, noting that too much GStreamer behavior is underspecified—knowledge is available only through "common understandings" between experienced GStreamer developers, and too much of the GStreamer API is "documented" only by the example plugins that ship with the releases. He also cited the need for more high-level base classes that provide useful abstractions, and suggested that the project needs to draw in a number of out-of-tree plugins, keeping them closer to GStreamer itself. That makes them easier for other developers to find, and helps the project maintain a better overview of what downstream projects are doing and what they need.

A lot of the challenges discussed during this year's conference indicate normal growing pains for a useful project; GStreamer has evolved into a reliable infrastructure project. That in turn means that fewer resources will be needed to develop wild new features, but more resources will be needed to test, refine, and battle-harden the code for frequent use. It is a good problem to have, but it may not be an exciting one to talk about. After all, the conventional wisdom is that new developers are attracted to writing interesting features—far fewer are attracted to a project for maintenance duties. However, if the talks at this year's conference are anything to judge by, there are a lot of interesting GStreamer-based projects downstream needing consultation—more than enough to attract the time and attention of developers for several years to come.

[The author would like to thank the Linux Foundation for travel assistance to Edinburgh for GStreamer Conference 2013.]