GStreamer Conf: The approach of GStreamer 1.0

The GStreamer framework handles media capture and delivery for many of desktop Linux's most popular applications, but more than ten years out, it still has not made a 1.0 release. To be sure, the actual version number does not matter nearly as much as the stability of the software and the reliability of the API, but GStreamer's 1.0 release has seemed tantalizingly close for the past several months. The third annual GStreamer Conference was held in San Diego the last week of August, and, while 1.0 did not make its debut at the event, the core developers did give a solid deadline, which should be welcome news to application authors and distributions alike.

Ironically, one of the key impediments to a 1.0 release has been the success of the current (0.10) stable release. With the framework working well, a 1.0 release might seem like a simple project governance decision. But because an N.0 release carries with it the implicit promise of API and ABI stability, the project decided it had to implement several key changes before it could put the final stamp on the release. GStreamer maintainer Wim Taymans opened the conference with a keynote talk covering the 0.11 development cycle, including what needed to change from the 0.10 series.

The two biggest challenges, he said, were the advent of embedded Linux (often system-on-chip (SoC) boards) systems using GStreamer and the move toward GPU-accelerated rendering on the desktop. SoC platforms posed a challenge because they include substantially different hardware, such as different memory architectures and hardware rendering pipelines (which could include discrete components such as a video scaler unit). GPU acceleration on the desktop can result in significant performance gains, but it comes at the cost of supporting multiple, incompatible GPU libraries from the various hardware vendors. Those library incompatibilities are not limited to the lowest levels, either. Some GPU chipsets support higher-level features like subtitle compositing, while others do not.

Along with the changing face of hardware, the project had several wishlist items it wanted to implement for a 1.0 release. One was an infrastructure that allowed a GStreamer application to attach useful non-media metadata to a buffer. Such data might include the coordinates of a "region of interest," for example. If the GStreamer pipeline processing a video knows that it will need to crop the video in order to display it, the optimal performance is usually obtained by marking the region, but let the video hardware do the cropping at render time. There are other uses for attaching metadata, of course, such as tagging parts of an image with facial recognition information. Along similar lines, GStreamer 1.0 will support passthrough of compressed audio to the playback hardware. This allows headsets and USB sound cards (an increasing number of which support MP3 or AAC decoding in hardware) to handle the audio stream, again saving CPU cycles.

Last but not least, the GStreamer team wanted to fix the outstanding issue of dynamic pipelines. GStreamer allows application authors to construct pipelines that connect media "source" elements, apply transformations and filters, then deliver the stream to a "sink" element. But up through the 0.10 series, changing the pipeline after its initial setup was painful. Pulling out and replacing an element caused downstream elements to lose context information like timing and formatting details, forcing the application to re-examine the pipeline. It was not esoteric use-cases hit by this limitation, either: plugging in a headset while audio was playing forced a pipeline change (by using a different sink element), as did applying a video filter in an application like Cheese (by inserting a filter element).

On top of the wishlist features were the usual improvements to memory management and cleanups that accumulate over the lifetime of a complex framework. Version 1.0 allows applications to pool buffers to reducing copying, query the supported memory types (such as video acceleration API (VA-API) surfaces), and split buffers up into separate memory blocks (which could allow faster handling of streaming media like VoIP packets). Also 1.0 strives to simplify application authors' work by providing a number of new audio and video "base classes" that represent the most commonly-needed media formats — in contrast to 0.10, which required the author to fully describe each media format.

Taymans reported that the 0.11 code had replaced GStreamer's master branch in March 2012, and was currently at version 0.11.93. The team is "kind of out of ideas of things to add," he said, and most of the common media players using GStreamer have already been ported over to the 0.11/1.0 API. Nevertheless, there are a few pieces that require more testing, including the dynamic pipeline support and the new base classes. However, the Linux distributions may have other challenges that hold up deployment. For example, he said, the Fluendo MP3 codec and DVD playback element shipped by Ubuntu are built to work on 0.10, and may not be updated for 1.0.

GStreamer 0.10 and 1.0 will be installable in parallel, which means Ubuntu could include the Fluendo elements in a 0.10 pipeline and still use 1.0 elsewhere. But this is not a good long-term solution, Taymans added. For one thing, 0.10 is deprecated and will eventually stop receiving updates. But the more pressing issue from GStreamer's perspective is that GNOME has decided to use 1.0 for its upcoming GNOME 3.6 release. GNOME cannot reasonably be expected to base its entire desktop environment on a deprecated version of GStreamer, but it cannot ship with a dependency on an unreleased branch either. As a result, GStreamer will formally release its version 1.0 before the release of GNOME 3.6. That does not give a precise date (although Taymans said the team hoped to settle on one over the course of the conference), but it does provide a more concrete anchor to the when-will-1.0-arrive question than GStreamer users have had up until now. GNOME 3.6 is scheduled for release on September 26, and while it might slip a few days, that still places GStreamer 1.0 within the month.

GStreamer also saw the first release of its software development kit (SDK) this summer, as Andoni Morales explained in another session. The GStreamer SDK is a bundle designed to help GStreamer application authors by providing stable, installable builds for Linux, Windows, Mac OS X, and soon for Android as well. The issue that led to the SDK's creation is that despite GStreamer's staunch commitment to being a cross-platform media framework, it is still predominantly a Linux-only tool. But a big part of the reason behind that situation is the lack of installers for Mac and Window developers. The SDK provides binary installers, plus tutorial-based documentation. In addition, it splits up the GStreamer package into more modular components — the "good,""bad," and "ugly" plugin packages found in Linux repositories are not particularly descriptive, and they are quite large.

The first version of the SDK is based on 0.10, and provides a tested set of playback codecs, plus Python bindings and integration with Apple's Xcode IDE and Windows' Visual Studio. Several of the other talks over the course of the event also dealt with GStreamer development tools, including Rene Stadler's debugging tool (which turns GStreamer's famously verbose log files into easier to parse visualizations) and Edward Hervey's survey of CPU and memory optimization utilities. Consequently, whenever GStreamer 1.0 does arrive, it looks as if the project will be prepared to make a bigger push to engage application developers.