What's new in GStreamer 1.6

The 1.6 release of the free-software multimedia framework GStreamer arrived on September 29. As usual, the suite of updated GStreamer plugins adds support for several new audio and video file formats. In addition, the new release adds several new core features, such as stereoscopic video support and the ability for Video4Linux2 sources to adjust their settings on the fly. There has also been a major rewrite of the video-scaling code, supporting more algorithms, more options, and allowing all video formats to be rescaled.

Video playback features

Stereoscopic video is a way of encoding 3D video signals by providing separate signals for the left and right eye (as opposed to anaglyphic 3D, which uses superimposed red and cyan tinted images). GStreamer 1.6 supports all of the options used to arrange the two sub-signals within a video stream (side-by-side, top-and-bottom, interleaved, etc.). The stereoscopic framework also allows GStreamer to support so-called "multiview" video, in which several camera angles are encoded into one signal. There is also a 3D demo application for developers to explore, since application-level support has yet to land in GStreamer-based video players.

GStreamer has a software-fallback module named gst-libav that applications can employ to transparently support media formats outside of GStreamer's existing plugin set. The module had used libav as its fallback media player (as the name might suggest), but as of the 1.6.0 release, it has instead switched to FFmpeg. To avoid breaking existing applications, though, the name of the module and its functions will not change.

There is a new API for "trick mode" (i.e., fast-forward, fast-reverse, and normal-speed reverse) playback. With the new API, audio can be disabled and video decoding can be pared down to keyframe-only decoding, both of which significantly reduce the processing overhead. Right now, the full API is used in only two modules (gst-libav and qtdemux), while the normal-speed reverse-playback function is available for audio-only files. Support in additional playback modules is expected to follow.

GStreamer's video-scaling capabilities have perhaps improved more in the this development cycle than any other section of the framework. In the 1.4 releases (the previous stable series), not all video formats could be rescaled, and the high-quality Lanczos scaling algorithm was only supported for an even smaller subset of formats. The 1.6 release fixes this, making scaling a general-purpose operation. The new rescaling function is also more efficient than its predecessor, avoiding several unnecessary color conversions and providing up to an eight-fold speedup. There are also several new resampling algorithms to choose from (including Hermite, Spline, Catmull-Rom, Mitchell, and Robidoux interpolation) and all of the resampling algorithms support an adjustable number of filter coefficients (e.g., "taps"), thus letting developers find the best trade off between output quality and computational overhead required.

In addition to allowing higher-quality rescaling operations, 1.6 provides configurable settings for converting high-bit-depth colors to the eight-bit-per-channel colors used on most displays. The settings can reduce the appearance of "color banding" (which occurs when color values are simply truncated) by dithering the output. A rescale operation can also be gamma-corrected now, which should produce more accurate results. Furthermore, a video frame can now be cropped or padded before the rescale operation is performed, which should allow for more efficient scaling, and video that has alpha transparency can now have its alpha channel operated on or passed through unchanged, whichever is more desirable.

Hardware features

On the hardware side, the new release includes a significant rewrite of the support for the BlackMagic Design DeckLink series of pro-level capture cards. This is a high-end device class used for 4K video processing; improving support makes GStreamer a better option for professional video production.

The project has also added several key improvements to OpenGL support, starting with initial support for OpenGL 3.0 (at this time, just in a subset of GStreamer elements). But there are enhancements to the framework's existing OpenGL support, too. GStreamer applications now have the ability to set up and use multiple OpenGL contexts, as well as to share access to those contexts with another process. That capability is used in several new video-sink elements for various toolkits (GTK+ and Qt on the Linux side, plus Apple's CoreAnimation library).

Many modern Video4Linux2 (V4L2) input devices support multiple resolutions, multiple frame rates, and other parameters, but older GStreamer releases did not support renegotiating those parameters without stopping and restarting the filter pipeline. GStreamer 1.6 fixes this shortcoming. When a V4L2 source needs to renegotiate its output format, it can signal the next element in the pipeline (be it a video sink or filter), then wait for all of its output buffers to be released. At that point, it can destroy the old buffer pool, create a new one with the new parameters, then signal the next element and resume sending video down the pipeline.

The new release also adds a sink element named splitmuxsink that is designed to let applications split recordings into chunks of easily configurable size. This is needed in a number of situations, such as recording media streams that may hit the filesystem's size limit, but it is also convenient in other situations, such as in video-surveillance systems, where the application may never stop recording, but it would be nice to split the output up for easier maintenance or to automatically rotate out old files. There is a corresponding playback element named splitmuxsrc that can seamlessly play files recorded with splitmuxsink.

Streaming media

Streaming-media support was improved, too. Support for Dynamic Adaptive Streaming over HTTP (DASH) was rewritten and now shares the same codebase as GStreamer's other modules for adaptive-streaming protocols (such as HTTP Live Streaming or HLS). There have been numerous improvements to Real Time Streaming Protocol (RTSP) and Real-time Transport Protocol (RTP) support, such as support for synchronizing playback between several devices over the network and support for RTP retransmission, which helps applications recover from packet loss.

Part of the streaming-media support relies on another new feature: support for time synchronization with network clock sources. 1.6.0 is also the first release to include usable (i.e., non-experimental) support for live mixing of media streams. That could make GStreamer useful for a range of new use cases, such as multi-source video capture or live video overlays.

Another feature related to streaming-media support is the inclusion of new APIs to support content-protection layers (in other words, DRM) and the Common Encryption Scheme (CENC). This set of APIs makes it more likely that commercial video providers will use GStreamer in their products. The release notes point out, though, that the GStreamer project does not ship any DRM integration modules.

Other features

Support for subtitles (and general-purpose text overlay) has been enhanced in the new release. Subtitles can now be rendered for the resolution of the output window when that window is higher resolution than the video source. Previous releases had tied the resolution of overlay text to the resolution of the video being played back, which often resulted in blocky, hard-to-read subtitles when the user watched lower-resolution content.

The GstValidate pipeline-validation tool now supports plugins. Three have been implemented so far: ssim can be used to compare individual video frames, fault_injection allows users to test error-handling by injecting bad data, and gtk allows users to simulate GTK+ events. There is also a new test framework named GstHarness available, which is designed to support writing unit tests for GStreamer itself.

There are many more individual features to be found in the new release that will be of interest to some GStreamer application developers—such as new functions available in existing GStreamer elements, improvements to codecs, and performance optimizations. Multimedia support is an ever-changing field; judging by the 1.6.0 release, the GStreamer project is proving quite capable of keeping pace.