VA API slowly -- but surely -- making progress

For years, video acceleration on Linux and Unix-like systems has been possible only through the X Video Motion Compensation (XvMC) API. A new, more modern replacement is in the works: the Video Acceleration API (VA API). XvMC offered a way for applications to partially offload decoding of MPEG-2-encoded video directly to the video card's graphics processing unit (GPU), which provided more speed and a smoother display when compared to processing on the system CPU. VA API will add more video formats to the accelerated decoding pipeline, as well as video encoding and post-processing.

Applications can use XvMC to save CPU cycles and reduce power consumption, but the API is extremely limited: it only supports MPEG-2 video, and only offloads the motion compensation calculations. On the other hand, XvMC is supported by a wide range of video hardware from different manufacturers, current and legacy. X developers talked about extending XvMC to support other formats and to add more steps in the video decoding process — such as the costly inverse discrete cosine transform (iDCT), but eventually decided to write a new API from the ground up.

A new specification

The VA API project was launched in December of 2006, spearheaded by Jonathan Bian from Intel. Initially, the API only attempted to cover video decoding, adding support for the VC-1 codec, and adding iDCT and variable-length decoding (VLD or "slice-level acceleration") to the techniques offloaded to the GPU. The specification subsequently added MPEG-4 ASP/H.263 and MPEG-4 AVC/H.264 to the list of supported codecs, and acceleration hooks for deblocking and inverse zigzag ordering (IZZ). VA API has also been designed to handle color space conversion, gamma correction, scaling, and other video processing techniques traditionally handled by the X video extension (Xv) rather than XvMC itself.

The most recent revision of the specification is 0.30, released in March of 2009, more than a year after the release of 0.29. 0.30 introduces a major new feature to the API: hardware acceleration for video encoding. The encoding support supports multiple codecs; currently H.263, H.264, and generic MPEG-4 are specified.

In addition to the specification, the project produces libVA, an implementation library targeting Linux systems. Using libVA requires supported hardware and a supported video driver, however, which has historically been the sticking point. VA API is maintained by a team at Intel: Bian, Austin Yuan, and Waldo Bastian. Despite calls for input and outside participation, the competing GPU manufacturers have not contributed, each preferring to put effort into its own hardware video acceleration project. NVIDIA has the Video Decode and Presentation API for Unix (VDPAU), and ATI has X-Video Bitstream Acceleration (XvBA). Thus, the only Linux video driver that natively supports VA API is Intel's own (closed source) psb-video, for its Poulsbo chipset, which is aimed at low-power devices like netbooks and features the GMA500 GPU.

Beyond Intel: participation and driver support

The single-vendor problem began to turn around in late 2008, when embedded systems vendor Splitted-Desktop Systems (SDS) decided to adopt VA API for some of its Linux-based low-power boxes. SDS's Gwenole Beauchesne developed bridge code to use VDPAU and XvBA as back-ends for VA API, bringing support for the API to NVIDIA and ATI card owners. Neither VDPAU nor XvBA support the entire product range from either manufacturer, of course, and both vendor APIs are limited in scope — neither supports encoding acceleration, and VDPAU does not support MPEG-4 ASP/H.263.

Taking advantage of this work requires a VDPAU- or XvBA-supported video card, a VDPAU- or XvBA-enabled driver, and a patched version of libVA. Beauchesne's patches to libVA add fields to VA API's control structures to pass parameters down to the VDPAU and XvBA back-ends. He publicly maintains a patch set at SDS in addition to patched libVA packages while the changes await upstream integration.

To use Beauchesne's bridge work for an NVIDA card a user would need to install his patched libVA library, the vdpau-video package that links VA API calls to VDPAU API calls, and a video driver that implements VDPAU. Thus far, only the binary nvidia driver from NVIDIA implements VDPAU, not the free nv driver from X.org. Similarly, an ATI user would need the patched libVA, the xvba-video bridging package, and a Radeon driver that implements XvBA — currently only the closed source fglrx.

In February of 2009, S3 Graphics released its first VA API-compatible driver, for the Chrome 400 and Chrome 500.

Application support

Regardless of the chipset or driver used, it is impossible to see the benefits of VA API if the application does not support it. Fortunately, open source video projects are beginning to take a look. "Official" support is currently limited; RealPlayer for Mobile Devices is designed for Poulsbo-based hardware, and thus to take advantage of VA API through libVA and the psb-video driver. Likewise, closed source codec vendor Fluendo makes specialized builds of its GStreamer codecs for Poulsbo devices that will use VA API. Users with NVIDIA or ATI GPUs would not gain hardware acceleration with either product because neither uses the patched version of libVA compatible with the VDPAU and XvBA back-ends.

Fortunately, Beauchesne provides patched versions of MPlayer and FFmpeg, making this the simplest way to experience hardware acceleration on desktop Linux systems. Build notes are included. The VA API-accelerated MPlayer will work with any back-end: VDPAU for NVIDIA, XvBA for ATI, or native Poulsbo drivers for Intel. Beauchesne has sent his patches upstream, but it is not yet clear when MPlayer or FFmpeg will include VA API in official releases.

VLC has added VA API support in its development branch, and a patch is available through the developers' mailing list that adds support to current releases. MythTV can be patched to support VDPAU, but thus far there has been no work on supporting VA API. It is possible, however, to set up MythTV to use MPlayer as a video playback engine, so it would be possible to leverage Beauchesne's patched version of MPlayer that way.

Benchmarks comparing VA API-accelerated decoding to CPU-based decoding are available from several sources. Beauchesne has run multiple comparisons on different systems: Opteron, Phenom, and Atom processors, and NVIDIA, ATI, and Intel GPUs, all using MPlayer. Jean-Baptiste Kempf has run real-world tests on the VA API-enabled version of VLC. The difference in CPU utilization can be dramatic; the CPU has little to do other than pass the compressed bits to the GPU. Beauchesne's numbers report CPU utilization between 66.3 and 98.4 percent for unaccelerated video playback, and as low as 0.6 percent with VA API acceleration.

Beauchesne said he hopes to see more applications take an interest in VA API, noting that the average desktop user only benefits from it when the players adopt it. He added that he had tried to add VA API support to the open source Flash decoder Gnash, but eventually abandoned the effort because the peculiarities of the Flash format (such as requiring all pixels to be converted to RGB24) sapped away most of the gains of hardware acceleration.

Future work

On the specification side, Bian echoed Beauchesne's sentiment, hoping that more developers would take an interest in contributing to the work. "Many aspects of the API could benefit from video application developer inputs ... as the app developers are ultimately the consumers of the API." Beauchesne's patches are an example of that, he said. Yuan added that moving the libVA code to Freedesktop.org's public Git repository should streamline the process for integrating contributions from outside Intel.

Video encoding is new to the specification, but has not been tested in the wild; that is expected to happen when Intel releases its Moorestown mobile platform late in 2009. Bian indicated that more video post-processing filters may make it into the next revision, and Beauchesne added that he would like VA API to support rendering directly to an OpenGL texture, as this would allow it to work with Clutter.

The bigger news is that Intel plans to launch a VA API-supporting driver for additional graphics chips. According to Bian and Yuan, the G45 chipset is next. G45 is a Core 2 chipset introduced in 2008, and already found in a variety of products, including desktop motherboards.

Still, at the moment the only "native" VA API video drivers are Intel's Poulsbo driver and S3's Chrome driver, neither of which is open source. Although Beauchesne's bridge code for VDPAU and XvBA is open source, both rely on the binaries drivers provided by NVIDIA and ATI to implement VDPAU and XvBA, respectively.

The open source RadeonHD and Nouveau drivers are not pursuing hardware video acceleration, through VDPAU, XvBA, or VA API. Gallium3D is another possibility for purely open source drivers, although developer Younes Manton said that VDPAU was the current focus of his attention.

Searching through the mailing lists, IRC logs, and discussion forums of Linux video applications reveals that a "wait and see" attitude was taken by most projects during the first year or so of VA API development. Some of that is, no doubt, caution that would be taken towards any entirely new API; perhaps some of it is attributable to the perception that VA API was an Intel-only project or at best fated to duke it out against VDPAU and XvBA until one of them becomes dominant. VA API is the only hardware video acceleration standard open to all vendors and with the advent of bridging code for NVIDIA and ATI it is finally possible to see its effects on desktop systems. Hopefully that bodes well for its continued development as a standard.