Development

On OpenGL debugging

During the Graphics and Display microconference at Linux Plumbers Conference (LPC) 2013, Ian Romanick presented a session detailing the state of the art in tools used for debugging and optimizing 3D graphics code. The state of said art is, clearly, poor—riddled with vendor-specific applications and options with limited functionality. The situation is not drastically better on proprietary operating systems than it is on Linux, but there is clearly more work that needs to be done to meet developers' needs.

Romanick's talk addressed 3D graphics in general terms, but the emphasis was primarily on 3D games—where achieving the maximum frame rate can both keep the players happier and sell more game licenses (to the game developer's benefit). On Linux systems, such games are written to the OpenGL API, while on Windows, the popular titles are split between OpenGL and Microsoft's Direct3D. Regardless of the API, however, the developer's needs tend to boil down to similar challenges: locating and fixing bugs in the shaders and GPU drivers, profiling the performance of the application, and tuning the code.

Debugging

The 3D game market on Windows dwarfs that of its Linux counterpart, so it is no surprise that there are far more 3D debugging tools for Windows. However, Romanick said, they are almost always proprietary and the majority of them are single-vendor. So developers must debug their code separately for AMD, NVIDIA, and Intel GPUs. These tools do offer a lot of 3D-specific debugging features, though, such as the ability to step through shader code, or the ability to inspect which on-screen pixels were drawn by which function calls. Several of the tools are what Romanick termed "driver assisted" debuggers, which let the GPU driver register callback functions and trigger a notification when "something the application should know about" happens—such as an error in shader code.

The Linux debugging landscape is far more sparse, he said. The best option at the moment is apitrace, which is free software and is under active development (including, he pointed out, with contributions from game maker Valve). But apitrace is not usable for debugging many kinds of application, particularly those that (like games) are concerned with interactivity. It requires running an application inside the apitrace environment, which captures and records the OpenGL commands so that they can be replayed and analyzed later. Another option, BuGLe, looked promising, but it has since stalled while the lead developer is off pursuing a PhD.

Other tools available for Linux tend to be proprietary, such as NVIDIA's Nsight and AMD's Gremedy gDEBugger. Both are vendor-specific, of course. In addition, Gremedy has been discontinued. The only option for "driver assisted" debugging on Linux is to use the GL_ARB_debug_output OpenGL extension and add gdb breakpoints in the driver.

Performance tuning

While apitrace and the vendor-specific debuggers offer functionality that at least approaches that of the Windows tools, where the Windows products truly excel is in performance tuning. In addition to the vendor tools, Windows developers can get a good system-wide view with GPUView. Together, they offer copious system data much like one might find in a CPU profiler, which allows the developer to see whether delays are caused by the application, the driver, the display manager, or other factors. The vendor tools also allow introspection, showing when drawing commands are submitted, how long they take to reach the GPU, and when the pixel finally make it to the screen.

On this front, Romanick said, Linux lags quite a bit. Apitrace is "not that useful," he said, and Nsight offers less than it does on Windows. Intel has a utility called intel_gpu_top which he described as "of very, very mild usefulness," providing a little system data, which was akin to "trying to use top to do performance tuning." There are a handful of driver-assisted techniques which can offer some help, such as instrumenting one's code with GL_ARB_timer_query and GL_ARB_performance_monitor calls. Gallium also offers a heads-up display that shows performance information, but ultimately the tools on Linux need a lot of work to rival their Windows counterparts.

Common problems

If there are so few good options, one might ask what 3D developers actually do in practice. The answer, Romanick said, is that they tend to use the vendor-specific tools where they can, and roll their own tools for everything else. But the vendor-specific tools are not much help on the development front, he said, since they often use undocumented ABIs and back-channels created solely for the vendor. Many of them are specific not just to one GPU vendor, but to a specific GPU, so that developers must repeat the debugging and tuning process for several generations of hardware for each vendor. In addition, the vendor-specific tools often target Direct3D ahead of OpenGL, and do not provide as many OpenGL features.

For rolling their own tools, developers tend to choose one of two routes. The first is to log data in their application, then export it for later analysis and visualization. For example, he said, at SIGGRAPH 2012 Valve described [PDF] Telemetry, the in-house tool it had developed to log and visualize OpenGL performance when it started its Linux porting effort. The second option is to collect data in the application and try to visualize it while the application is running. This, he said, usually takes the form of a "heads-up display" that shows the frame rate, where "hiccups" occur, and so on.

But both routes are awful, he said. They each require manual insertion of trace points in the application, which he described as putting "fancy rdtsc() calls everywhere." In the end, developers are faced with two unappealing alternatives: relying on generic, imprecise data collection methods hand-instrumented in the code, or using vendor-specific tools that must be run for each separate target system. "It's a fairly dire situation," he concluded, "I don't envy developers trying to make things faster."

The $64,000 question

Considering the direness of the situation, Romanick said, the big question for LPC attendees is whether the Linux community can provide help—most likely through new interfaces at the kernel level. He described a couple of basic requirements. First, developers need more fine-grained data about GPU execution than can be provided by GL_ARB_timer_query. This means timing information that tracks the entire process from the submission of a draw command to its completion.

Second, this information needs to include semantic information about the system. For example, when the drawing commands come relative to system events like the display's vertical blanking interval, or what portions of time were taken up by the compositor and display server, rather than the application itself. But the implementation must be careful not to leak system information that would constitute a security risk, he said. Also, the interfaces designed must be implementable by closed-source drivers in addition to open source drivers, and not provoke "too much rage" from driver maintainers. Romanick ended by appealing to the microconference audience to help him sort out the specific requirements. "I know who I can tell to go do this; I just need to know what to tell them to do."

The audience did provide some feedback at the end of the session, although the conversation had to move on to other presentations. One audience member asked what sort of events OpenGL programmers might be interested in that could be logged by the kernel's perf system. Romanick listed three: API usage patterns that cause the graphics driver to consume a lot of CPU, usage patterns in shaders that result in significantly different performance characteristics, and how specific drawing commands are pipelined through the GPU.

Another audience member asked if the architectural differences between GPUs would permit much in the way of a general-purpose solution. Romanick responded that different GPUs would of course generate different data, but that by comparison "GL_ARB_timer_query sucks"—reporting numerous operations as taking zero time, followed by one large time-chunk for the final command. In addition, the "coarse-grained stuff" would be similar for all GPUs, such as how long different shaders take to execute.

The holy grail is knowing what goes on inside of shaders, another participant said. Android has some tools (such as trace-viewer) that are currently only usable for Android Views, but they are open source, and do provide some of the semantic information Romanick referred to.

Improving the OpenGL debugging and optimization picture on Linux is likely to be a lengthy process. But it is clearly one that needs addressing; the topic frequently comes up whenever game development is discussed (as it did in Valve CEO Gabe Newell's keynote at LinuxCon North America before LPC began). Furthermore, when one considers how unpleasant the graphics debugging and tuning process is on Windows, Linux could surpass the proprietary offerings, which would benefit quite a few cross-platform developers at the same time.

Comments (1 posted)

Quotes of the week

Comments (3 posted)

GTK+ 3.10 released

GTK+ 3.10 is now available. New are support for Wayland 1.2, scaled output on high-DPI screens, client-side window decorations, and several new widgets.

Full Story (comments: none)

Mozilla Lightning 2.6 released

Version 2.6 of Mozilla's Lightning calendar component has been released as an add-on for Thunderbird 24 and Seamonkey 2.21. New in this release is support for Google Calendars over CalDAV using most recent API revision, the ability to assign multiple categories to an event, the ability to schedule recurring events on the last day of the month, and updated time zone data.

Comments (none posted)

WebRTC enabled in Firefox for Android

At the Mozilla Hacks blog, Maire Reavy announces that support for WebRTC has been switched on by default in Firefox for Android, starting with Firefox 24. This allows web applications to capture audio and video from Android devices, although Reavy notes that some caution is called for. "This is still a hard-hat area, especially for mobile. We’ve tested our Android support of 1:1 calling with a number of major WebRTC sites, including talky.io, apprtc.appspot.com, and codeshare.io."

Comments (none posted)

GNOME Shell and Mutter-Wayland 3.10 available

Version 3.10.0 of GNOME Shell, the official GNOME Shell Extensions, and the Wayland-backed branch of the Mutter window manager have all been released. These releases preview the imminent release of GNOME 3.10 itself, but incorporate a number of changes on their own as well.

Comments (none posted)

GStreamer 1.2 released

Version 1.2 of the GStreamer multimedia framework has been released. Packages for GStreamer Core and GStreamer Plugins are available. The 1.2 release is API- and ABI-backwards-compatible with GStreamer 1.0, however there are several new features introduced. Several new plugins are included, including support for DASH adaptive streaming, JPEG2000 images, VP9 and Daala video, and decoding-only support for WebP. There is also a new command-line playback tool called gst-play-1.0 (designed for testing purposes), as well as numerous bugfixes and improvements.

Comments (none posted)

GNOME 3.10 Released

Full Story (comments: 104)

Development newsletters from the past week

• Caml Weekly News (September 24)
• What's cooking in git.git (September 20)
• What's cooking in git.git (September 23)
• GNU Toolchain Update (September 24)
• Haskell Weekly News (September 18)
• OpenStack Community Weekly Newsletter (September 20)
• Perl Weekly (September 23)
• PostgreSQL Weekly News (September 23)
• Ruby Weekly (September 19)
• Tor Weekly News (September 25)

Comments (none posted)

Apache Foundation embraces real time big data cruncher 'Storm' (The Register)

The Register reports on the recent decision by the Apache Foundation to accept the Storm project into the Apache incubator program. "Storm aims to do for real time data processing what Hadoop did for batch processing: queue jobs and send them off to a cluster of computers, then pull everything back together into usable form."

Comments (none posted)