X and SteamOS

In a talk entitled "SteamOS Magic", longtime X developer Keith Packard looked at the new Linux "distribution" and the effort to turn the Linux desktop into a gaming console. It turns out that, with a fairly small amount of code, Steam and SteamOS creator, Valve, was able to take the existing X-based desktop and turn it into a "living-room experience". He gave the talk at LinuxCon North America in Chicago on August 21.

Packard's employer, Intel, has been "working closely with Valve" on graphics issues for some time now. Valve's interest is to see more Linux gaming and, to that end, the Steam game-management client was ported to Linux with Intel's help. Packard said that Valve was "happy to work with us", since Intel has a "nice, compliant OpenGL" running on its GPUs.

Steam and SteamOS

From the perspective of a game developer, there is not much difference between Steam and SteamOS. The latter is a way to create a new console, without developing new hardware. Instead, a standard PC platform is used, which might sound familiar. The architecture is similar to that of the Xbox, he said. SteamOS is meant to support the existing catalog of Steam games, many of which now run on Linux.

There are three basic elements to the Steam experience. First, there is the "enormous library of games" that have been ported to Steam and are tested to ensure they run well on Steam. There is a store to sell hats and other merchandise. Finally, there is a social network in Steam that uses both voice and text integrated into the game. "Even gamers have friends", Packard said with a chuckle.

The user interface revolves around the "Steam overlay", which is an in-game interface that "sits on top of your game". It is "amazing" that the overlay works at all, he said. To make it work, it redirects the internals of OpenGL to show the overlay. He is not really a video gamer (instead preferring a different game that "involves Emacs and typing"), though his children are, so he has seen the overlay in action both in testing and with them.

SteamOS is based on Debian Linux, with the standard X Window System on top of it. There are a number of customizations, as well as the SteamOS compositing manager (steamcompmgr). There is also "a long list of kluges" that make X work for SteamOS, he said.

Steam and SteamOS use essentially the same software, but provide a different experience for users, which embodies the difference between the desktop and console worlds. On the desktop, there may be many applications running on multiple monitors, with desktop widgets, icons, menus, and so forth. On a console, there is just one full-screen application. Any system UI in a console is either an overlay, as in Steam, or simply replaces the game entirely.

The ecosystems for the desktop and console are quite different as well. Developers focus on the newest CPU/GPU combinations for the desktop, typically developing on pre-release hardware and targeting future hardware configurations. Desktop games will tolerate a wide range of systems, with configurable rendering options and output sizes. The console ecosystem is more homogeneous, with the same CPU, GPU, monitor size, and basic input devices on every system. It also features limited configuration, no screen-size changes, and few rendering options.

So the "SteamOS challenge" is to take a desktop operating system and window system, add in desktop-oriented games, and run it all on desktop hardware—all while providing a living-room experience, Packard said.

Enter X

X makes for a good user experience (UX) toolkit because it focuses on "mechanism, not policy", he said. There is no built-in UX and there are hooks that can be used to design any UX, such as external window management and external compositing. In addition, of course, the source is available to be hacked up in whatever way is needed.

The SteamOS UX has two modes. "Desktop mode" is a standard GNOME desktop running the Steam client. "Big picture mode" uses the SteamOS Console UX, which is what the rest of the talk would be covering, he said.

Desktop computers have keyboards, but consoles have controllers. In addition, desktop games expect to get their input from a keyboard and mouse. So Valve built a custom, programmable controller that looks like a keyboard and mouse to the game. The user sees and interacts with a controller, though, that maps various buttons and other controls to keyboard and mouse events.

It turns out that for the living room, "mode switching is bad". A TV can go black for five or ten seconds if you switch modes. That is only getting worse with modern video links, Packard said. Also, the monitor may not support the mode that the game wants, especially if it wants a large mode. To combat that problem, SteamOS never changes the video mode, it simply scales the game to fit the size of the output display.

UI components

SteamOS has several different UI components, starting with an X server that has some "custom modifications". There is also the steamcompmgr, which is a custom compositing and window manager. It talks to the X server, as does the Steam UI, which appears to be the same as the desktop version, though there is no source available for that piece. The Steam UI is a proprietary piece of code that is "watching everything you do", he said, which might be a bit worrisome for some.

The games also talk to the X server, but there is an LD_PRELOAD library that intercepts mode-switching calls. A game might make an RandR call to change the mode, and the intercepting library just "says yes and does nothing". It records the information for use in calculating the appropriate scaling, however.

Steamcompmgr is based on the original X compositing manager, which is some "really bad X code", he said. It is GL-based, which ensures VBLANK synchronization and provides the fastest drawing path. Window management is integrated into the compositing manager, which is common for X environments. There is just one full-screen application window that is scaled to fit the screen (with letterboxing or pillarboxing to preserve the aspect ratio), with the Steam UI overlay optionally on top of that. In many ways, SteamOS is simply using the X Window System to "do what it supposed to do", he said.

Some implementation details

Many games use Simple DirectMedia Layer (SDL), which creates two windows, one application window centered inside the other, full-screen window. Steamcompmgr digs the second window out from within the first and uses that directly.

In order to fake mode switches, the LD_PRELOAD library hooks a bunch of X calls, which simply pretend that they worked. It also drops certain "fullscreen" input events so that steamcompmgr does not get confused. It uses the "zoom lock" in the XF86VidMode extension to prevent applications that somehow evade the LD_PRELOAD from causing mode switches.

The application output is scaled as needed in the compositor to make it fill the screen. The overlay is not scaled, however, so it will still be high-resolution even if the game itself has been scaled.

The cursor may also need scaling. To do that, the X server cursor is hidden and the compositor fetches the cursor image from the server. It then paints a scaled version into the display. Cursor position scaling also has to be done. The position will get "warped" if it ever goes outside the display boundaries. In addition, applications are prevented from grabbing the cursor themselves.

Future

There are two separate features that are likely to lead to future work in X. Handling the cursor scaling in the server, rather than having to do it in the compositor is one. Another is handling applications that are larger than the screen. Packard presented several possibilities for supporting these features in X.

"Hack number one" would be to add a "mouse proxy window" that would redirect and scale mouse events from one window to another. That would work for both windowed and full-screen mode, and have no impact on the output path, but it might be hard to position menus and other pop-up windows correctly.

Another option would be to run the game within a nested X server instance. It would be easy to implement with Xephyr, he said, and not require any server changes. But inter-client communication (to, say, cut and paste between client applications) would be hard to do and there is some potential for added latency on input and output.

Finally, there is an RandR-based screen scaling option. It has several advantages, including that there would be no issues with window and input positions, but there are some problems as well. For one thing, the code to do the output scaling would need to be changed to allow the compositing manager to do it, rather than being done only by the X server, as it is today. Also, the existing code does an unnecessary mode switch even when the mode doesn't need to change.

In answer to a question, Packard said that Valve's code has been released, but not proposed for upstream inclusion. "Their requirements are useful", but not the implementation.

Packard wrapped up by noting that SteamOS shows "how easy it was" to take an existing Linux desktop, add a bit of code, and turn it into a living-room, console experience. We will have to wait and see how successful it is in the market, however.

[I would like to thank the Linux Foundation for travel assistance to Chicago for LinuxCon North America.]