LFCS 2012: X and Wayland

Keith Packard has been working on the X window system since the early days, but more recently has been doing lots of work to enable its replacement. X has long held the position as the way that graphics is done on Linux (and other Unix) systems, but that is changing. He came to the Linux Foundation Collaboration Summit, which was held April 3-5 in San Francisco, to talk about the Wayland protocol and the Weston server, and how they could interoperate with X. Wayland looks to be an interesting change for desktop graphics on Linux.

Wayland is an alternative window system, and is part of long-term effort to integrate more modern technology into the Linux desktop. But there are lots of existing X applications out there that need a migration path so that they don't have to be rewritten. That's part of the plan too, Packard said.

Supporting X in Wayland

The current architecture of X has it rendering via the kernel and getting input from the kernel. The X server handles the geometry of the screen. Packard said that X isn't doing anything other than multiplexing output to the screen, while demultiplexing input. These days, X is generally run with a compositing window manager, so applications create graphics that get sent to the X server, which sends it to the window manager that does the OpenGL rendering, before handing it back to the X server, which then gives it to the kernel to actually get it on the screen.

That is a complicated path that can be simplified by merging the compositor into the window manager. That gets rid of the extra compositor process, which simplifies things considerably. For Wayland, the Weston server is the merger of the two separate pieces in X.

From an application's perspective, there aren't that many changes. It still talks to the server for geometry handling and still talks to the kernel or server to create its graphics. But the protocol is different, which will save a bunch of time, system complexity, and memory, Packard said.

Some applications will still need to talk X, however. One way to do that would be to make Wayland talk X too, but that would violate one of the goals of the project, which is to be lightweight. Weston is currently around 10,000 lines of code, so gluing in "half a million lines of X code" is not the right approach.

Instead, a new xwayland component was created. The X server will still be running, and applications will talk to it as they do now, but that server will talk to the xwayland hardware backend that will in turn talk to Weston. This is similar to what is done to support X on Mac OS X and Windows, which normally "sucks for performance", but the Wayland developers have come up with a way around that problem.

An application can do OpenGL or VAAPI (Video Acceleration API) rendering via the kernel in a buffer that is owned by the X server. The server passes that buffer to Weston via xwayland, but it is just a handle that is passed, not the actual data. The handles are just "tiny little names", he said, so there is very little bandwidth required to transfer them. It does require an extra context switch, but shouldn't add any latency compared to existing X.

X vs. Wayland

There are really only a few differences between X and Wayland; Packard was only able to come up with three. First, X has an external compositor, while Wayland's is internal. X deliberately put the compositor into a separate process because people wanted to experiment with compositing. In addition, the X server generally runs as root, so pushing the compositing code out of the server itself was safer. It has been a successful experiment overall, but it does add some latency and requires that identical state information is held by both the X server and compositor. Both have all of the information about all of the windows on the screen, and the compositor has to mirror the X server data structures and do incremental updates to those based on damage events.

Second, X has external window management, and Wayland internalizes that. The reason X did that originally was because of a lack of shared libraries in 1987 when window management was added. In order to share common user interface (UI) elements between applications, they had to be centralized in an external window manager. One nice bonus of that decision is that an application's window can still be managed (e.g. minimized) even if the application itself locks up. That is a side effect of having a common UI for all the applications on the screen.

Lastly, X applications do not paint their window decorations, while Wayland applications will need to. This is why a Qt application today in a Gtk environment gets Gtk window decorations. In Wayland, clients will need to do their own decorations, but the toolkits (e.g. Qt, Gtk) already know how to do so. Toolkits have been using window manager hints to get the window manager to decorate the way they want for years, now the toolkits will just do it themselves.

We own the stack

One nice benefit of working on free software is that "we own the whole stack", so whatever changes are needed can just be made. The last seven years have been spent moving things out of the X server and into the kernel. Things like PCI mappings, mode setting, rendering, and more are all in the kernel now so they can be shared by all window systems and applications. In addition, any window system can use OpenGL just by using the libraries that are available. The same OpenGL library can be used by both X and Wayland.

Several large barriers to entry for new window systems have been removed. OpenGL support is just one of those. The VAAPI code is "window system agnostic" now as well. Mode setting, which was a "huge barrier to entry" is now available to any window system because it lives in the kernel. Similarly, Wayland can just use the kernel execution management code to get accelerated graphics rendering. In addition, because they had the ability to fix both X and Wayland, they could make performance improvements like using handles instead of passing buffers around.

The performance of X typically becomes "abysmal" when it runs on top of another window system like Windows or Mac OS X because the server has to do a lot of bulk memory copies. But, because Wayland is cooperating with X, you get "full speed 2D rendering" while direct rendering is unchanged. Buffer swaps will happen every 16ms, when the X server tells xwayland it has new content and Weston tells the kernel to display it. Putting X atop Wayland will actually "reduce the context switches to once every 16ms", which may make it perform better than native X, Packard said. He doesn't have any numbers, and the effect will likely be imperceptible, but it could be faster. From the audience, Greg Kroah-Hartman also noted that it may result in power savings.

"It's not all ponies and rainbows", he said, as there are still some problems that need to be addressed. One is that X atop Wayland will still require an X window manager. The current plan is to have a Wayland-specific window manager that will translate from the X domain to the Wayland domain. It will take size hints, for example, and talk to Weston using the Wayland protocol to handle the hints. It will also have to provide window decorations, which is an "opportunity for more confusion on the Linux desktop", Packard said. Because desktop environments have enforced a particular style of decorations via their window manager, all applications on the desktop had the same decorations regardless of the toolkit used, but that will no longer be true when each toolkit does its own decorations.

It took around 50 patches to the X server to make it work with Wayland, which is "much much easier" than it was to get X working on Windows, he said. The patches were rebased on the 1.12 server code and mostly cover changing the input handling so that it comes from Weston. X video drivers also had to change, once again mostly to defer things like mode setting and acquisition of the DRM master to Weston. In addition, some "ugly hacks" for window moving and resizing were replaced by the X/Wayland window manager.

Window management

The window manager that is part of Weston needs client-side window decorations so that they can be painted into the same buffer as the rest of the application's output. That way, scaling or rotating the output will properly handle the window decorations as well. In addition, it simplifies Weston by not requiring a lot of UI code to handle the decorations, and the applications (or toolkits) can respond appropriately to events from the decorations.

For example, resizing windows in X is currently "messy" and that causes "ugly output". But that will be fixed with Wayland. In addition, with client-side decorations, those decorations no longer need to be rectangular, so applications are free to create decorations of their own size and shape. There are some acceleration features in Weston so that applications can tell it to move a window following the mouse, or indicate a place for users to click to minimize a stuck application.

The X/Wayland window manager will have to handle decorations as well as dealing with ICCCM (inter-client communications conventions manual) and EWMH (extended window manager hints). That means it will need to "implement all the bugs in X", but it stops there, he said, and the X/Wayland window manager will talk "nice stuff" to Wayland. For example, there is a huge amount of ICCCM that deals with installing and managing color maps that no one uses anymore, which will be handled directly by the window manager.

X was created before there was MIME or Unicode, so there are many pages expended in the X specifications to do things that are more easily handled with MIME types and UTF-8 these days. For cut-and-paste and drag-and-drop, Wayland uses MIME-labeled UTF-8 encoded objects. For client-to-client data transfer, one client just hands a file descriptor to Weston which hands it to the other client. That eliminates another ten pages of ICCCM which "reimplemented TCP on top of X properties".

The X server could be started automatically at session initialization time, but that would slow down session startup and add extra memory overhead. It could be started manually by the user instead, but a better way is for Weston to listen on the X socket. When a client connects, Weston will launch X and pass the sockets to the X server. When the last X client closes, the X server can then exit. Kroah-Hartman asked about using systemd to start X, but Packard said that Lennart Poettering indicated that systemd cannot start X, though he's not sure why that is. In any case, it is only 20 lines of code in Weston.

Remaining issues

There are, of course, some remaining issues to be worked on. For one thing, Wayland input handling is still in flux. There are questions surrounding keyboard support, as well as lots of code needed to support touchscreens and touch pads. There has been a lot of work on gestures and global system-wide touch support that will come to the Wayland environment with Xinput 2.2 support.

Another area that needs to be worked on is remote Wayland applications. The X community has had this capability for a long time and would like to be able to continue to use it. Supporting remote Wayland applications is not fundamentally different from local applications, he said. Applications will still create a new image and deliver it to the window server. In the local case, though, the pixels don't need to be transferred, but over the network, there is no shared memory between application and server.

There are "lots of nice network image transports", Packard said, that could be used to transfer the image data. That data could be compressed, perhaps even using a lossy compression, then sent to the server. Both sides could use any available hardware assistance (e.g. MPEG encode/decode), which would help with performance. Wayland avoids one of the major sources of latency in X, which is round-trips, so remote Wayland may actually have better performance than remote X. There are still lots of things to try, he said, but that's the current plan of attack.

The xwayland X server backend is working today. It redirects windows correctly and is started automatically. The synthetic keyboard and mouse that use the current Wayland keyboard handling is also working. The Intel video drivers bypass mode setting to work in Wayland environments. The X/Wayland window manager, cut-and-paste and drag-and-drop, as well as Xinput 2.2 support (for things like scroll wheels and touchscreens) are all things that he listed as still needing to be done. But it is clear that we are well on our way to having Wayland-based desktops that can still support the vast array of legacy X applications. It will be an interesting transition to watch.