X at OLS

Keith Packard's talk had to do with separating the X server from the hardware. X currently does all kinds of interesting things, from direct programming of video modes through to remapping the PCI bus (to get around interesting BIOS behavior), scary DMA operations from user mode, direct interpretation of mouse events, and no end of user-space busy-waiting. There are all kinds of unfortunate side effects from this behavior; remapping the PCI bus and attempting DMA from user mode can bring down the whole system without much trouble. X's busy-waiting behavior leads, among other things, to poor interaction with the scheduler, which sees X as a CPU hog process whose priority should be lowered. Things would improve greatly if X did not get its hands quite so dirty with the hardware.

Interestingly, Keith blames himself for much of the current situation. Very early X11 implementations dealt with the operating system for their frame buffer access. Then Sun came out with a closed video adaptor which required some user-space tweaking; Keith duly hacked this up. More closed hardware came out, making the problem worse. Things led to the current situation, where X knows all about the hardware, deals with it at a very low level, and occasionally lets things go wrong in spectacular ways.

Things are, says Keith, starting to get better. The direct rendering infrastructure (DRI) is "a glimmer of sanity"; it returns DMA and interrupt handling to the kernel. The kernel frame buffer subsystem has helped to move hardware detection and mode selection back into the kernel. And /dev/input has gotten X out of the business of directly interpreting mouse sequences and trying to figure out which protocol is in use; the input subsystem also allows marking input events with proper, accurate timestamps, which is important to many applications. The input subsystem is still not perfect, however, especially on multi-head systems: there is no easy way to associate input devices with specific displays.

The process of separating X from the hardware needs to continue, however. X is still often in the business of configuring video modes, which is a complex and unforgiving task. "Everybody just hopes that X gets it right." This task should be moved out of X, and into some appropriate combination of kernel and user-space code. Keith's vision goes beyond mode selection, though; he would eventually like to see the X server using GL to speak with the hardware. Turning X into a GL client would make proper 3D support easier, would further separate X from the hardware, and would give hardware vendors a single set of drivers to write.

A huge remaining issue with the hardware side of X is hotplug monitors. The assumption that the available screens will never change is wired very deeply into the X server, and will not be easy to root out. But this work must be done; people are increasingly wanting to, for example, plug their laptop into different projectors from one day to the next, and it would be nice if it actually worked.

Jim Gettys's talk covered just about every aspect of the window system - and most of those are currently in flux. The fork in X development, which saw almost all of the XFree86 developers moving over to X.org, has revitalized development in a number of ways. Many longstanding problems are being addressed, and lots of new ideas are being kicked around.

Jim started with a list of current issues with X:

• The X font architecture, as originally designed, was terminally broken.

• The internal 2D graphics engine was poor; the plan from the beginning had been to replace it quickly. Once again, we see how "temporary" code can stay with us for decades.

• For various reasons, screen magnifiers and other accessibility aids are very hard to implement.

• X needs better "eye candy" support: translucent graphics, drop shadows, etc. Competing window systems (OS X, Longhorn) are raising the bar in this area, and X needs to stay up with them - if not ahead.

• Three-dimensional graphics are increasingly commonplace, and must be properly supported.

• Shared use of X (shared projector walls were an example that came up repeatedly) is poorly supported currently.

Anybody who has worked with X in any depth knows that its font mechanism has not aged well. What is happening now is that fonts are, increasingly, the responsibility of the client; the centralized font engine model has failed. This is, says Jim, the biggest single architectural change happening with X. Putting font support in the clients allows quicker distribution of new technologies; Jim notes that there have been five generations of font formats so far during the lifetime of X, and there is no reason to expect things to slow down. Any change which requires X server support cannot expect to see widespread deployment for at least two years; client-side changes can propagate much more quickly.

One of the reasons for putting font support in the X server initially was to keep bandwidth usage down. As it turns out, moving font support into the clients has left bandwidth usage almost unchanged, but has eliminated about 1/4 of the round trips to the server during application startup. Anybody who has ever run an X application over a high-latency link knows that those round trips hurt; eliminating them can only be a good thing.

Jim noted some lessons which come from the font experience. One is that downward compatibility is absolutely necessary. New technologies (such as newer font schemes) only became accepted when libraries were provided to make them work with older servers. No developer is interested in adopting technologies which restrict the distribution of their applications. It is also necessary to solve the full problem; in the case of fonts, that means solving the problem for printing as well. With that in place, "even OpenOffice" is moving over to the new font code.

Next point: eye candy is important. To a great extent, "eye candy" means translucent windows in one form or another. The real use for translucence is likely to be in window managers; once the new capabilities become available, Jim expects yet another round of window manager experimentation. "Another thousand flowers will bloom; many of them will stink." But what survives will enrich the graphical experience for all of us.

Keith took the stage for one of his famous demonstrations. His desktop is looking better all the time, with menus fading in and out and extensive (some might say excessive) use of translucence everywhere. He pointed out that translucence is a pixel-by-pixel property. He runs an xterm with a translucent background, but the text is opaque, and casts solid shadows. High bandwidth applications, such as full-motion video, run in a translucent mode with no apparent performance problems - on a fast laptop, anyway.

"Eye candy" also means tools like screen magnifiers (which keep up in real time) for people who have a hard time reading the screen's contents. "Thumbnailers" (such as those often used for virtual desktops) will also benefit from this work. Animation needs better support, and X needs better integration into three-dimensional environments. "We live in a 3D world, perhaps our work environments should be 3D as well." Eliminating "flashing" (making desktop updates look nice) is also an important goal.

All of this is happening, in classic computer science manner, through the addition of another level of indirection. X applications have typically rendered their graphics directly into the frame buffer, unless they made explicit use of off-screen pixmaps. In the new world, all rendering will happen into off-screen memory. A new "compositing manager," which typically works in conjunction with the window manager, renders the off-screen windows onto the display, perhaps transforming them along the way. It is this process which, for example, causes a newly maximized window to fade, fully rendered, onto the screen, rather than just flashing up blank and being redrawn in place. The policy for when and how such effects should be used lives entirely within the compositing manager.

To accomplish all of this, four new extensions have been added to the X protocol. "XFixes" is a sort of repository for minor repairs for things that were done wrong in the original protocol. Among other things, XFixes promotes Regions to a first-class X protocol object. The "Damage" extension allows X clients to be notified when the contents of windows changes; it can be used by the compositing manager, screen magnifiers, thumbnailers, and any other process which needs to know when things happen. Interestingly, the Damage extension actually shrinks the size of the X server; a sign, according to Jim, of something that should have been done that way from the beginning. The "Composite" extension handles the off-screen rendering of windows and the interactions with the compositing manager. And "Xevie" allows the interception of input events; it is used both for compositing (where event locations may have to be transformed on their way to the client) and for accessibility functions.

The latency issue is on the X developers' minds. X client startup can be slow, especially on high-latency links. This slowness is caused by the large number of round trips required between the client and the server during initialization. Client-side fonts eliminated 25% of those, but, for a typical client, some 130 round trips still need to happen before any actual work can get done. Jim believes that 90% of those can be eliminated; some will be harder than others. One of the biggest remaining offenders, it turns out, is X extensions; apparently an "Extension extension" may be required to batch together extension information.

In the 2D world, the Cairo library, by Carl Worth and Keith Packard, is starting to come into use. This engine, says Jim, brings "world-class, two-dimensional, anti-aliased graphics" to the free software world; it is second to none. Along with top-quality graphics, Cairo turns printing into a first-class citizen; the audience reacted to this statement with loud applause. There is work left to be done with Cairo, including the addition of more backends and, inevitably, optimization.

Security is another issue. Currently, the only rational way to use X remotely is through an ssh tunnel, but that has its costs: extra processes in the loop, more latency, etc. X clients need to be able to talk directly with the server in a secure manner. Jim notes that X terminals are coming back. Maintenance of desktop clients is costly, and the deployment of simple X terminals can reduce those costs. But it has to be possible to use them in a secure manner.

The other interesting security issue is access to shared resources. It would be nice to be able to walk up to a projector wall with a laptop and start making use of it. Handling of input devices also figures into this equation; how does one connect a mouse to a wall screen? If this problem could be solved, there is the potential to "explode the PC into its component pieces" and create an interesting, dynamic world. There are no security policy options in X for this kind of shared use, however. Jim briefly mentioned some ideas involving integration with SELinux as a way of addressing this area.

A related topic is mobility. Wouldn't it be nice, asks Jim, if you could leave the office, go home, and pick up with your applications on your home system? This is a hard problem, but it is being addressed; there is prototype migration code in GTK now. There are security issues to deal with, of course, along with little things like teaching Xlib to deal gracefully with connection loss.

Other issues mentioned in passing include proper network audio support ("all of the current audio servers are lame; we need one good one"), better color management (for things like printers too), and proper support for compression in the X protocol. It seems that the bulk of X traffic, anymore, is image data, and much of that data compresses very well.

Finally, there are the packaging and development issues. Work to split the X distribution into its component pieces (server, libraries, applications, fonts) continues; there will eventually be no "tarball from hell" to deal with. Modern building tools are being adopted; imake is on its way out. Much of this work is done, but some pieces remain.

On the community side, Jim notes that X.org is a very different place from what it was a year ago. Most of the XFree86 developers have come over, and it is clearly the focal point for X development.

Perhaps the most important part of Jim's talk was repeated several times: the X developers have ambitious aims and a great deal of work that needs to be done; they are actively looking for developers to take on pieces of it. For anybody out there who is looking for a project to support, X.org represents a major opportunity. X is the core, the kernel, as it were, of the Linux graphical environment. The development effort is healthy again, and it has no end of interesting and challenging tasks to address. It is hard to imagine a better, more interesting, more visible, or more important project to work with. If you are looking to do some hacking, X would be an ideal place to do it.