LWN.net Weekly Edition for July 29, 2004

The 2004 Ottawa Linux Symposium

The 2004 Ottawa Linux Symposium is now history. OLS has, once again, proved itself to be one of the leading development conferences worldwide. By many accounts, this was one of the best years yet. This view is supported by the standing ovation given to conference organizer Andrew Hutton at the end.

This year, OLS expanded to fill a larger part of the Ottawa Conference Center, to good effect; the extra space was appreciated by all. AMD sponsored an opening reception featuring a talk by author Jim Munroe; unfortunately, in your editor's opinion, Mr. Munroe's speeches are not as interesting as his books. He characterized Linux as being a response to Microsoft, rather than something valuable in its own right, and he thought that his audience, full of IBM, SGI, AMD, Intel, Novell, etc. employees, would be interested in a lecture on the evils of corporate power. The closing reception, held at the newly-expanded Black Thorn, was as successful as ever - to the obvious regret of the crowd of hungover developers on the airport shuttle the next morning.

Kernel developer Andrew Morton was this year's keynote speaker. He called for unifying the kernels shipped by the distributions, and asked that distributors work toward getting their patches into the mainline quickly. He acknowledged that some distributors see kernel enhancements as part of the value they can add, but asked those distributors to find some other way to provide value to their customers. Fragmenting the kernel may be within the rights granted by the license, he said, but he sees it as being bad for the long-term future of Linux. He warned these distributors that he would actively work to undermine that strategy.

Andrew spent much of his time on the advantages of having a community-run platform upon which to build products, noting that "system software" is often where free software is most successful. He urged developers working on kernel code - drivers or new features - to get their code into the kernel early, so that it can benefit from the community process. He also acknowledged the community's debt to Richard Stallman.

Andrew finished by noting that, while Linux tends to enter companies from the bottom, it does not stay there. And neither do the people who brought in Linux in the first place. Some of them eventually get promoted into management, which helps the process along. "World domination," says Andrew, "is proceeding according to plan."

Along those lines, it is worth noting that the mix of attendees was a little different this year. The core of developers which defines OLS was as strong as ever, but, on the edges, one could see a fair number of management types, representatives from technical companies worldwide, and members of the press. The visibility of this conference, in other words, is growing beyond the developer community that it serves.

(LWN's coverage from a few OLS talks can be found below. The slides from LWN editor Jonathan Corbet's talk (on what to expect from kernel development in the next year) are available for the curious.)

Comments (none posted)

X at OLS

Two sessions at the Ottawa Linux Symposium hosted by Keith Packard and Jim Gettys combined to give a good overview of where the X window system is going. This article is an attempt to pull together the material covered in those talks. X, which, in Jim's words, "just sat there for years with nothing interesting happening," is now the focus of a flurry of activity. Expect to see interesting things happening over the next year or two - especially if a few more developers can be brought into the fold.

Keith Packard's talk had to do with separating the X server from the hardware. X currently does all kinds of interesting things, from direct programming of video modes through to remapping the PCI bus (to get around interesting BIOS behavior), scary DMA operations from user mode, direct interpretation of mouse events, and no end of user-space busy-waiting. There are all kinds of unfortunate side effects from this behavior; remapping the PCI bus and attempting DMA from user mode can bring down the whole system without much trouble. X's busy-waiting behavior leads, among other things, to poor interaction with the scheduler, which sees X as a CPU hog process whose priority should be lowered. Things would improve greatly if X did not get its hands quite so dirty with the hardware.

Interestingly, Keith blames himself for much of the current situation. Very early X11 implementations dealt with the operating system for their frame buffer access. Then Sun came out with a closed video adaptor which required some user-space tweaking; Keith duly hacked this up. More closed hardware came out, making the problem worse. Things led to the current situation, where X knows all about the hardware, deals with it at a very low level, and occasionally lets things go wrong in spectacular ways.

Things are, says Keith, starting to get better. The direct rendering infrastructure (DRI) is "a glimmer of sanity"; it returns DMA and interrupt handling to the kernel. The kernel frame buffer subsystem has helped to move hardware detection and mode selection back into the kernel. And /dev/input has gotten X out of the business of directly interpreting mouse sequences and trying to figure out which protocol is in use; the input subsystem also allows marking input events with proper, accurate timestamps, which is important to many applications. The input subsystem is still not perfect, however, especially on multi-head systems: there is no easy way to associate input devices with specific displays.

The process of separating X from the hardware needs to continue, however. X is still often in the business of configuring video modes, which is a complex and unforgiving task. "Everybody just hopes that X gets it right." This task should be moved out of X, and into some appropriate combination of kernel and user-space code. Keith's vision goes beyond mode selection, though; he would eventually like to see the X server using GL to speak with the hardware. Turning X into a GL client would make proper 3D support easier, would further separate X from the hardware, and would give hardware vendors a single set of drivers to write.

A huge remaining issue with the hardware side of X is hotplug monitors. The assumption that the available screens will never change is wired very deeply into the X server, and will not be easy to root out. But this work must be done; people are increasingly wanting to, for example, plug their laptop into different projectors from one day to the next, and it would be nice if it actually worked.

Jim Gettys's talk covered just about every aspect of the window system - and most of those are currently in flux. The fork in X development, which saw almost all of the XFree86 developers moving over to X.org, has revitalized development in a number of ways. Many longstanding problems are being addressed, and lots of new ideas are being kicked around.

Jim started with a list of current issues with X:

• The X font architecture, as originally designed, was terminally broken.

• The internal 2D graphics engine was poor; the plan from the beginning had been to replace it quickly. Once again, we see how "temporary" code can stay with us for decades.

• For various reasons, screen magnifiers and other accessibility aids are very hard to implement.

• X needs better "eye candy" support: translucent graphics, drop shadows, etc. Competing window systems (OS X, Longhorn) are raising the bar in this area, and X needs to stay up with them - if not ahead.

• Three-dimensional graphics are increasingly commonplace, and must be properly supported.

• Shared use of X (shared projector walls were an example that came up repeatedly) is poorly supported currently.

Anybody who has worked with X in any depth knows that its font mechanism has not aged well. What is happening now is that fonts are, increasingly, the responsibility of the client; the centralized font engine model has failed. This is, says Jim, the biggest single architectural change happening with X. Putting font support in the clients allows quicker distribution of new technologies; Jim notes that there have been five generations of font formats so far during the lifetime of X, and there is no reason to expect things to slow down. Any change which requires X server support cannot expect to see widespread deployment for at least two years; client-side changes can propagate much more quickly.

One of the reasons for putting font support in the X server initially was to keep bandwidth usage down. As it turns out, moving font support into the clients has left bandwidth usage almost unchanged, but has eliminated about 1/4 of the round trips to the server during application startup. Anybody who has ever run an X application over a high-latency link knows that those round trips hurt; eliminating them can only be a good thing.

Jim noted some lessons which come from the font experience. One is that downward compatibility is absolutely necessary. New technologies (such as newer font schemes) only became accepted when libraries were provided to make them work with older servers. No developer is interested in adopting technologies which restrict the distribution of their applications. It is also necessary to solve the full problem; in the case of fonts, that means solving the problem for printing as well. With that in place, "even OpenOffice" is moving over to the new font code.

Next point: eye candy is important. To a great extent, "eye candy" means translucent windows in one form or another. The real use for translucence is likely to be in window managers; once the new capabilities become available, Jim expects yet another round of window manager experimentation. "Another thousand flowers will bloom; many of them will stink." But what survives will enrich the graphical experience for all of us.

Keith took the stage for one of his famous demonstrations. His desktop is looking better all the time, with menus fading in and out and extensive (some might say excessive) use of translucence everywhere. He pointed out that translucence is a pixel-by-pixel property. He runs an xterm with a translucent background, but the text is opaque, and casts solid shadows. High bandwidth applications, such as full-motion video, run in a translucent mode with no apparent performance problems - on a fast laptop, anyway.

"Eye candy" also means tools like screen magnifiers (which keep up in real time) for people who have a hard time reading the screen's contents. "Thumbnailers" (such as those often used for virtual desktops) will also benefit from this work. Animation needs better support, and X needs better integration into three-dimensional environments. "We live in a 3D world, perhaps our work environments should be 3D as well." Eliminating "flashing" (making desktop updates look nice) is also an important goal.

All of this is happening, in classic computer science manner, through the addition of another level of indirection. X applications have typically rendered their graphics directly into the frame buffer, unless they made explicit use of off-screen pixmaps. In the new world, all rendering will happen into off-screen memory. A new "compositing manager," which typically works in conjunction with the window manager, renders the off-screen windows onto the display, perhaps transforming them along the way. It is this process which, for example, causes a newly maximized window to fade, fully rendered, onto the screen, rather than just flashing up blank and being redrawn in place. The policy for when and how such effects should be used lives entirely within the compositing manager.

To accomplish all of this, four new extensions have been added to the X protocol. "XFixes" is a sort of repository for minor repairs for things that were done wrong in the original protocol. Among other things, XFixes promotes Regions to a first-class X protocol object. The "Damage" extension allows X clients to be notified when the contents of windows changes; it can be used by the compositing manager, screen magnifiers, thumbnailers, and any other process which needs to know when things happen. Interestingly, the Damage extension actually shrinks the size of the X server; a sign, according to Jim, of something that should have been done that way from the beginning. The "Composite" extension handles the off-screen rendering of windows and the interactions with the compositing manager. And "Xevie" allows the interception of input events; it is used both for compositing (where event locations may have to be transformed on their way to the client) and for accessibility functions.

The latency issue is on the X developers' minds. X client startup can be slow, especially on high-latency links. This slowness is caused by the large number of round trips required between the client and the server during initialization. Client-side fonts eliminated 25% of those, but, for a typical client, some 130 round trips still need to happen before any actual work can get done. Jim believes that 90% of those can be eliminated; some will be harder than others. One of the biggest remaining offenders, it turns out, is X extensions; apparently an "Extension extension" may be required to batch together extension information.

In the 2D world, the Cairo library, by Carl Worth and Keith Packard, is starting to come into use. This engine, says Jim, brings "world-class, two-dimensional, anti-aliased graphics" to the free software world; it is second to none. Along with top-quality graphics, Cairo turns printing into a first-class citizen; the audience reacted to this statement with loud applause. There is work left to be done with Cairo, including the addition of more backends and, inevitably, optimization.

Security is another issue. Currently, the only rational way to use X remotely is through an ssh tunnel, but that has its costs: extra processes in the loop, more latency, etc. X clients need to be able to talk directly with the server in a secure manner. Jim notes that X terminals are coming back. Maintenance of desktop clients is costly, and the deployment of simple X terminals can reduce those costs. But it has to be possible to use them in a secure manner.

The other interesting security issue is access to shared resources. It would be nice to be able to walk up to a projector wall with a laptop and start making use of it. Handling of input devices also figures into this equation; how does one connect a mouse to a wall screen? If this problem could be solved, there is the potential to "explode the PC into its component pieces" and create an interesting, dynamic world. There are no security policy options in X for this kind of shared use, however. Jim briefly mentioned some ideas involving integration with SELinux as a way of addressing this area.

A related topic is mobility. Wouldn't it be nice, asks Jim, if you could leave the office, go home, and pick up with your applications on your home system? This is a hard problem, but it is being addressed; there is prototype migration code in GTK now. There are security issues to deal with, of course, along with little things like teaching Xlib to deal gracefully with connection loss.

Other issues mentioned in passing include proper network audio support ("all of the current audio servers are lame; we need one good one"), better color management (for things like printers too), and proper support for compression in the X protocol. It seems that the bulk of X traffic, anymore, is image data, and much of that data compresses very well.

Finally, there are the packaging and development issues. Work to split the X distribution into its component pieces (server, libraries, applications, fonts) continues; there will eventually be no "tarball from hell" to deal with. Modern building tools are being adopted; imake is on its way out. Much of this work is done, but some pieces remain.

On the community side, Jim notes that X.org is a very different place from what it was a year ago. Most of the XFree86 developers have come over, and it is clearly the focal point for X development.

Perhaps the most important part of Jim's talk was repeated several times: the X developers have ambitious aims and a great deal of work that needs to be done; they are actively looking for developers to take on pieces of it. For anybody out there who is looking for a project to support, X.org represents a major opportunity. X is the core, the kernel, as it were, of the Linux graphical environment. The development effort is healthy again, and it has no end of interesting and challenging tasks to address. It is hard to imagine a better, more interesting, more visible, or more important project to work with. If you are looking to do some hacking, X would be an ideal place to do it.

Comments (57 posted)

OLS: Enforcing the GPL

Harald Welte, the current netfilter maintainer, ran a session on dealing with GPL violations. Harald has made a name for himself over the last year by reaching settlements with several manufacturers who were shipping products containing the netfilter code without complying with its licensing requirements. In most cases, these settlements have been relatively easy to reach, with the offending companies releasing their code and, in many cases, making a donation to the FSF or a related worthy cause. Harald is also the force behind the injunction in the Sitecom case, which just received another court ruling that the GPL is valid and enforceable in Germany.

Harald's approach differs from that of the Free Software Foundation. The FSF tries to resolve GPL violations in the most quiet, friendly way possible; in most cases, the wider world never even hears that there is a problem. Harald found himself frustrated with this way of doing things; it takes far too long (perhaps longer than the lifetime of the offending product) and gives no real disincentive for companies considering ignoring the GPL. So he took matters into his own hands; by bringing in lawyers early, making violations public, and threatening immediate damage to the offenders' bottom line, he has gotten some real results.

Some interesting features of the German legal system have worked in Harald's favor in this campaign. By (his lawyers' interpretation of) German law, simple use of the netfilter/iptables internal API is a strong indication of a derivative work. Since it is not a "standard" API, there is no copyright boundary there; by this same reasoning, any binary-only kernel module is a GPL violation. This interpretation of the GPL also rules out putting GPL-licensed code onto hardware which only runs signed binaries - unless the key is distributed with the source.

German law requires that any request for injunctive relief be filed within four weeks of the discovery of the infringement. This is an inflexible, externally-imposed deadline which forces companies to move quickly to resolve the issue. When a company has received a cease-and-desist notice, it knows that it cannot drag it issue out over time; it will, instead, find itself in court in short order.

Harald was asked about the completeness of the code releases he has won so far; apparently not all vendors have released kernels which actually can be rebuilt into a working image - or which even compile. His response is that his ability to compel code releases really only extends as far as his copyrights in the netfilter code. If a vendor does not release a full kernel, copyright holders in other parts of the system will have to get involved.

The last part of the talk covered things developers can do to help make copyright enforcement easier. They include:

• Don't fix spelling errors and typos, and leave strange messages (example: "Rusty needs more caffeine") in place. The presence of this sort of text in a binary image is an obvious sign of copying. The removal of this text, instead, would be a clear sign of a willful violation, which raises the stakes considerably.

• The copyright message needs to be in the binary as well.

• It is important to track the names of all contributors to the code, so they can be found for enforcement actions or the distribution of any sort of damages.

• When a violation is found, it should not be made public at the time, as that can make getting injunctive relief harder. Instead, a lawyer should be brought in to prepare and send the "cease and desist" note.

Expect to hear about more resolved violations in the near future. Harald indicated that a couple of settlements have been reached, but the companies involved have been given a grace period to prepare for the fallout once the situation has been made public. Once that period has passed, the press releases will go out.

Comments (35 posted)

OLS: An introduction to Conary

Specifix, a company founded by a number of early Red Hat developers, recently came out of hiding. At the 2004 Ottawa Linux Symposium, Eric Troan gave a presentation on Conary, the company's system for package, repository, and distribution management. It was a technical talk from the beginning to the end; Eric would not talk about Specifix's business model even when asked (though he offered to do so in private). If nothing else, he understands what the OLS crowd is looking to hear.

Package management systems have come into use in almost every distribution out there. They are a clear step up from what came before, but, as Eric pointed out, significant problems have been building for years. These include:

• Repositories are an afterthought. A typical repository is a simple collection of files in whatever package format is being used, perhaps with a bit of metadata.

• The version scheme used by most package managers follows a straight line model; there is no provision for branches. That makes it hard, for example, to determine which version of a package is appropriate for a specific release of a given distribution.

• Packages contain scripts which handle parts of the installation and removal process which go beyond the simple management of files. These scripts tend to contain a lot of boilerplate, and are replicated in every package file. Bugs, too, are replicated, and there is no one place to go to fix them. The scripts are also not portable across distributions (even those using the same package format) and cannot be customized for an individual site's needs.

Conary was developed as a way of addressing the above limitations and to make it possible for users to create their own, customized distributions in an easy manner. In the simplest sense, one can think of Conary as a package management system with a more consistent view of objects from the repository level down to individual files, combined with a version management scheme.

Conary treats files as "first class objects," which are managed by the framework as a whole. Files have a unique ID and a version history; they also have a set of attributes. One of those attributes is the file's location in the filesystem; moving a file is a simple matter of changing that attribute.

A "trove" is a container holding one or more files and other troves. Files are contained by reference. A "component" is a collection of files, by reference. Example components listed by Eric for the bzip2 package might be bzip2:runtime (binary files to run the program), bzip2:lib, bzip2:doc, and, of course, bzip2:source. Components can be aggregated together into packages. Both components and packages are considered to be "troves," for what it's worth.

Version strings are hung onto everything; Specifix has added some complexity to the versioning system, though. Each version string includes the repository name, a namespace (think of it as a distribution name), a branch name (for the creation of trees in the version space), the upstream package version, and a two-part local revision number. Needless to say, the version strings get long, but the system hides the full string most of the time. Creating versions in this way allows the system to easily determine which version of a package is the newest, which version of which distribution is built for, and so on.

Branching is done by adding a branch name to the version string. Branching allows the tracking of versions of packages which were shipped with a specific distribution, along with updates to those packages. There is also a special type of branch called a "shadow" which tracks changes to the trunk it was branched from. Essentially, the shadow is automatically merged with each new version of the trunk it is following. This feature would be useful for somebody maintaining a derivative distribution; they want to keep up with what the source distribution is doing without losing track of their own changes. The only problem with shadows is that, like a number of other Conary features, they are not actually implemented yet.

"Flavors" are another Conary feature; they seem to be patterned after Gentoo's "USE flags." A flavor is a set of configuration options describing how all packages are to be built. This feature is used for multiple architecture support, or for building versions of distributions with different feature sets (e.g. creating a distribution without PAM support). Multiple flavors of a package can be installed on a system if they don't conflict with each other; this allows, for example, the installation of 32-bit libraries on x86-64 systems.

Then, there is the concept of "changesets." A changeset is a collection of modifications to files (including attribute changes) and the troves which contain them. A changeset is, essentially, a patch to a package or a distribution. Changesets, which track only changes, can be much smaller than the packages they describe, and can thus be an efficient way of distributing updates. Changesets describe changes to configuration files in diff format, which often allows them to be merged automatically with local changes. A system administrator can also create a changeset describing his or her local changes to the system; that changeset can then be used for merging with updates, or replicating the system elsewhere. Local changesets can also be used for version control and the tracking of system changes.

"Tags" are Conary's answer to the package script problem (and, also, to the complex set of interactions represented by the RPM "trigger" mechanism). A tag is a file attribute describing the type of the file, be it "shared library," "info file," or any of a long list of alternatives. Most files can be tagged automatically by Conary. Tags have scripts associated with them; there is, for example, a script which handles the installation of an info file and updating the relevant directory. These scripts are distributed separately; there is only one copy of them on the system. The scripts are thus easily fixed when bugs turn up, and they can be customized by the local administrator if need be. Separating out the management scripts in this way should also make it easier to install packages from other distributions.

A "fileset" is an arbitrary collection of files built from components in the repository. Filesets seem to be intended to help in the creation of small system images for embedded systems; they allow an easy picking and choosing of an exact set of desired files. "Groups" are, instead, the analog of the Debian "task" or Anaconda "component." They allow the management of several packages as a unit, but they come with their own local changesets so that local changes to the group are tracked properly.

The paper from the OLS proceedings (PDF format) is worthwhile reading for anybody wanting more details on how Conary works.

Interested parties can download an early Conary release from the Specifix web site. Be warned, however, that a few features are still missing; they include shadows, dependencies (an important issue that they "think" they know how to implement), flavors, package signatures, and more. "Release early" is an important part of the free software development process, however, and the Specifix founders understand that process well. Conary's vaporware features will, beyond doubt, be filled in soon. As that happens, expect interest in this tool to increase; it truly does have the potential to change the way we set up and manage our projects, distributions, and systems.

Comments (24 posted)

Page editor: Rebecca Sobol

Inside this week's LWN.net Weekly Edition

• Security: The value of privacy policies; New vulnerabilities in courier, samba, sox, subversion, ...
• Kernel: Another look at the new development process; Voluntary preemption and interrupt handlers; The kernel events layer.
• Distributions: A look at Progeny Debian Beta; Xandros Desktop 2.5; Catux-USB, PHP Solutions Live
• Development: gputils, the GNU PIC Utilities, new versions of Speex, Samba, GNOME, KE, CWirc, Liferea, Gnumeric, Furthur, Devhelp, GCL, SBCL, Ponie, Durus, Vex, DrPython.
• Press: Choosing an open source license, OLS and OSCON coverage, Asianux 1.0, NASA's new Linux cluster.
• Announcements: $149 xterm, OSDN becomes OSTG, Eclipse launches Web Tools, The Perl Review, Free Software Printing Summit summary, aKademy Lineup, ApacheCon CFP, FSF Seminars on GPL.

Next page: Security>>