The 2004 Ottawa Linux
is now history. OLS has, once again, proved itself to be one
of the leading development conferences worldwide. By many accounts, this
was one of the best years yet. This view is supported by the standing
ovation given to conference organizer Andrew Hutton at the end.
This year, OLS expanded to fill a larger part of the Ottawa Conference
Center, to good effect; the extra space was appreciated by all. AMD
sponsored an opening reception featuring a talk by author Jim Munroe;
unfortunately, in your editor's opinion, Mr. Munroe's speeches are not as
interesting as his books. He characterized Linux as being a response to
Microsoft, rather than something valuable in its own right, and he thought
that his audience, full of IBM, SGI, AMD, Intel, Novell, etc. employees,
would be interested in a lecture on the evils of corporate power. The
closing reception, held at the newly-expanded Black Thorn, was as
successful as ever - to the obvious regret of the crowd of hungover developers
on the airport shuttle the next morning.
Kernel developer Andrew Morton was this year's keynote speaker. He called
for unifying the kernels shipped by the distributions, and asked that
distributors work toward getting their patches into the mainline quickly. He
acknowledged that some distributors see kernel enhancements as part of the
value they can add, but asked those distributors to find some other way to
provide value to their customers. Fragmenting the kernel may be within the
rights granted by the license, he said, but he sees it as being bad for the
long-term future of Linux. He warned these distributors that he would
actively work to undermine that strategy.
Andrew spent much of his time on the advantages of having a community-run
platform upon which to build products, noting that "system software" is
often where free software is most successful. He urged developers working
on kernel code - drivers or new features - to get their code into the
kernel early, so that it can benefit from the community process. He also
acknowledged the community's debt to Richard Stallman.
Andrew finished by noting that, while Linux tends to enter companies from
the bottom, it does not stay there. And neither do the people who brought
in Linux in the first place. Some of them eventually get promoted into
management, which helps the process along. "World domination," says
Andrew, "is proceeding according to plan."
Along those lines, it is worth noting that the mix of attendees was a
little different this year. The core of developers which defines OLS was
as strong as ever, but, on the edges, one could see a fair number of
management types, representatives from technical companies worldwide, and
members of the press. The visibility of this conference, in other words,
is growing beyond the developer community that it serves.
(LWN's coverage from a few OLS talks can be found below. The slides from LWN editor Jonathan Corbet's
talk (on what to expect from kernel development in the next year) are
available for the curious.)
Comments (none posted)
Two sessions at the Ottawa Linux Symposium hosted by Keith Packard and Jim
Gettys combined to give a good overview
of where the X window system is going. This article is an attempt to pull
together the material covered in those talks. X, which, in Jim's words,
"just sat there for years with
nothing interesting happening," is now the focus of a flurry of activity.
Expect to see interesting things happening over the next year or two -
especially if a few more developers can be brought into the fold.
Keith Packard's talk had to do with separating the X server from the
hardware. X currently does all kinds of interesting things, from direct
programming of video modes through to remapping the PCI bus (to get around
interesting BIOS behavior), scary DMA operations from user mode, direct
interpretation of mouse events, and no end of user-space busy-waiting.
There are all kinds of unfortunate side effects from this behavior;
remapping the PCI bus and attempting DMA from user mode can bring down the
whole system without much trouble. X's busy-waiting behavior leads, among
other things, to poor interaction with the scheduler, which sees X as a CPU
hog process whose priority should be lowered. Things would improve greatly
if X did not get its hands quite so dirty with the hardware.
Interestingly, Keith blames himself for much of the current situation.
Very early X11 implementations dealt with the operating system for their
frame buffer access. Then Sun came out with a closed video adaptor which
required some user-space tweaking; Keith duly hacked this up. More closed
hardware came out, making the problem worse. Things led to the current
situation, where X knows all about the hardware, deals with it at a very
low level, and occasionally lets things go wrong in spectacular ways.
Things are, says Keith, starting to get better. The direct rendering
infrastructure (DRI) is "a glimmer of sanity"; it returns
DMA and interrupt handling to the kernel. The kernel frame buffer
subsystem has helped to move hardware detection and mode selection back
into the kernel. And /dev/input has gotten X out of the business
of directly interpreting mouse sequences and trying to figure out which
protocol is in use; the input subsystem also allows marking input events
with proper, accurate timestamps, which is important to many applications.
The input subsystem is still not perfect, however, especially on multi-head
systems: there is no easy way to associate input devices with specific
The process of separating X from the hardware needs to continue, however.
X is still often in the business of configuring video modes, which is a
complex and unforgiving task. "Everybody just hopes that X gets it right."
This task should be moved out of X, and into some appropriate combination
of kernel and user-space code.
Keith's vision goes beyond mode selection, though; he would eventually like
to see the X server using GL to speak with the hardware. Turning X into a
GL client would make proper 3D support easier, would further separate X
from the hardware, and would give hardware vendors a single set of drivers
A huge remaining issue with the hardware side of X is hotplug monitors.
The assumption that the available screens will never change is wired very
deeply into the X server, and will not be easy to root out. But this work
must be done; people are increasingly wanting to, for example, plug their
laptop into different projectors from one day to the next, and it would be nice if it
Jim Gettys's talk covered just about every aspect of the window system -
and most of those are currently in flux. The fork in X development, which
saw almost all of the XFree86 developers moving over to X.org, has
revitalized development in a number of ways. Many longstanding problems
are being addressed, and lots of new ideas are being kicked around.
Jim started with a list of current issues with X:
- The X font architecture, as originally designed, was terminally
- The internal 2D graphics engine was poor; the plan from the beginning
had been to replace it quickly. Once again, we see how "temporary"
code can stay with us for decades.
- For various reasons, screen magnifiers and other accessibility aids
are very hard to implement.
- X needs better "eye candy" support: translucent graphics, drop
shadows, etc. Competing window systems (OS X, Longhorn) are
raising the bar in this area, and X needs to stay up with them - if
- Three-dimensional graphics are increasingly commonplace, and must be
- Shared use of X (shared projector walls were an example that came up
repeatedly) is poorly supported currently.
Anybody who has worked with X in any depth knows that its font mechanism
has not aged well. What is happening now is that fonts are, increasingly,
the responsibility of the client; the centralized font engine model has
failed. This is, says Jim, the biggest single architectural change
happening with X. Putting font support in the clients allows quicker
distribution of new technologies; Jim notes that there have been five
generations of font formats so far during the lifetime of X, and there is
no reason to expect things to slow down. Any change which requires X
server support cannot expect to see widespread deployment for at least two
years; client-side changes can propagate much more quickly.
One of the reasons for putting font support in the X server initially was
to keep bandwidth usage down. As it turns out, moving font support into
the clients has left bandwidth usage almost unchanged, but has eliminated
about 1/4 of the round trips to the server during application startup.
Anybody who has ever run an X application over a high-latency link knows
that those round trips hurt; eliminating them can only be a good thing.
Jim noted some lessons which come from the font experience. One is that
downward compatibility is absolutely necessary. New technologies (such as
newer font schemes) only
became accepted when libraries were provided to make them work with older
servers. No developer is interested in adopting technologies which
restrict the distribution of their applications. It is also necessary to
solve the full problem; in the case of fonts, that means solving the
problem for printing as well. With that in place, "even OpenOffice" is
moving over to the new font code.
Next point: eye candy is important. To a great extent, "eye candy" means
translucent windows in one form or another. The real use for translucence
is likely to be in window managers; once the new capabilities become
available, Jim expects yet another round of window manager
experimentation. "Another thousand flowers will bloom; many of them will
stink." But what survives will enrich the graphical experience for all of
Keith took the stage for one of his famous demonstrations. His desktop is
looking better all the time, with menus fading in and out and extensive
(some might say excessive) use of translucence everywhere. He pointed out
that translucence is a pixel-by-pixel property. He runs an xterm with a
translucent background, but the text is opaque, and casts solid shadows.
High bandwidth applications, such as full-motion video, run in a
translucent mode with no apparent performance problems - on a fast laptop,
"Eye candy" also means tools like screen magnifiers (which keep up in real
time) for people who have a hard time reading the screen's contents.
"Thumbnailers" (such as those often used for virtual desktops) will also
benefit from this work. Animation needs better support, and X needs better
integration into three-dimensional environments. "We live in a 3D world,
perhaps our work environments should be 3D as well." Eliminating
"flashing" (making desktop updates look nice) is also an important goal.
All of this is happening, in classic computer science manner, through the
addition of another level of indirection. X applications have typically
rendered their graphics directly into the frame buffer, unless they made
explicit use of off-screen pixmaps. In the new world, all rendering will
happen into off-screen memory. A new "compositing manager," which
typically works in conjunction with the window manager, renders the
off-screen windows onto the display, perhaps transforming them along the
way. It is this process which, for example, causes a newly maximized
window to fade, fully rendered, onto the screen, rather than just flashing
up blank and being redrawn in place. The policy for when and how such
effects should be used lives entirely within the compositing manager.
To accomplish all of this, four new extensions have been added to the X
protocol. "XFixes" is a sort of repository for minor repairs for things
that were done wrong in the original protocol. Among other things, XFixes
promotes Regions to a first-class X protocol object. The "Damage"
extension allows X
clients to be notified when the contents of windows changes; it can be used
by the compositing manager, screen magnifiers, thumbnailers, and any other
process which needs to know when things happen. Interestingly, the Damage
extension actually shrinks the size of the X server; a sign, according to
Jim, of something that should have been done that way from the beginning.
The "Composite" extension
handles the off-screen rendering of windows and the interactions with the
compositing manager. And "Xevie" allows the
interception of input events; it is used both for compositing (where event
locations may have to be transformed on their way to the client) and for
The latency issue is on the X developers' minds. X client startup can be
slow, especially on high-latency links. This slowness is caused by the
large number of round trips required between the client and the server
during initialization. Client-side fonts eliminated 25% of those, but, for
a typical client, some 130 round trips still need to happen before any
actual work can get done. Jim believes that 90% of those can be
eliminated; some will be harder than others. One of the biggest remaining
offenders, it turns out, is X extensions; apparently an "Extension
extension" may be required to batch together extension information.
In the 2D world, the Cairo
library, by Carl Worth and Keith Packard, is
starting to come into use. This engine, says Jim, brings "world-class,
two-dimensional, anti-aliased graphics" to the free software world; it is
second to none. Along with top-quality graphics, Cairo turns printing into
a first-class citizen; the audience reacted to this statement with loud
applause. There is work left to be done with Cairo, including the addition
of more backends and, inevitably, optimization.
Security is another issue. Currently, the only rational way to use X
remotely is through an ssh tunnel, but that has its costs: extra processes
in the loop, more latency, etc. X clients need to be able to talk directly
with the server in a secure manner. Jim notes that X terminals are coming
back. Maintenance of desktop clients is costly, and the deployment of
simple X terminals can reduce those costs. But it has to be possible to
use them in a secure manner.
The other interesting security issue is access to shared resources. It
would be nice to be able to walk up to a projector wall with a laptop and
start making use of it. Handling of input devices also figures into this
equation; how does one connect a mouse to a wall screen? If this problem
could be solved, there is the potential to "explode the PC into its
component pieces" and create an interesting, dynamic world.
There are no security policy options in X for this
kind of shared use, however. Jim briefly mentioned some ideas involving
integration with SELinux as a way of addressing this area.
A related topic is mobility. Wouldn't it be nice, asks Jim, if you could
leave the office, go home, and pick up with your applications on your home
system? This is a hard problem, but it is being addressed; there is
prototype migration code in GTK now. There are security issues to deal
with, of course, along with little things like teaching Xlib to deal
gracefully with connection loss.
Other issues mentioned in passing include proper network audio support
("all of the current audio servers are lame; we need one good one"), better
color management (for things like printers too), and proper support for
compression in the X protocol. It seems that the bulk of X traffic,
anymore, is image data, and much of that data compresses very well.
Finally, there are the packaging and development issues. Work to split the
X distribution into its component pieces (server, libraries, applications,
fonts) continues; there will eventually be no "tarball from hell" to deal
with. Modern building tools are being adopted; imake is on its way out.
Much of this work is done, but some pieces remain.
On the community side, Jim notes that X.org is a very different place from
what it was a year ago. Most of the XFree86 developers have come over, and
it is clearly the focal point for X development.
Perhaps the most important part of Jim's talk was repeated several times:
the X developers have ambitious aims and a great deal of work that needs to
be done; they are actively looking for developers to take on pieces of it.
For anybody out there who is looking for a project to support, X.org
represents a major opportunity. X is the core, the kernel, as it were, of
the Linux graphical environment. The development effort is healthy again,
and it has no end of interesting and challenging tasks to address. It is
hard to imagine a better, more interesting, more visible, or more important
project to work with. If you are looking to do some hacking, X would be an
ideal place to do it.
Comments (57 posted)
Harald Welte, the current netfilter maintainer, ran a session on dealing
with GPL violations. Harald has made a name for himself over the last year
by reaching settlements with several manufacturers who were shipping
products containing the netfilter code without complying with its licensing
requirements. In most cases, these settlements have been relatively easy
to reach, with the offending companies releasing their code and, in many
cases, making a donation to the FSF or a related worthy cause. Harald is
also the force behind the injunction in the Sitecom case, which just
received another court ruling that the GPL is valid and enforceable in
Harald's approach differs from that of the Free Software Foundation. The
FSF tries to resolve GPL violations in the most quiet, friendly way
possible; in most cases, the wider world never even hears that there is a
problem. Harald found himself frustrated with this way of doing things; it
takes far too long (perhaps longer than the lifetime of the offending
product) and gives no real disincentive for companies considering ignoring
the GPL. So he took matters into his own hands; by bringing in lawyers
early, making violations public,
and threatening immediate damage to the offenders' bottom line, he
has gotten some real results.
Some interesting features of the German legal system have worked in
Harald's favor in this campaign. By (his lawyers' interpretation of)
German law, simple use of the netfilter/iptables internal API is a strong
indication of a derivative work. Since it is not a "standard" API, there
is no copyright boundary there; by this same reasoning, any binary-only
kernel module is a GPL violation. This interpretation of the GPL also
rules out putting GPL-licensed code onto hardware which only runs signed
binaries - unless the key is distributed with the source.
German law requires that any request for injunctive relief be filed within
four weeks of the discovery of the infringement. This is an inflexible,
externally-imposed deadline which forces companies to move quickly to
resolve the issue. When a company has received a cease-and-desist notice,
it knows that it cannot drag it issue out over time; it will, instead, find itself
in court in short order.
Harald was asked about the completeness of the code releases he has won so
far; apparently not all vendors have released kernels which actually can be
rebuilt into a working image - or which even compile. His response is that
his ability to compel code releases really only extends as far as his
copyrights in the netfilter code. If a vendor does not release a full
kernel, copyright holders in other parts of the system will have to get
The last part of the talk covered things developers can do to help make
copyright enforcement easier. They include:
- Don't fix spelling errors and typos, and leave strange messages
(example: "Rusty needs more caffeine") in place. The presence of this
sort of text in a binary image is an obvious sign of copying. The
removal of this text, instead, would be a clear sign of a willful
violation, which raises the stakes considerably.
- The copyright message needs to be in the binary as well.
- It is important to track the names of all contributors to the code, so
they can be found for enforcement actions or the distribution of any
sort of damages.
- When a violation is found, it should not be made public at the time,
as that can make getting injunctive relief harder. Instead, a lawyer
should be brought in to prepare and send the "cease and desist" note.
Expect to hear about more resolved violations in the near future. Harald
indicated that a couple of settlements have been reached, but the companies
involved have been given a grace period to prepare for the fallout once the
situation has been made public. Once that period has passed, the press
releases will go out.
Comments (35 posted)
, a company founded by a
number of early Red Hat developers, recently came out of hiding. At the
2004 Ottawa Linux Symposium, Eric Troan gave a presentation on Conary, the
company's system for package, repository, and distribution management. It
was a technical talk from the beginning to the end; Eric would not talk
about Specifix's business model even when asked (though he offered to do so
in private). If nothing else, he understands what the OLS crowd is looking
Package management systems have come into use in almost every distribution
out there. They are a clear step up from what came before, but, as Eric
pointed out, significant problems have been building for years. These
- Repositories are an afterthought. A typical repository is a simple
collection of files in whatever package format is being used, perhaps
with a bit of metadata.
- The version scheme used by most package managers follows a straight
line model; there is no provision for branches. That makes it hard,
for example, to determine which version of a package is appropriate
for a specific release of a given distribution.
- Packages contain scripts which handle parts of the installation and
removal process which go beyond the simple management of files. These
scripts tend to contain a lot of boilerplate, and are replicated in
every package file. Bugs, too, are replicated, and there is no one
place to go to fix them. The scripts are also not portable across
distributions (even those using the same package format) and cannot be
customized for an individual site's needs.
Conary was developed as a way of addressing the above limitations and to
make it possible for users to create their own, customized distributions in
an easy manner. In the simplest sense, one can think of Conary as a
package management system with a more consistent view of objects from the
repository level down to individual files, combined with a version
Conary treats files as "first class objects," which are managed by the
framework as a whole. Files have a unique ID and a version history; they
also have a set of attributes. One of those attributes is the file's
location in the filesystem; moving a file is a simple matter of changing
A "trove" is a container holding one or more files and other troves. Files
are contained by reference. A "component" is a collection of files, by
reference. Example components listed by Eric for the bzip2 package might
be bzip2:runtime (binary files to run the program),
bzip2:lib, bzip2:doc, and, of course,
bzip2:source. Components can be aggregated
together into packages. Both components and packages are considered to be
"troves," for what it's worth.
Version strings are hung onto everything; Specifix has added some
complexity to the versioning system, though. Each version string includes
the repository name, a namespace (think of it as a distribution name), a
branch name (for the creation of trees in the version space), the upstream
package version, and a two-part local revision number. Needless to say,
the version strings get long, but the system hides the full string most of
the time. Creating versions in this way allows the system to easily
determine which version of a package is the newest, which version of which
distribution is built for, and so on.
Branching is done by adding a branch name to the version string. Branching
allows the tracking of versions of packages which were shipped with a
specific distribution, along with updates to those packages. There is also
a special type of branch called a "shadow" which tracks changes to the
trunk it was branched from. Essentially, the shadow is automatically
merged with each new version of the trunk it is following. This feature
would be useful for somebody maintaining a derivative distribution; they
want to keep up with what the source distribution is doing without losing
track of their own changes. The only problem with shadows is that, like a
number of other Conary features, they are not actually implemented yet.
"Flavors" are another Conary feature; they seem to be patterned after
Gentoo's "USE flags." A flavor is a set of configuration options
describing how all packages are to be built. This feature is used for
multiple architecture support, or for building versions of distributions
with different feature sets (e.g. creating a distribution without PAM
support). Multiple flavors of a package can be installed on a system if
they don't conflict with each other; this allows, for example, the
installation of 32-bit libraries on x86-64 systems.
Then, there is the concept of "changesets." A changeset is a collection of
modifications to files (including attribute changes) and the troves which
contain them. A changeset is, essentially, a patch to a package or a
distribution. Changesets, which track only changes, can be much smaller
than the packages they describe, and can thus be an efficient way of
distributing updates. Changesets describe changes to configuration files
in diff format, which often allows them to be merged automatically
with local changes. A system administrator can also create a changeset
describing his or her local changes to the system; that changeset can then
be used for merging with updates, or replicating the system elsewhere.
Local changesets can also be used for version control and the tracking of
"Tags" are Conary's answer to the package script problem (and, also, to the
complex set of interactions represented by the RPM "trigger" mechanism). A
tag is a file attribute describing the type of the file, be it "shared
library," "info file," or any of a long list of alternatives. Most files
can be tagged automatically by Conary. Tags have scripts associated with
them; there is, for example, a script which handles the installation of an
info file and updating the relevant directory. These scripts are
distributed separately; there is only one copy of them on the system. The
scripts are thus easily fixed when bugs turn up, and they can be customized
by the local administrator if need be. Separating out the management
scripts in this way should also make it easier to install packages from
A "fileset" is an arbitrary collection of files built from components in
the repository. Filesets seem to be intended to help in the creation of
small system images for embedded systems; they allow an easy picking and
choosing of an exact set of desired files. "Groups" are, instead, the
analog of the Debian "task" or Anaconda "component." They allow the
management of several packages as a unit, but they come with their own
local changesets so that local changes to the group are tracked properly.
from the OLS proceedings (PDF format) is worthwhile reading for anybody
wanting more details on how Conary works.
Interested parties can download an early Conary release from the Specifix web site.
Be warned, however, that a few features are still missing; they include
shadows, dependencies (an important issue that they "think" they know how
to implement), flavors, package signatures, and more. "Release early" is an
important part of the free software development process, however, and the
Specifix founders understand that process well. Conary's vaporware
features will, beyond doubt, be filled in soon. As that happens, expect
interest in this tool to increase; it truly does have the potential to
change the way we set up and manage our projects, distributions, and
Comments (24 posted)
Page editor: Rebecca Sobol
Inside this week's LWN.net Weekly Edition
- Security: The value of privacy policies; New vulnerabilities in courier, samba, sox, subversion, ...
- Kernel: Another look at the new development process; Voluntary preemption and interrupt handlers; The kernel events layer.
- Distributions: A look at Progeny Debian Beta; Xandros Desktop 2.5; Catux-USB, PHP Solutions Live
- Development: gputils, the GNU PIC Utilities,
new versions of Speex, Samba, GNOME, KE, CWirc, Liferea, Gnumeric,
Furthur, Devhelp, GCL, SBCL, Ponie, Durus, Vex, DrPython.
- Press: Choosing an open source license,
OLS and OSCON coverage,
Asianux 1.0, NASA's new Linux cluster.
- Announcements: $149 xterm, OSDN becomes OSTG, Eclipse launches Web Tools,
The Perl Review, Free Software Printing Summit summary,
aKademy Lineup, ApacheCon CFP, FSF Seminars on GPL.