LWN.net Weekly Edition for January 25, 2007

linux.conf.au 2007

The seventh linux.conf.au, held in Sydney, has run its course. Your editor has written several articles based on what was discussed there, but they do not even begin to do justice to this event. In an attempt to fill in some of the gaps, this article will cover some of the remaining high points, from your editor's point of view.

One interesting talk was Jono Bacon's discussion of Jokosher, a relatively new entrant into the crowded field of audio recording and editing tools. Jono wanted a simple and useful tool to help in the creation of podcasts and his particular variety of frenetic metal music, but the currently-available studio tools were rather intimidating in their complexity. So he made a new one, giving a lot of thought in the process to how it might actually be used. Jokosher features a simpler waveform display, large buttons (easy to hit by a standing guitarist), tools which don't clutter the display when they are not relevant, etc. Version 0.2 is usable; there is a 0.9 release coming soon. Your editor has not yet had a chance to play with Jokosher, but it's on the list for that mythical day (always next month) when he has some free time.

Virtualization was a hot topic at the event - registrations for the associated miniconf exceeded those for the Debian miniconf, the first time Debian has ever been upstaged. During the main conference, Rusty Russell, Jeremy Fitzhardinge, Zachary Amsden, and Chris Wright all got together to present the Lguest, Xen, and VMI technologies. It was good to see that many developers from competing projects (competing for business as well as for developers and mindshare) having fun together on stage, even if the short time slot did not allow for the conveyance of a great deal of technical information. Given the level of interest in Lguest, it really should have had a session of its own. An unexpected lesson from this session concerned the hazards of leaving a popup-enabled IRC client running on a network-connected laptop during a presentation. One does not normally expect one's slides to turn into a real-time wiki. The audience is to be commended for its restrained and tasteful use of this new communication path.

The kernel miniconf was well attended - to the point that it displaced Debian from the largest room. Your editor attended a panel session where various topics of interest were discussed. On Xen: "crap is good - it shows you how not to do stuff." The ongoing maintenance of 2.6.16 was described as "mostly irrelevant" and "a nice try." The developers expressed their general happiness with the new development model, and Linus Torvalds claimed that there would only be a 2.7 kernel when some smart new hacker wrests control of the kernel away. David Miller noted that he does miss the old development kernels, however; it is harder now to make major changes which break things for long periods of time. There was also a session on suspend and resume where a couple of laptops were made to work - with Linus, at one point, getting up and jumping into the process.

Open Day

A true high point of the conference was "Open Day," a sort of exhibit populated by people doing truly interesting things with Linux and more. The event was open to the public, with the hope that children, in particular, would attend. The display of enthusiasm, ingenuity, and just plain fun was breathtaking; it was an expression of the spirit which infuses the entire conference. Open Day showed off games (TuxRacer, Stepmania, Wesnoth, etc.), a home-made Segway-like device, serious amateur rocketry, the Qtopia Green Phone, Blender, MythTV, and much more. RepRap was worth the price of admission by itself; RepRap is a 3D printer which can create its own parts - and which is powered by free software, of course.

The reaction from participants and attendees was strong - it is hard not to be swept away by that many people passionately doing amazing things. Open Day is a great addition to the LCA program; hopefully next year's conference will do it again.

Andi Kleen talked about the "generations" of kernel hackers and how well they work with the development process. The Janitor Generation does useful work, but there needs to be better ways of graduating janitors to more complex projects. The "login name tree" generation created a proliferation of specialized kernel trees - useful before the arrival of BitKeeper, when the patch submission process was much more lossy. The "Russian mathematician" generation will be familiar to anybody who watches the kernel lists; Andi suspects that some of them are actually a room full of hackers working under a single name. The "deadline generation" has been brought about by the new kernel development process: there is now significant pressure for maintainers to get their patches in during the brief merge window period. Andi complains that kernel development used to be more relaxed before the merge window came into existence. The "bugmaster generation," alas, does not exist yet; it's an opportunity for somebody to become famous.

The final keynote was presented by Kathy Sierra, who is, perhaps, better known in the Java community. She talked about how to create passionate users - those who will put free software on their desktops and get their friends to do it as well. The ideas presented are the sort which seem obvious after the fact (help users "suck less" at what they do, provide "levels" like games do to reward increasing expertise, make a supportive community), but, all together, they make a compelling picture of how we can develop more empathy for our users and bring more people in. The talk was well received, and rightly so.

Linux.conf.au hit some interesting milestones this time around. At some 800 attendees, it was by far the largest LCA event yet. Even more significant is the fact that a full 10% of those attendees were female. Any community which feels that 10% female participation is worthy of celebration clearly has some ground to cover yet, but that's where we stand. 10% is a step in the right direction and a base from which the community can work to fully include a group of people whose energy and perspective is sorely needed.

It is hard to imagine an event better suited to bringing people into our community in general. Your editor has attended quite a few Linux and free software events on a number of continents, but there are few which bring together the combination of serious technology, community, and flat-out fun found at LCA. The LCA formula, which involves a fresh group of volunteers running the show with guidance from past veterans, does well at preserving a successful model while bringing in new energy and ideas. The result is one of the most vibrant, vital, and (increasingly) important free software events anywhere on the planet. Your editor hopes to be lucky enough to report from LCA2008 (to be held in Melbourne) next year.

[Postscript: one of the standout performances in this nearly flawlessly run conference is the group, led by Silvia Pfeiffer, which recorded the presentations. Recordings hit the web site while the conference was still open, and the rest are being added quickly. See the program page to view the talks - in free formats, of course.]

Comments (4 posted)

LCA: Andrew Tanenbaum on creating reliable systems

Andrew Tanenbaum is a huge figure in the field of computer science; developers who work in the area of operating systems tend to have at least one of his books on their shelf. Linus Torvalds also occupies a prominent position. But when these two people are discussed together, the topic is almost always the famous debate between the two which happened early in the history of Linux. Mr. Tanenbaum called Linux "obsolete," and made it clear that he would not have been proud to have Mr. Torvalds as a student; Linus made some choice comments of his own in return.

So it was pleasant to see Andrew Tanenbaum introduced in Sydney by none other than Linus Torvalds. According to Linus, Andrew introduced him to Unix by way of Minix. Minix also convinced Linus (wrongly, he says) that writing an operating system was not hard. The similarities between the two, he said, far outweigh any differences they may have had.

The talk began with a quoting of Myhrvold's laws: (1) software is a gas which expands to fill its container, and (2) software is getting slower faster than hardware is getting faster. Software bloat, he says, is a huge problem. He discussed the size of various Windows releases, ending up with Windows XP at 60 million lines. Nobody, he says, understands XP. That leads to situations where people - even those well educated in computer science, do not understand their systems and cannot fix them.

The way things should be, instead, is described by the "TV model." Generally, one buys a television, plugs it in, and it just works for ten years. The computer model, instead, goes something like this: buy the computer, plug it in, install the service packs, install the security patches, install the device drivers, install the anti-virus application, install the anti-spyware system, and reboot...

...and it doesn't work. So call the helpdesk, wait on hold, and be told to reinstall Windows. A recent article in the New York Times reported that 25% of computer users have become so upset with their systems that they have hit them.

So what we want to do is to build more reliable systems. The working definition of a reliable system is this: a typical heavy user never experiences a single failure, and does not know anybody who has ever experienced a failure. Some systems which can meet this definition now include televisions, stereos, DVD players, cellular phones (though some in the audience have had different experiences), and automobiles (at least, with regard to the software systems they run). Reliability is possible, and it is necessary: "Just ask Grandma."

As an aside, Mr. Tanenbaum asked whether Linux was more reliable than Windows. His answer was "probably," based mainly on the fact that the kernel is much smaller. Even so, doing some quick back-of-the-envelope calculations, he concluded that there must be about 10,000 bugs in the Linux kernel. So Linux has not yet achieved the level of reliability he is looking for.

Is reliability achievable? It was noted that there are systems which can survive hardware failures; RAID arrays and ECC memory were the examples given. TCP/IP can survive lost packets, and CDROMs can handle all kinds of read failures. What we need is a way to survive software failures too. We'll have succeeded, he says, when no computer comes equipped with a reset button.

It is time, says Mr. Tanenbaum, to rethink operating systems. Linux, for how good it is, is really a better version of Multics, a system which dates from the 1960's. It is time to refocus, bearing in mind that the environment has changed. We have "nearly infinite" hardware, but we have filled it with software weighed down with tons of useless features. This software is slow, bloated, and buggy; it is a bad direction to have taken. To achieve the TV model we need to build software which is small, modular, and self-healing. In particular, it needs to be able to replace crashed modules on the fly.

So we get into Andrew Tanenbaum's notion of "intelligent design," as applied to software. The core rules are:

• Isolate components from each other so that they cannot interfere with each other - or even communicate unless there is a reason to do so.

• Stick to the "principle of least authority"; no component should have more privilege than it needs to get its job done.

• The failure of one component should not cause others to fail.

• The health of components should be monitored; if one stops operating properly, the system should know about it.

• One must be prepared to replace components in a running system.

There is a series of steps to take to apply these principles. The first is to move all loadable modules out of the kernel; these include drivers, filesystems, and more. Each should run as a separate process with limited authority. He pointed out that this is beginning to happen with Linux with the interest in user-space drivers - though it is not clear how far Linux will go in that direction.

Then it's time to isolate I/O devices. One key to reliability is to do away with memory-mapped I/O; it just brings too many race conditions and opportunities for trouble. Access to devices is through I/O ports, and that is strictly limited; device drivers can only work with the ports they have been specifically authorized to use. Finally, DMA operations should be constrained to memory areas which the driver has been authorized to access; this requires a higher level of support from the hardware, however.

The third step is minimizing privileges to the greatest extent possible. Kernel calls should be limited to those which are needed to get a job done; device drivers, for example, should not be able to create new processes. Communication between processes should be limited to those which truly need to talk to each other. And, when dealing with communications, a faulty receiver should never be able to block the sender.

Mr. Tanenbaum (with students) has set out to implement all of this in Minix. He has had trouble with people continually asking for new features, but, he has been "keeping it simple waiting for the messiah." That remark was accompanied with a picture of Richard Stallman in full St. Ignucious attire. Minix 3 has been completely redesigned with reliability in mind; the current version does not have all of the features described, but 3.1.3 (due around March) will.

Minix is a microkernel system, so, at the bottom level, it has a very small kernel. It handles interrupts, the core notion of processes, and the system clock. There is a simple inter-process communication mechanism for sending messages around the system. It is built on a request/reply structure, so that the kernel always knows which requests have not yet been acted upon.

There is also a simple kernel API for device drivers. These include reading and writing I/O ports (drivers do not have direct access to ports), setting interrupt policies, and copying data to and from a process's virtual address space. For virtual address space access, the driver will be constrained to a range of addresses explicitly authorized by the calling process.

Everything else runs in user mode. Low-level user-mode processes include the device drivers, filesystems, a process server, a "reincarnation server," an information server, a data store, a network server (implementing TCP/IP), and more. The reincarnation server's job is to be the parent of all low-level system processes. It gets notified if any of them die, and occasionally pings them to be sure that they are still responsive. Should a process go away, a table of actions is consulted to see how the system should respond; often that response involves restarting the process.

If, for example, a disk driver dies, the reincarnation server will start a new one. It will also tell the filesystem process(es) about the fact that there is a new disk driver; the filesystems can then restart any requests that had been outstanding at the time of the failure. Things pick up where they were before. Disks are relatively easy to handle this way; servers which maintain a higher level of internal or device state can be harder.

A key point is that most operating system failures in deployed systems tend to result from transient events. If a race condition leads to the demise of a device driver, that same race is unlikely to repeat after the driver is restarted. Algorithmic errors which are repeatable will get fixed eventually, but the transient problems can be much harder to track down. So the next best thing is to be able to restart failing code and expect that things will work better the second time.

There were a number of performance figures presented. Running disk benchmarks while occasionally killing the driver had the unsurprising result of hurting performance a bit - but the system continued to run. Another set of numbers made the claim that the performance impact of the microkernel architecture was on the order of 5-10%. It's worth noting that not everybody buys those numbers; there were not a whole lot of details on how they were generated.

In summary, Mr. Tanenbaum listed a number of goals for the Minix project. Minix may well be applicable for high-reliability systems, and for embedded applications as well. But, primarily, the purpose is to demonstrate the the creation of ultra-reliable systems is possible.

The talk did show that it is possible to code systems which can isolate certain kinds of faults and attempt to recover from them. It was an entertaining and well-presented discussion. Your editor has not, however, noticed a surge of sympathy for the idea of moving Linux over to a microkernel architecture. So it is not clear whether the ideas presented in this talk will have an influence over how Linux is developed in the future.

Comments (87 posted)

LCA: Updates on the X Window System

The X Window System is, in some sense, the kernel of our graphical desktop systems; it controls access to the hardware and ensures that applications play well together. So the capabilities provided by X matter, and that importance can only increase as free software developers work toward the creation of more complete and compelling desktop experiences. Keith Packard gave a couple of talks at linux.conf.au in Sydney on where X is going; your editor had no choice but to be there and listen.

In its early days, X would normally be run on some sort of Unix workstation. The display hardware in use in those days was not normally expected to change while X was running - or over the life of the system in general. One connected The Monitor to The Adapter and things stayed that way forevermore. So the X protocol was set up to enumerate all of the available screens whenever an application made its connection. There was no way to add more screens on the fly or change their geometry, and there was no way to move windows from one screen to another. Fixing this was a hard problem.

As graphics hardware has become more powerful and flexible, a number of extensions have been developed in an attempt to provide proper support in X. The Xinerama extension uses a clever technique: merging all of the monitors into a single, large, virtual screen. Applications can then move between monitors, because they think they are just moving around on the same screen. The XFree86 VidModeExtension tried to address hardware changes by allowing the video modes to be changed on the fly. Then along came the first version of the Resize and Rotate (RandR) extension, which tried to improve the handling of mode changes and implement rotation - especially useful on handheld devices, where the screen can be used in both landscape and portrait orientations. RandR 1.0 was limited by a policy (imposed by the XFree86 maintainers) that the driver API could not be changed; as a result it was nowhere near as flexible as its developers would have liked.

All of this came together into "a kludge tower of extensions" which was guaranteed to fall down, sooner or later.

Since then, the X Window System has come under new management and the need for display flexibility has continued to grow. Enter RandR 1.2, soon to come to an X server near you. The new RandR release comes with the intention of being able to fully express (and use) the capabilities of the hardware. All configuration options will be brought back together into a single file, and they will all be adjustable at run time. Much of the driver-specific code has been moved back into the core, allowing all hardware to be configured in the same way. This was a much-needed change; according to Keith there are currently five independent Xinerama implementations in the X server.

RandR 1.2 uses a combination of new and old concepts. A "screen" retains its current meaning, and the one big screen is still present. Each screen, however, can work with one or more "CRT controllers," (CRTCs) each of which grabs a rectangular portion of the big screen and sends it to a monitor (highly unlikely to actually be a CRT anymore). Each CRTC, in turn, has one or more outputs which connect to physical devices.

The flexibility of this approach was easily demonstrated on Keith's shiny little laptop. The hardware is able to implement a 2K pixel square screen, which is then scanned by three different CRTCs: the built-in display, the video output, and the (unconnected) TV output. By default, they all look at the same portion of the screen, but, with a little command line magic, that can be changed. So Keith's laptop can display an entirely different set of windows out of each CRTC; the video output can send his talk slides to the projector while the laptop screen shows something else. The display areas can overlap if desired.

If a new monitor is plugged into the system, the RandR code will detect the event and react accordingly. The new output will be turned on and given screen space according to whatever policy is in effect. If need be, the user's desktop area will be expanded to cover a wider display. Similar things happen if a monitor is removed. It all Just Works.

While he was at it, Keith extended RandR to cover some other useful hardware capabilities. These include the ability to configure the gamma lookup table, allowing for on-the-fly contrast and brightness adjustments. Applications can get the monitor's EDID identification data, should they be interested, and parameters like the brightness of the backlight can be tweaked.

The current status is that the protocol and device-independent work are done. The Intel driver works now, and the Radeon driver is "nearly usable." This code is getting ready for people to use. When most people will actually use this code depends on the release schedule, however. At a separate talk (in the middle of the Debian miniconf) Keith covered what's coming up from the X.org project.

Coming soon is the X server 1.2 release. This one looks mostly like a maintenance release; Keith says that a lot of Coverity-found bugs have been fixed. Things have been cleaned up to the point that this release has 40,000 fewer lines of code - but more functionality. Keith noted that the policy of splitting the X drivers from the core server has not worked as well as they would have liked. It adds a whole set of API compatibility issues between the two, making it hard to develop and release improved versions of the server. Keith now thinks that the Linux kernel developers got it right by keeping drivers inside the kernel.

LibX11 1.1.1 is coming soon. The big change there is that the new XCB interface is being used underneath the old Xlib API, making it easy to migrate applications in an incremental manner.

Later on we can expect release 1.2.1 of the X server. This release will include an EXA acceleration implementation "that actually works." The RandR 1.2 code described above will also make its appearance here. Further ahead, the 1.3 release (to be part of a general X.org 7.3 release) will include significant ABI changes. A lot of the "PCI munging" is coming out of the drivers. Yes, he said, this will mess up the proprietary NVidia and ATI drivers. There will also be better support for hotplugging of input devices.

There is a Mesa 6.5.2 release coming with OpenGL 2.0 API support. It also has a new memory manager which can work with the memory management unit found in modern graphics cards; it can do things like map arbitrary regions of host memory into the adapter's address space. Among other things, this means that off-screen objects can be made writable, which will be a big performance win.

On the Intel driver front, the mode setting code has been much improved in recent times. Not surprisingly (considering that Keith works for Intel these days), this driver is the first to have full RandR 1.2 support. All outputs are fully supported, and EXA is as well. Intel has set a goal of having drivers available for new chipsets on the day those chipsets are launched. When asked if Intel planned to start selling discrete adapters, he became very silent, however.

Looking further ahead, the X developers would like to move video card mode setting into the kernel. There are a lot of reasons for doing this, starting with simple robustness. It would also enable better suspend and resume support, and better handling of panics: if the system goes into an oops, an in-kernel mode-setting routine can switch back to a text mode, allowing the oops text to actually be read. There is a lot of interest in supporting multiple, simultaneous X sessions on the same screen without using Linux virtual terminals; the goal here is to enable fast switching between user accounts. And there is interest in H.264 acceleration, facilitating the display of important things like HDTV. It seems that even contemporary CPUs can have trouble keeping up with HDTV streams.

Overall, Keith painted a picture of a revitalized X project which is truly beginning to hit its stride. A lot of work is being done toward the goals of fully supporting current hardware and providing the foundation for the creation of the best desktop available anywhere. One cannot help but look forward to where things will go from here.

Comments (55 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: The OpenLiberty Project; New vulnerabilities in ed, gtk2, squid, xine, ...
• Kernel: An introduction to Lguest; Video4Linux2: Colors and formats.
• Distributions: The Fedora Extras licensing audit; Fedora Core 6 Re-Spins from Fedora Unity; Gamix Basesystem Beta 3; openSUSE build service; a new life cycle for Mandriva Linux; UbuCon New York; BOSS
• Development: The new GNOME 2.18 modules, new versions of SQLite, BusyBox, Django, Rails, Tina POS, eSpeak, MaklerPlus, Varkon, PLplot, Xfce, SQL-Ledger, Cyphesis, SeaMonkey, bzip2, cairomm, IT++.
• Press: Groklaw on OOXML, The Big Picture, LCA 2007 coverage, Kerala, India chooses FOSS, Alan Cox addresses House of Lords, OO.o configuration, Samba share tutorial, reviews of Amarok, FreeNAS, audio players, OVIS, Psion.
• Announcements: FSFE becomes guardian of OpenSwarm, Two open gaming platforms, openLiberty Project, OpenMoko phones coming soon, OSDL and FSG to merge, Macraigor Systems joins Eclipse, ODF Toolkit Project, PyWeek four, GNU Radio/USRP training, Linux USB device driver training, Cell processor hack-a-thon coverage, Gelato ICE registration, OpenSolutions Summit, Open Source Think Tank.

Next page: Security>>