Weekly Edition Return to the Front page |
Trip report: the Ottawa Linux Symposium
The 2003 Ottawa Linux Symposium has run its course. Once again, OLS has
established itself as the premier North American Linux developers'
conference. A solid roster of speakers delivered four days' worth of
intensely technical talks on where Linux development (and kernel
development in particular) is headed. It is always nice to attend an event
where the talk is technical and nobody is trying to sell you anything.
Your editor was not able to attend all of the presentations, of course, and did not write up every one he attended. Below, however, you'll find quick summaries of several of the more interesting talks given at OLS this year. Those looking for all the details can find them in the OLS 2003 Proceedings. In response to popular demand (i.e. somebody actually asked), I have also put up the slides for my talk on porting drivers to the new kernel. Giving this sort of talk at OLS is a unique challenge, given that, for any topic, there's certain to be at least one person in the audience who knows way more than the speaker. Happily, the hecklers were kind... Thanks are due to the OLS organizers for putting together another high-quality event, for the event's sponsors for helping making it possible, and to all the speakers for presenting their work.
Ugly ducklings - resurrecting unmaintained codeDave Jone's talk covered the work he has done in 2.5 to fix up the MTRR and AGPGART drivers. Dave has observed a common sort of lifecycle for drivers. A driver is initially written for a specific vendor's widget. Over time, it is extended to support compatible widgets from other vendors, then slightly different widgets from yet other vendors. The number of special cases increases. Meanwhile, the maintainer gets bored and moves on. Eventually you end up with thousands of lines of spaghetti which is unmaintainable.Dave's approach to such drivers includes splitting code into separate files by vendor (usually) and separating code which should never have been run together in the first place. "Useless abstractions" can be cleaned out. Eventually you end up with a code body which is sufficiently clean and understandable that it can be updated for modern hardware, new features, etc. But one should not underestimate the amount of work it can take to get there. Large projects and bugzillaLuis Villa discussed his experience working as GNOME's quality assurance person. He has, he estimates, read some 30,000 bug reports over the last few years. The experience appears to not have warped him too badly, though such things can take years to show.He is, as one might expect, a strong proponent of organized bug tracking. A good QA system, he says, makes writing software easier (through reductions in mailing list traffic, among other things), eases the release process, makes the software better, and, important, makes writing software more fun. The key point of the talk, perhaps, was that QA people have less power in free software projects than they do in the proprietary world. That makes it even more imperative that they not forget that they are providing a service to the developers, and that they have to understand what the developers need from them. Filtering ("triage") is especially important; developers should not have to deal directly with the full flow of bug reports. If the bug trackers are providing the sort of bug filtering and categorization that the hackers need, all will be well. Otherwise the bug tracking system will degenerate into an unused pile of old information. Interactive kernel performanceRobert Love's talk covered work done in the 2.5 kernel to improve interactive performance. What's interactive? Robert takes a wide view; interactive applications are "everything except Oracle." The topics covered will be familiar to LWN Kernel Page readers; they include the anticipatory disk scheduler, the O(1) process scheduler, the preemptible kernel and other low-latency work, etc. In his opinion, the single most important bit of work to go in this time around (with regard to interactive performance) is the anticipatory scheduler.
udev - devfs done rightGreg Kroah-Hartman described udev, his user-space devfs replacement (covered here last April) in a standing room only session. Progress on udev has been slow since April (Greg has been busy with other stuff), but some things have happened. There is now a set of configuration files to allow the user to specify how device naming and permissions should be handled; it uses various attributes of a device (it's serial number, label, position in the bus topology, etc.) to figure out what the system administrator would like it to be called. Future versions will use the "tdb" database to track devices and handle persistent naming.Future work includes changing udev to run as a daemon process; this change is required to properly handle out-of-order hotplug events. For those wanting to experiment with it, the udev code can be found on kernel.org in /pub/linux/utils/kernel/hotplug/. Why doesn't my laptop suspend?Pat Mochel's talk was on power management, or "why doesn't my laptop suspend?" He asked for a show of hands: how many in the audience have laptops? Well, this is OLS, so most of the attendees raised their hands. How many of those suspend correctly? Most hands went down.Older, APM-based machines would handle suspend operations entirely in the firmware; it "just worked" for most people. Newer ACPI systems, however, push the suspend task into the software; this is evidently an improvement. And Linux software has not yet caught up with that. ACPI support is pretty much in place, but that is the easy part. The harder part is working power management support into all the drivers, coming up with a reliable way of suspending the system, and implementing a reasonable user-level interface to it all. Much of this work has been done for 2.5; it still languishes in Pat's tree, however, and has not been merged into the mainline. The changes include a new set of driver power management methods; there is also a cleaned up software suspend subsystem with a safer snapshot mechanism and the ability to write the system image to any persistent media. Pat has said that he will finish this work, though it was clear that he would appreciate some help from other developers as well. His hope is to get the work merged by August 20. Should he be successful, appreciative users should send him a birthday present ("small, unmarked bills") on that day. Toward an O(1) VMRik van Riel discussed recent work with virtual memory management; the talk covered page replacement strategies, the reverse mapping VM, etc. The key point of his talk, however, was this: by many metrics relevent to VM, our newer, "faster" machines are actually slower. Over the years, the time required to perform tasks like reading an entire disk, or writing a system's entire RAM to disk, has gone up by a couple of orders of magnitude or more. Much of the previous century's research into VM is losing its relevance as memory and disk sizes increases faster then the transfer speeds between them. VM hackers increasingly find themselves having to make things up as they go.
Integrating DMA into the device modelJames Bottomley discussed the new generic DMA layer; these functions have been documented in this article, which is part of the LWN driver porting series. What was new in this talk is James's discussion of where the DMA API needs to go in the future. The current version has no way of returning a failure code from the DMA mapping routines. But failure can happen: a system can run out of I/O memory management unit space, a problem which will be exacerbated in the future as GART hardware is used as a poor man's IOMMU. At some point, that sort of failure must be communicated back to the caller.Device drivers can provide a DMA mask describing the range of addresses that their devices can handle. But there is no way for the system to pass back a mask saying which addresses the device needs to handle. Better performance can often be maintained when devices are operated in their smaller-memory modes; the system should provide the information that allows those modes to be used when they are applicable. Finally, the current approach to cache coherency needs some work; drivers should be able to find out just how coherency works on the host system. The means by which the CPU and peripherals share DMA buffers needs to be reorganized into a straightforward ownership model; in the current system, it is not always clear who has the right to change a buffer, and that can lead to data corruption. The partyYour reporter was going to write up the legendary OLS closing party at the Black Thorn, but the whole event has become somewhat fuzzy and difficult to recall. Suffice to say that a lot of fun was had.(Log in to post comments)
Rik van Riel's Talk Posted Jul 29, 2003 1:26 UTC (Tue) by tjc (subscriber, #137) [Link] Can anyone tell me more about Rik van Riel's talk? I'm interested in what he had to say about page replacement algorithms in particular.
Rik van Riel's Talk Posted Jul 29, 2003 1:55 UTC (Tue) by Webexcess (subscriber, #197) [Link] One thing he talked about was how hard it is to deal with programs that behave badly. An example mentioned was how some Java virtual machines that do garbage collection "at some point in the future" tend to crawl through all available memory object allocation by object allocation. This puts pressure on the memory of other applications that haven't been used recently, like say your mail app. When you try to switch back to that other app you find that it's been completely paged out (chug.. chug.. chug..).Also the current page table is growing too big, and needs to be replaced. The obvious options include dynamic page table allocation and swapping out page tables. Neither of these options is good - the first imposes a general system-wide slowdown on page allocation, the second could have horrible impact when you swap some page tables in only to realize that you need to swap the pages themselves in too (bad for the elevator).
Rik van Riel's Talk Posted Jul 29, 2003 3:08 UTC (Tue) by tjc (subscriber, #137) [Link] An example mentioned was how some Java virtual machines that do garbage collection "at some point in the future" tend to crawl through all available memory object allocation by object allocation.Does the JVM walk through it's entire virtual address space, or does it just visit pages that have been marked in some way? The former would be horrible, but I'm not sure the later is possible on most architectures. Also the current page table is growing too big, and needs to be replaced. Isn't this tied to the hardware architecture, more or less? (bad for the elevator) By "bad for the elevator", you are talking about the disk allocation algorithm? Someone needs to write a "Kernel Hacking for Dummies" book to help out us application programmers. :-) I've picked up bits and pieces on the net, but some of them are probably assembled in the wrong order. How in the world do you learn this stuff?
Rik van Riel's Talk Posted Jul 29, 2003 4:19 UTC (Tue) by Webexcess (subscriber, #197) [Link] Does the JVM walk through it's entire virtual address space, or does it just visit pages that have been marked in some way?I don't know enough about Java VMs to say. :-| By "bad for the elevator", you are talking about the disk allocation algorithm? The elevator code is there to reduce disk seeking by reordering disk operations. Unfortunately, when one operation depends on the preceeding one, it never gets a chance. If your page tables are on disk, then you don't know where on disk the page data you need is until after you've fetched the tables. Ie, paging gets alot slower.
Rik van Riel's Talk Posted Jul 29, 2003 6:08 UTC (Tue) by walters (subscriber, #7396) [Link] Does the JVM walk through it's entire virtual address space, or does it just visit pages that have been marked in some way? The former would be horrible, but I'm not sure the later is possible on most architectures.Neither. Most of them only traverse allocated objects, looking at pointers. But I think what Rik is talking about is that this may end up hitting the majority of pages allocated. See any paper on garbage collection for more information.
Rik van Riel's Talk Posted Jul 29, 2003 8:49 UTC (Tue) by dancres (guest, #4360) [Link] I tend to agree but the number of objects traversed at any point is also dependent on the algorithm chosen. Certain types of GC will be far worse in respect of this behaviour than others.I'd sure be interested in hearing Rik "name and shame" the various VM implementations.
Rik van Riel's Talk Posted Jul 30, 2003 20:06 UTC (Wed) by stuart2048 (subscriber, #6241) [Link] For years I have dreamt of a Java environment that, instead ofrelying on the OS to blindly manage pages, integrates the paging system directly into the object heap. So that the Java VM pages objects, or parts of objects that aren't being used (but are not quite garbage yet). Of course I expect some of you will tell me this was implemented
RE: Rik van Riel's Talk Posted Oct 11, 2003 17:32 UTC (Sat) by Zakaelri (guest, #15087) [Link] Someone needs to write a "Kernel Hacking for Dummies" book to help out us application programmers. :-) I've picked up bits and pieces on the net, but some of them are probably assembled in the wrong order. How in the world do you learn this stuff?Well, simple google search got me these few links on Kernel Hacking: http://www.kernelhacking.org/ http://www.linuxchix.org/content/courses/kernel_hacking/ And I could swear there was an O'Reilly book on it too... but I can't find the listing on their site. I would assume that many of the people who do kernel hacking have been in the field for quite some time, and that they have learned about kernel hacking from either college or a *nix-related job somewhere around the line.
Posted Jul 29, 2003 8:26 UTC (Tue) by pjdc (guest, #6906) [Link] The paper accompanying Rik's talk is here.
Link to Rik van Riel's paper Posted Jul 29, 2003 15:08 UTC (Tue) by tjc (subscriber, #137) [Link] Thank you!
Keynote talk Posted Jul 31, 2003 17:37 UTC (Thu) by dac (subscriber, #9260) [Link] I thought Rusty's keynote address was excellent. Does anyone remember the 5 or 6 applications that he said were key to kernel development? Unfortunately his talk is not in the proceedings.
Keynote talk Posted Jul 31, 2003 21:03 UTC (Thu) by mulix (subscriber, #3487) [Link] valgrindccache distcc gprof and two others I don't recall.
Keynote talk Posted Aug 1, 2003 9:22 UTC (Fri) by yzhuang (guest, #3178) [Link] gperfqemu rsync valgrind ccache distcc I guess it is Rusty's personal interest...
Keynote talk Posted Aug 1, 2003 12:50 UTC (Fri) by dac (subscriber, #9260) [Link] Thanks.Are there others people find valuable?
Keynote talk Posted Aug 4, 2003 23:00 UTC (Mon) by daniel (subscriber, #3181) [Link] "Are there others people find valuable?"zinf
Keynote talk Posted Aug 7, 2003 8:09 UTC (Thu) by Lavonardo (subscriber, #13164) [Link] Here's the URL for Rusty's slides: http://ozlabs.org/~rusty/ols-2003-keynote/ols-keynote-2003.html.
Keynote talk Posted Aug 7, 2003 13:16 UTC (Thu) by cspalletta (subscriber, #12402) [Link] On slide 4 from the OLS Keynote Address, Rusty Russell says:> Read the Original TDB Source code.. It seems to me I should know what TDB is but somehow I can't remember.
Keynote talk Posted Aug 7, 2003 16:23 UTC (Thu) by guest1 (guest, #13780) [Link] http://sourceforge.net/projects/tdb/
MP3s of the speeches Posted Aug 4, 2003 20:41 UTC (Mon) by ebresie (guest, #13090) [Link] I like the proceedings provided for this iteration of the symposium, but was curious if there will be mp3 versions of the speeches like what seems to exist for past years.
Why doesn't my laptop suspend? Posted Aug 8, 2003 22:24 UTC (Fri) by otterley (guest, #3500) [Link] Lack of consistent suspend/resume features isn't the only thing missing from Linux laptop support. Also missing is the ability to adequately deal with docking stations. Windows has the capability to change hardware profiles -- not quite on-the-fly, but definitely across a suspend/resume cycle -- and Linux has no such capability at present.Lots of user-level support will be needed to handle this too. For example, the window system will have to handle a display connection and disconnection -- when the laptop is docked, it will likely be plugged into a monitor with a higher resolution and different H/V frequency ranges. All without shutting down and restarting the X server. As you can see, we have a long way to go.
|
Copyright © 2003, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds
Powered by Rackspace Managed Hosting.