Leading items

Development process latency

Your editor recently reached a point where a replacement of his main desktop system became imperative. The old one was showing certain signs of hardware flakiness, coupled with a basic inability to run some of the software your editor is trying to play with for a future article. Let's just say that the grumpy editor was becoming more so than usual. But, being a good US citizen, your editor knows how to deal with a bad mood: go shopping. So it seemed like maybe a good time to put a bit of money where the keyboard was and have a look at those Ubuntu-based systems being sold by Dell.

That plan did not go too far, unfortunately; it would seem that those systems, while seeming like nice boxes, contain NVIDIA graphics adapters. Buying hardware which needs proprietary drivers was most emphatically not on the agenda. Fortunately, it was not necessary to look too far to find a no-system-installed box with integrated Intel graphics. Said box is now being used to write this article; the simple difference in noise levels between this one and its predecessor is enough to make your editor almost cheerful.

The Intel chipset in this box, as it turns out, is quite new, to the point that not all distributions support it. But Fedora 8 was able to use everything installed on this box from the very beginning, and some early experiments showed that Ubuntu 7.10 was just as happy. Once upon a time, full support for very-new hardware was a rare surprise. Now, one can almost just take it for granted. This happy result comes from a combination of factors:

• The hardware vendor (Intel) is strongly committed to shipping free drivers for its products as soon as those products become available. Your editor can now definitively say that this policy is responsible for at least one sale for that vendor.

• The "new" (since 2005) kernel development model is strongly focused on a short release cycle and getting code out to users quickly. Prior to 2.6.x, new kernel code could take years to get into an official stable kernel release. Now three months is sufficient in many cases. So Intel's drivers became immediately available with no need to hassle with backports, driver disks, or other such inconveniences.

• Contemporary distributions, at least outside of the "enterprise" category, have gotten very good at packaging, integrating, and stabilizing leading-edge software. So those fresh kernels, along with a lot of other goodies, end up in a supported distribution surprisingly quickly.

The end result is that, increasingly often, things Just Work. It is hard to get grumpy about that.

Of course, not everything Just Worked. Many years ago, your editor spent some hours experimenting with the available bitmap fonts for X in the search for the one which provided the optimal combination of information density and readability. The resulting choice (one of the 75dpi Adobe Courier fonts) served well for a long time, even as various components on the desktop moved to more sophisticated font engines. In recent times, only Emacs was still using that font. Emacs is an important part of your editor's desktop, however, so this one remaining user was a crucial one.

That font vanished when the new system came in, with the result that editor windows could no longer simultaneously fit into their assigned screen space and provide highly readable text. Your editor became grumpy again.

What followed was a brief effort to figure out where the special font had gone, and why it did not appear to render the same way even when the requisite font files were brought over from the old system. But even your editor, who can be somewhat slow on the uptake, eventually asked himself: why, exactly, was it necessary to chase down bitmapped fonts from the early 1990's - fonts which he first used on a diskless Sun 3 system - in 2007?

In fact, it's not necessary to mess with those archaic fonts - as long as one isn't tied to the concept of stable releases. Support for the Xft font library in Emacs exists, and has for a while; here's a description in a weblog entry from 2005. This support is, even, in the Emacs code repository on Savannah. But it is not in the Emacs 22 release. It's not even in the development trunk yet.

With a bit of digging, your editor found this page which describes how to check out and build a version of Emacs with proper font support. The results are striking. Here's what standard emacs looks like:

And here is what the Xft-enabled version can do instead:

Your editor is wishing he had investigated this code some time ago; perhaps it would not have been necessary to buy those new, stronger eyeglasses after all.

Building crucial tools directly from development repositories brings a certain thrill; it's part of the free software experience. This version of Emacs has not, yet, exploded in real use. But, all of that notwithstanding, there is something warm, fuzzy, and comforting about getting things like text editors in a stable, supported form from one's distributor. So one might well wonder: when are we likely to see Emacs 23, which will contain the Xft support (along with a lot of other things like proper Unicode support, the multi-term patches, etc.), in a stable form?

The history here is not encouraging: the Emacs 22 release was five years in the making. Richard Stallman, who still keeps a firm hand on Emacs development, is famously averse to making guesses about release dates, so there is little point in asking him when the next release might happen. But it is worth noting that there has been no public discussion of release timelines, or of any desire to tweak the process to get the next version out in less than five years. There is some very nice code sitting in the unicode-2 branch of the GNU Emacs repository; it has been there for a while, but most users may well not see it before the end of this decade.

Different free software projects have different management styles, and nobody would argue that things should be otherwise. Experience has shown that each project needs to develop in its own way. But experience has also shown, over quite a few years now, that confining useful code to development repositories for years on end brings little benefit to anybody. There is value in getting features into stable releases and out where people can make use of them.

Comments (53 posted)

Book review: Linux System Programming

"System programming" is not easily defined, but is typically considered to consist of programming at a lower level than regular application programming. As Robert Love points out in the introductory chapter of Linux System Programming, there is no technical difference between the two – the same system calls are used – it is more of a difference between programs that implement the infrastructure and programs that use it. Programmers faced with either task will find that understanding how to best use the system call interface is very important. Love sets out to provide that understanding in his book.

The book is organized into ten chapters: an introduction, three on I/O, two on process management, and one on each of file and directory handling, memory management, signals, and time handling. Each chapter does a good job of covering the subject matter at a level that will help programmers make good choices in the various trade-offs available. The main focus of each chapter is the system calls that Linux provides to perform tasks specific to that area.

The history of each call is described, along with information about which members of the UNIX family make it available, so that the right choices can be made for portability. Also, various historical (perhaps vestigial is more accurate) calls are documented, with readers being warned away from using them. Each call itself is given a treatment similar to a man page, but with greater detail. Where the book really shines is in its comparisons of "similar" system calls.

The trade-offs between using select() and poll() or the advantages and disadvantages of using mmap() vs. traditional file I/O mechanisms are just two of the comparisons presented. For example, after listing five bulleted advantages of poll(), select() gets its due:

This is the kind of information that only comes with experience; this book will help a programmer get to that point more quickly. Even for experienced programmers, the comparisons will help crystallize some thoughts that have been floating around. It is definitely one of the better features of the book.

The book is not without its faults, though, especially in the example code. For each system call, a small example of calling it is provided, but the code snippets are simplistic and do not really provide much meat. There are very few code examples that tie together the various concepts. Had Love done that, there might have been complaints about the size of the resulting book, but the benefit to budding system programmers would be huge.

There are other problems with the book; for instance, the pirate motif in the examples did not seem to provide anything useful. More seriously, some of the major problems faced by system programmers: race conditions, concurrent data access synchronization, reentrant code, etc. were not covered in much detail. These topics are certainly something a system programmer will need to understand, but they will have to be found elsewhere.

The back cover of the book describes it as "an insider's guide to writing smarter, faster code" – it lives up to some of that, but not all. It is a useful book, however, that will find a home on the bookshelf of many Linux programmers. For those who are relatively new to the topic, there will be a wealth of information. But, even for those who are old hands, there will be useful tidbits, system calls that had escaped notice, and lots of reference material.

Comments (9 posted)

Enterprise realtime and cooperative development

At the end of November, LWN posted a pointer to Novell's announcement for its SUSE Linux Enterprise Realtime offering. The resulting comments were surprisingly negative. Some readers took exception to the language of the release - though it really is just the standard tortured English which is seemingly required for press releases. But others question the need for realtime response in "enterprise" settings. Anybody who is still wondering about the value of that product will be doubly confused now that Red Hat has announced a realtime distribution service of its own. Clearly somebody sees a need for deterministic response in big corporate deployments.

What's going on here is that corporate operations are, increasingly, being run by automated systems. One immediate example is in the financial trading field, where automated systems execute customer trades and, often, make the decisions to perform the trades in the first place. Often the conditions that make a particular trade advantageous last for very short periods of time - perhaps only as long as it takes for the first interested party to arrive on the scene. So predictably fast response to trading decisions is an absolute requirement. Losing too many milliseconds in the execution of an order can cost real money.

It does not take much imagination to see that, as these systems become more capable, more corporate dealings will happen via automatic agents which require lightning-fast response. So enterprise realtime has the look of a growth industry. It's not surprising that the two companies most interested in selling Linux-related services into the enterprise market have announced offerings within a week of each other.

What is surprising is the amount of silly sniping which has come with these releases. Consider this quote from the Red Hat side:

Or from the Novell side:

Despite their competitive relationship, Linux distributors have traditionally dealt with each other in a friendly, even cooperative manner. At the development level, things are still that way: developers for a given project work together and only very rarely does anybody care about who a given developer's employer is. Developers, it seems, are more polite than managers and PR people.

So who is forking the kernel? In fact, both distributors will be shipping something which is pretty far from the mainline. Back in October, LWN looked at the contents of the realtime tree, finding some 400 patches which have not yet made it into the mainline. Anybody who is shipping a true realtime kernel will have to include the bulk of those patches, and probably some others as well. In recent years, much work has been done to enable distributors to ship kernels which are much closer to the mainline, but these realtime offerings are a step in the opposite direction. They are, for all practical purposes, forked kernels.

That statement should not be taken as a criticism; there is no other way to ship realtime Linux at this point. While much of the realtime code has been merged, some of the deepest, most necessary components remain outside of the mainline. The process of getting those patches merged has taken quite a bit longer than anybody would have expected; among other things, some of the core realtime developers have been distracted by small side projects like the i386/x86_64 architecture merger. Until the process of [PULL QUOTE: Every attempt to take Unix and add hard real time to it has been a failure.
(Larry McVoy, 2004). END QUOTE] getting the realtime patches into the mainline runs its course - something which could happen over the next year - anybody shipping realtime distributions will necessarily have to roll their own kernels.

More than almost any other area of kernel development, the realtime code has been the subject of recurring debates over who deserves the credit for the work. See this LWN article from 2005 for an example. This time around, Red Hat would appear to be claiming ownership of the realtime work. In fact, much of this work, including the crucial low-level preemption work which got the current realtime effort going, was done at Red Hat. But other components have come from companies like MontaVista, Linutronix, TimeSys, and, yes, Novell (and others, of course). For these two companies to be arguing about credit is a little silly; both are clearly significant contributors to the kernel (and beyond).

We may see more of this kind of talk, though. This market looks like it could be big, so the companies working in that area are going to make a serious effort to be successful there. The result may well be that Linux ends up as the dominant system in the fast-moving, agent-driven world where much of corporate operations appears to be heading. That cause will be helped, though, if the relevant managers and spokespeople take a clue from the developers who are making all of this work actually happen. We are all building this system together; pointless mud slinging can only get in the way.

Comments (35 posted)

Page editor: Jonathan Corbet
Next page: Security>>