LWN.net Weekly Edition for May 15, 2008

Debian, OpenSSL, and a lack of cooperation

A rather nasty security hole in the Debian OpenSSL package has generated a lot of interest—along with a fair amount of controversy—amongst Linux users. The bug has been lurking for up to two years in Debian and other distributions, like Ubuntu, based on it. There are a number of lessons to be learned here about distributions and projects working together or, as in this case, failing to work together.

Back in April 2006, a Debian user reported a problem using the OpenSSL library with valgrind, a tool that can check programs for memory access problems. It was reporting that OpenSSL was using uninitialized memory in parts of the random number generator (RNG) code. Using memory before it is initialized to a known value is a well known way to create hard-to-find bugs, so it is not surprising that the valgrind report caused some consternation.

Debian hacker Kurt Roeckx tracked the problem down to what he thought were two offending lines of code and posted a question on the openssl-dev mailing list:

There were few responses, but they were not opposed to removing the lines, including one from Ulf Möeller using an openssl.org email address: "If it helps with debugging, I'm in favor of removing them." Unfortunately, as was discovered recently, removing one of the two lines was harmless, the other essentially crippled the RNG so that OpenSSL-generated cryptographic keys were easy to predict. (For more technical details on the bug and what should be done to respond to it, see our article on this week's Security page.)

It turns out, at least according to OpenSSL core team member Ben Laurie, that openssl-dev is not for discussing development of OpenSSL. That may be true in practice, but the OpenSSL support web page describes it as: "Discussions on development of the OpenSSL library. Not for application development questions!" In addition, the address suggested by Laurie (openssl-team-AT-openssl.org) does not appear in any of the OpenSSL documentation or web pages. If it wasn't the right place, it would seem that the OpenSSL developers could have provided a helpful pointer to the right address, but that did not occur.

It probably was not clear that Roeckx was asking the questions in an official Debian capacity, nor that he was planning to change the Debian package based on the answer to his questions. As Laurie rightly points out, he should have submitted a patch, proposing that it be accepted into the upstream OpenSSL codebase. That probably would have garnered more attention, even if it was only posted to openssl-dev. It seems very unlikely that the patch in question would have ever made it into an OpenSSL release.

It is in the best interests of everyone, distributions, projects, and users, for changes made downstream to make their way back upstream. In order for that to work, there must be a commitment by downstream entities—typically distributions, but sometimes users—to push their changes upstream. By the same token, projects must actively encourage that kind of activity by helping patch proposals and proposers along. First and foremost, of course, it must be absolutely clear where such communications should take place.

Another recently reported security vulnerability also came about because of a lack of cooperation between the project and distributions. It is vital, especially for core system security packages like OpenSSH and OpenSSL, that upstream and downstream work very closely together. Any changes made in these packages need to be scrutinized carefully by the project team before being released as part of a distribution's package. It is one thing to let some kind of ill-advised patch be made to a game or even an office application package that many use; SSH and SSL form the basis for many of the tools used to protect systems from attackers, so they need to be held to a higher standard.

Another of Laurie's points, which also bears out the need for a higher standard, is the timing of the check-in to a public repository when compared to that of the advisory. Any alert attacker could have made very good use of the five or six day head start, they could have gotten by monitoring the repository, to exploit the vulnerability. While it is certainly possible that some of malicious intent already knew about the flaw, though no exploits have been reported, alerting potential attackers to this kind of hole well in advance of alerting the vulnerable users is unbelievably bad security protocol.

This is the kind of problem that could have been handled quickly and quietly by all concerned. All affected distributions—though it might be difficult to list all of the Debian-derived distributions out there—could have been contacted so that the advisory and updates to affected packages could have been coordinated. One of these days, one of these problems is going to give Linux a security black eye unless the community can do a better job of working together.

Comments (16 posted)

Release synchronization

One could quibble with this claim in a number of ways, but it is true that Ubuntu got out a release designed to be supported for a number of years, and they did it when they said they would. That, of course, is only part of the job; now they have to follow through on that little promise of supporting this distribution into 2011. The initial signs are good: Ubuntu's support thus far has been solid, and it would appear that the distribution will not be going away anytime soon.

One might well question whether the timely release of 8.04 is noteworthy. As a community we are increasingly spoiled; an increasingly large number of projects and distributions manage to get out regular releases on a reasonably predictable schedule. Even kernel releases, once known to slip for a year or more, are now predictable to within a couple of weeks. Now that free software releases are rather more predictable and reliable than, say, airline departures, why is the Ubuntu 8.04 release noteworthy?

The answer is the long-term support commitment. Theoretically, a distribution intended for this sort of long lifetime will have had a degree of extra care put into its preparation. Important components will have been given extra time to stabilize so that the distribution will be more reliable from the outset. Some thought will have gone into the selection of packages shipped with an emphasis on supportability over the long term. The whole process requires more effort and a higher degree of assurance that all of the pieces are truly ready.

The degree to which Ubuntu has done all of that work should become clear over time. Certainly the software selected for this release is rather less seasoned than the packages found in a Red Hat or SUSE enterprise release. But "older" does not necessarily mean "better" or "more stable," so the real proof will be in how well this distribution holds up for the next three years.

Meanwhile, Mr. Shuttleworth has already stated that the next long-term support release will be happening in April, 2010. Ubuntu's success with 8.04, he says, allows this commitment to be made almost two years in advance. There is, however, a possibility that things could change:

This idea is not new, but Mr. Shuttleworth seems to be particularly attached to it. There is no doubt that there would be advantages to aligning schedules in this way. The kernel developers, who have been known to make a special effort for a release destined to be used by a major enterprise distributor, could focus especially hard on a stable release knowing that it would be widely used. Higher-level projects could do the same. The distributors could also, perhaps, find a way to collaborate on the long-term maintenance of these components, rather than duplicating the effort of backporting patches into older code. Perhaps they could even get together for a joint release party, saving even more money.

Or perhaps this is all a nice idea which fails to survive its encounter with reality. Enterprise distribution releases tend to be highly-publicized events. Ubuntu might be happy to share its limelight with the larger distributors, but that feeling might not be reciprocated on the other side. It is hard to imagine Red Hat or Novell wanting to have their big enterprise distribution release be just one of many happening during the same month.

It is also hard to see Ubuntu making an agreement with the enterprise distributors which specifies both a release date and the versions of the major components. 8.04 released with the 2.6.24 kernel, which was almost exactly three months old at the time. Red Hat Enterprise Linux 5 released in mid-March, 2007, when the 2.6.20 kernel was current - but Red Hat shipped the six-month-old 2.6.18 kernel instead. Aligning schedules would require more than picking a date; it would also require adopting similar stabilization periods. It is far from clear that Ubuntu would want to fall that far behind the leading edge for the sake of alignment.

And, frankly, it's hard to imagine Debian making a credible commitment (within one month) to a release date at all.

So the aligned schedules for enterprise distributions seems like a hard sell. A better approach might be to try to wean these distributions off the "freeze and backport" model of support; this model is expensive to sustain, brings risks of its own, and doesn't always fit the needs of enterprise customers.. If the enterprise distributors were able to track more current software - rather than backporting pieces of it into older software - better alignment of releases might just come naturally.

Comments (8 posted)

Distributed bug tracking

But, while any developer can now commit changes to a project while strapped into a seat in a tin can flying over the Pacific Ocean, that developer generally cannot simultaneously work with the project's bug database. Committing changes and making bug tracker changes are activities which often go together, but bug tracking systems remain strongly in the centralized mode. Our ocean-hopping developer can commit a dozen fixes, but updating the related bug entries must wait until the plane has landed and network connectivity has been found.

There are a number of projects out there which are trying to change this situation through the creation of distributed bug tracking systems. These developments are all in a relatively early state, but their potential - and limitations - can be seen.

One of the leading projects in this area is Bugs Everywhere, which has recently moved to a new home with Chris Ball as its new maintainer. Bugs Everywhere, like the other systems investigated by your editor, tries to work with an underlying distributed source code management system to manage the creation and tracking of bug entries. In particular, Bugs Everywhere creates a new directory (called .be) in the top level of the project's directory. Bugs are stored as directories full of text files within that directory, and the whole collection is managed with the underlying SCM.

The advantages to an approach like this are clear. The bug database can now be downloaded along with the project's code itself. It can be branched along with the code; if a particular branch contains a fix for a bug, it can also contain the updated bug tracker entry. That, in turn, ensures that the current bug tracking information will be merged upstream at exactly the same time as the fix itself. Contemporary projects are characterized by large numbers of repositories and branches, each of which can contain a different set of bugs and fixes; distributing the bug database into these repositories can only help to keep the code and its bug information consistent everywhere.

There are also some disadvantages to this scheme, at least in its current form. Changes to bug entries don't become real until they are committed into the SCM. If a bug is fixed, committing the fix and the bug tracker update at the same time makes sense; in cases where one is trying to add comments to a bug as part of an ongoing conversation the required commit is just more work to do. That fact that, in git at least, one must explicitly add any new files created by the bug tracker (which have names like 12968ab9-5344-4f08-9985-ef31153e504f/comments/97f56c43-4cf2-4569-9ef4-3e8f2d9eb1fe/body) does not help the situation.

Beyond that, tracking bugs this way creates two independent sets of metadata - the bug information itself, and whatever the developer added when committing changes. There is currently no way of tying those two metadata streams together. Then, there is the issue of merging. Bugs Everywhere appears to reflect some thought about this problem; most changes involve the creation of new, (seemingly) randomly-named files which will not create conflicts at merge time. It did not take long, however, for your editor to prove that changing the severity of a bug in two branches and merging the result creates a conflict which can only be resolved by hand-editing the bug tracker's files. Said files are plain text, but that is less comforting than one might think.

[PULL QUOTE: All of this can make distributed bug tracking look like a source of more work for developers, which is not the path to world domination. END QUOTE] All of this can make distributed bug tracking look like a source of more work for developers, which is not the path to world domination. What is needed, it seems, is a combination of more advanced tools and better integration with the underlying SCM. Bugs Everywhere, by trying to work with any SCM, risks not being easily usable with any of them.

A project which is trying for closer integration is ticgit, which, as one might expect, is based on git. Ticgit takes a different approach, in that there are no files added to the project's source tree, at least not directly; instead, ticgit adds a new branch to the SCM and stores the bug information there. That allows the bug database to travel with the source (as long as one is careful to push or pull the ticgit branch!) while keeping the associated files out of the way. Ticgit operations work on the git object database directory, so there is no need for separate commit operations. On the other hand, this approach loses the ability to have a separate view of the bug database in each branch; the connection between bug fixes and bug tracker changes has been made weaker. This is something which can be fixed, and it would appear (from comments in the source) that dealing with branches is on the author's agenda.

Ticgit clearly has potential, but even closer integration would be worthwhile. Wouldn't it be nice if a git commit command would also, in a single operation, update the associated entry in the bug database? Interested developers could view a commit which is alleged to fix a bug without the need for anybody to copy commit IDs back and forth. Reverting a bugfix commit could automatically reopen the bug. And so on. In the long run, it is hard to see how a truly integrated, distributed bug tracker can be implemented independently of the source code management system.

There are some other development projects in this area, including:

• Scmbug is a relatively advanced project which aims "to solve the integration problem once and for all." It is not truly a distributed bug tracker, though; it depends on hooks into the SCM which talk to a central server. Regardless, this project has done a significant amount of thinking about how bug trackers and source code management systems should work together.

• DisTract is a distributed bug tracker which works through a web interface. To that end, it uses a bunch of Firefox-specific JavaScript code to run local programs, written in Haskell, which manipulate bug entries stored in a Monotone repository. Your editor confesses that he did not pull together all of the pieces needed to make this tool work.

• DITrack is a set of Python scripts for manipulating bug information within a Subversion repository. It is meant to be distributed (and, eventually, "backend-agnostic"), but its use of Subversion limits how distributed it can be for now.

• Ditz is a set of Ruby scripts for manipulating bug information within a source code management system; it has no knowledge of the SCM itself.

As can be seen, there is no shortage of work being done in this area, though few of these projects have achieved a high level of usability. Only Scmbug has been widely deployed so far. A few of these projects have the potential to change the way development is done, though, once various integration and user interface issues are addressed.

There is one remaining problem, though, which has not been touched upon yet. A bug tracker serves as a sort of to-do list for developers, but there is more to it than that. It is also a focal point for a conversation between developers and users. Most users are unlikely to be impressed by a message like "set up a git repository and run these commands to file or comment on a bug." There is, in other words, value in a central system with a web interface which makes the issue tracking system accessible to a wider community. Any distributed bug tracking system which does not facilitate this wider conversation will, in the end, not be successful. Creating a distributed tracker which also works well for users could be the biggest challenge of them all.

Comments (43 posted)

Page editor: Jonathan Corbet

Security

Debian vulnerability has widespread effects

The recent Debian advisory for OpenSSL could lead to predictable cryptographic keys being generated on affected systems. Unfortunately, because of the way keys are used, especially by ssh, this can lead to problems on systems that never installed the vulnerable library. In addition, because the OpenSSL library is used in a wide variety of services that require cryptography, a very large subset of security tools are affected. This is a wide-ranging vulnerability that affects a substantial fraction of Linux systems.

For a look at the chain of errors that led to the vulnerability, see our front page article. Here, we will concentrate on some of the details of the code, the impact of the vulnerability, and what to do about it.

An excellent tool for finding memory-related bugs, Valgrind was used on an application that used the OpenSSL library. It complained about the library using uninitialized memory in two locations in crypto/rand/md_rand.c:

    247:
            MD_Update(&m,buf,j);

    467:
    #ifndef PURIFY
            MD_Update(&m,buf,j); /* purify complains */
    #endif

The first is contained in the ssleay_rand_add() function, which is normally called via the RAND_add() function. It adds the contents of the passed in buffer to the entropy pool of the pseudo-random number generator (PRNG). The other is contained in ssleay_rand_bytes(), normally called via RAND_bytes(), which is meant to return random bytes. It adds the contents of the passed in buffer—before filling it with random bytes to return—to the entropy pool as well. The major difference is that removing the latter might marginally reduce the entropy in the PRNG pool, while removing the former effectively stops any entropy from being added to the pool.

For both RAND_add() and RAND_bytes(), the buffer that gets passed in may not have been initialized. This was evidently known by the OpenSSL folks, but remained undocumented for others to trip over later. The "#ifndef PURIFY" is a clue that someone, at some point, tried to handle the same kind of problem that Valgrind was reporting for the similar, but proprietary, Purify tool. While it isn't necessarily wrong to add these uninitialized buffers to the PRNG pool, it is something that tools like Valgrind will rightly complain about. Since it is dubious whether it adds much in the way of entropy, while constituting a serious hazard for uninitiated, some kind of documentation in the code would seem mandatory.

The major response from the OpenSSL team seems to be from core team member Ben Laurie's weblog, where he has a rant entitled "Vendors Are Bad For Security". In it, and its follow-up, he makes some good points about mistakes that were made, while seeming to be unwilling for OpenSSL to take any share of the blame.

The end result is that OpenSSL would create predictable random numbers, which would then result in predictable cryptographic keys. According to the advisory:

A program that can detect some weak keys has also been released. It uses 256K hash values to detect the bad keys, which would imply 18-bits of entropy in the PRNG pool of vulnerable OpenSSL libraries. By using hashes of the keys in the detection program, the authors do not directly give away the key values that get generated, but it should not be difficult for an attacker to generate and use that list.

For affected Debian-derived systems, the cleanup is relatively straightforward, if painful. The SSLkeys page on the Debian wiki has specific information on how to remove weak keys along with how to generate new ones for a variety of services affected. Obviously, none of those steps should be taken until the OpenSSL package itself has been upgraded to a version that fixes the hole.

A bigger problem may be for those installations based on distributions that were not directly affected because they did not distribute the vulnerable OpenSSL library. Those machines may very well have weak keys installed in user accounts as ssh authorized_keys. A user who generated a key pair on some vulnerable host may have copied the public key to a host that was not vulnerable. This would allow an attacker to access the account of that user by brute forcing the key from the 256K possibilities.

Because of that danger, the Debian project suspended public key authentication on debian.org machines. In addition, all passwords were reset because of the possibility that an attacker could have captured them by decrypting the ssh traffic using one of the weak keys. One would guess that debian.org machines would have a higher incidence of weak keys, but any host that allows users to use ssh public key authentication is potentially at risk.

The weak key detector (dowkd) has some fairly serious limitations:

In order to ensure that there are no weak keys installed as public keys on other hosts, it may be necessary to remove all authorized_keys (and/or authorized_keys2) entries for all users. It may also be wise to set all passwords to something unknown. Until that is done, there still remains a chance that a weak key may allow access to an attacker. It is a unpleasant task that needs to be done for those who administer a multi-user system.

Comments (31 posted)

Security news

Brute-Force SSH Server Attacks Surge (InformationWeek)

Comments (37 posted)

Mozilla ships a compromised extension

Comments (6 posted)

Cryptographic weakness on Debian systems

Comments (111 posted)

Tech Insight: Finding & Prioritizing Web Application Vulnerabilities (Dark Reading)

Comments (none posted)

New vulnerabilities

bugzilla: multiple vulnerabilities

Comments (none posted)

cdf: buffer overflow

Comments (none posted)

egroupware: denial of service

Comments (none posted)

gforge: temporary file vulnerability

Comments (none posted)

firebird: information disclosure

Comments (1 posted)

imagemagick: heap-based buffer overflows

Comments (none posted)

inspIRCd: buffer overflow

Comments (none posted)

libid3tag: infinite loop

Comments (none posted)

libvorbis: multiple vulnerabilities

Comments (none posted)

licq: denial of service

Comments (none posted)

ltsp: multiple vulnerabilities

Comments (none posted)

moinmoin: privilege escalation

Comments (none posted)

nagios: cross-site scripting

Comments (none posted)

openssl: predictable random number generator

Comments (none posted)

php5: multiple vulnerabilities

Comments (none posted)

php: PATH_TRANSLATED miscalculation

Comments (none posted)

rdesktop: multiple vulnerabilities

Comments (none posted)

sarg: stack buffer overflows

Comments (none posted)

sipp: arbitrary code execution

Comments (none posted)

X11 terms: privilege escalation

Comments (none posted)

xen: multiple vulnerabilities

Comments (none posted)

zoneminder: arbitrary code execution

Comments (none posted)

Page editor: Jake Edge

Kernel development

Release status

Kernel release status

The current -mm tree is 2.6.26-rc2-mm1. Recent changes to -mm include a rebasing on top of the linux-next tree, a memory initialization debugging framework, the removal of the v850 port, a new set of system calls with additional flags arguments (see below), a WARN() macro which takes printk()-style arguments, ext4 online defragmentation, ext4 FIEMAP support, and the OMFS filesystem.

The current stable 2.6 kernel is 2.6.25.3, released on May 9. This release came out earlier than planned due to the inclusion of a couple of security fixes.

As of this writing, the 2.6.25.4 update - containing a few dozen fixes - is in the review process.

Comments (none posted)

Kernel development news

Quotes of the week

Comments (1 posted)

The big kernel lock strikes again

The AIM benchmark attempts to measure system throughput by running a large number of tasks (perhaps thousands of them), each of which is exercising some part of the kernel. Yanmin Zhang reported that his AIM results got about 40% worse under the 2.6.26-rc1 kernel. He took the trouble to bisect the problem; the guilty patch turned out to be the generic semaphores code. Reverting that patch made the performance regression go away - at the cost of restoring over 7,000 lines of old, unlamented code. The thought of bringing back the previous semaphore implementation was enough to inspire a few people to look more deeply at the problem.

It did not take too long to narrow the focus to the BKL, which was converted to a semaphore a few years ago. That part of the process was easy - there aren't a whole lot of other semaphores left in the kernel, especially in performance-critical places. But the BKL stubbornly remains in a number of core places, including the fcntl() system call, a number of ioctl() implementations, the TTY code, and open() for char devices. That's enough for a badly-performing BKL to create larger problems, especially when running VFS-heavy benchmarks with a lot of contention.

Ingo Molnar tracked down the problem in the new semaphore code. In short: the new semaphore code is too fair for its own good. When a semaphore is released, and there is another thread waiting for it, the semaphore is handed over to the new thread (which is then made runnable) at that time. This approach ensures that threads obtain the semaphore in something close to the order in which they asked for it.

The problem is that fairness can be expensive. The thread waiting for the semaphore may be on another processor, its cache could be cold, and it might be at a low enough priority that it will not even begin running for some time. Meanwhile, another thread may request the semaphore, but it will get put at the end of the queue behind the new owner, which may not be running yet. The result is a certain amount of dead time where no running thread holds the semaphore. And, in fact, Yanmin's experience with the AIM benchmark showed this: his system was running idle almost 50% of the time.

The solution is to bring in a technique from the older semaphore code: lock stealing. If a thread tries to acquire a semaphore, and that semaphore is available, that thread gets it regardless of whether a different thread is patiently waiting in the queue. Or, in other words, the thread at the head of the queue only gets the semaphore once it starts running and actually claims it; if it's too slow, somebody else might get there first. In human interactions, this sort of behavior is considered impolite (in some cultures, at least), though it is far from unknown. In a multiprocessor computer, though, it makes the difference between acceptable and unacceptable performance - even a thread which gets its lock stolen will benefit in the long run.

Interestingly, the patch which implements this change was merged into the mainline, then reverted before 2.6.26-rc2 came out. The initial reason for the revert was that the patch broke semaphores in other situations; for some usage patterns, the semaphore code could fail to wake a thread when the semaphore became available. This bug could certainly have been fixed, but it appears that things will not go that way - there is a bit more going on here.

What is happening instead is that Linus has committed a patch which simply turns the BKL into a spinlock. By shorting out the semaphore code entirely, this patch fixes the AIM regression while leaving the slow (but fair) semaphore code in place. This change also makes the BKL non-preemptible, which will not be entirely good news for those who are concerned with latency issues - especially the real time tree.

The reasoning behind this course of action would appear to be this: both semaphores and the BKL are old, deprecated mechanisms which are slated for minimization (semaphores) or outright removal (BKL) in the near future. Given that, it is not worth adding more complexity back into the semaphore code, which was dramatically simplified for 2.6.26. And, it seems, Linus is happy with a sub-optimal BKL:

So the end result of all this may be a reinvigoration of the effort to remove the big kernel lock from the kernel. It still is not something which is likely to happen over the next few kernel releases: there is a lot of code which can subtly depend on BKL semantics, and there is no way to be sure that it is safe without auditing it in detail. And that is not a small job. Alan Cox has been reworking the TTY code for some time, but he has some ground to cover yet - and the TTY code is only part of the problem. So the BKL will probably be with us for a while yet.

Comments (5 posted)

Getting a handle on caching

As readers of What every programmer should know about memory will have learned, the caching mechanisms used by contemporary processors are crucial to the performance of the system. Memory is slow; without caching, systems will run much slower. There are situations where caching is detrimental, though, so the hardware must provide mechanisms which allow for control over caching with specific ranges of memory. With 2.6.26, Linux is (rather belatedly) starting to catch up with the current state of the art on x86 hardware; that, in turn, is bringing some changes to how caching is managed.

It is good to start with a definition of the terms being used. If a piece of memory is cachable, that means:

• The processor is allowed to read that memory into its cache at any time. It may choose to do so regardless of whether the currently-executing program is interested in reading that memory. Reads of cachable memory can happen in response to speculative execution, explicit prefetching, or a number of other reasons. The CPU can then hold the contents of this memory in its cache for an arbitrary period of time, subject only to an explicit request to release the cache line from elsewhere in the system.

• The CPU is allowed to write the contents of its cache back to memory at any time, again regardless of what any running program might choose to do. Memory which has never been changed by the program might be rewritten, or writes done by a program may be held in the cache for an arbitrary period of time. The CPU need not have read an entire cache line before writing that line back.

What this all means is that, if the processor sees a memory range as cachable, it must be possible to (almost) entirely disconnect the operations on the underlying device from what the program thinks it is doing. Cachable memory must always be readable without side effects. Writes have to be idempotent (writing the same value to the same location several times has the same effect as writing it once), ordering-independent, and size-independent. There must be no side effects from writing back a value which was read from the same location. In practice, this means that what sits behind a cachable address range must be normal memory - though there are some other cases.

If, instead, an address range is uncachable, every read and write operation generated by software will go directly to the underlying device, bypassing the CPU's caches. The one exception is with writes to I/O memory on a PCI bus; in this case, the PCI hardware is allowed to buffer and combine write operations. Writes are not reordered with reads, though, which is why a read from I/O memory is often used in drivers for PCI devices as a sort of write barrier.

A variant form of uncached access is write combining. For read operations, write-combined memory is the same as uncachable memory. The hardware is, however, allowed to buffer consecutive write operations and execute them as a smaller series of larger I/O operations. The main user of this mode is video memory, which often sees sequential writes and which offers significant performance improvements when those writes are combined.

The important thing is to use the right cache mode for each memory range. Failure to make ordinary memory cachable can lead to terrible performance. Enabling caching on I/O memory can cause strange hardware behavior, corrupted data, and is probably implicated in global warming. So the CPU and the hardware behind a given address must agree on caching.

Traditionally, caching has been controlled with a CPU feature called "memory type range registers," or MTRRs. Each processor has a finite set of MTRRs, each of which controls a range of the physical address space. The BIOS sets up at least some of the MTRRs before booting the operating system; some others may be available for tweaking later on. But MTRRs are somewhat inflexible, subject to the BIOS not being buggy, and are limited in number.

In more recent times, CPU vendors have added a concept known as "page attribute tables," or PAT. PAT, essentially, is a set of bits stored in the page table entries which control how the CPU does caching for each page. The PAT bits are more flexible and, since they live in the page table entries, they are difficult to run out of. They are also completely under the control of the operating system instead of the BIOS. The only problem is that Linux doesn't support PAT on the x86 architecture, despite the fact that the hardware has had this capability for some years.

The lack of PAT support is due to a few things, not the least of which has been problematic support on the hardware side. Processors have stabilized over the years, though, to the point that it is possible to create a reasonable whitelist of CPU families known to actually work with PAT. There have also been challenges on the kernel side; when multiple page table entries refer to the same physical page (a common occurrence), all of the page table entries must use the same caching mode. Even a brief window with inconsistent caching can be enough to bring down the system. But the code on the kernel side has finally been worked into shape; as a result, PAT support was merged for the 2.6.26 kernel. Your editor is typing this on a PAT-enabled system with no ill effects - so far.

On most systems, the BIOS will set MTRRs so that regular memory is cachable and I/O memory is not. The processor can then complicate the situation with the PAT bits. In general, when there is a conflict between the MTRR and PAT settings, the setting with the lower level of caching prevails. The one exception appears to be when one says "uncachable" and the other enables write combining; in that case, write combining will be used. So the CPU, through the management of the PAT bits, can make a couple of effective changes:

• Uncached memory can have write combining turned on. As noted above, this mode is most useful for video memory.

• Normal memory can be made uncached. This mode can also be useful for video memory; in this case, though, the memory involved is normal RAM which is also accessed by the video card.

Linux device drivers must map I/O memory before accessing it; the function which performs this task is ioremap(). Traditionally, ioremap() made no specific changes to the cachability of the remapped range; it just took whatever the BIOS had set up. In practice, that meant that I/O memory would be uncachable, which is almost always what the driver writer wanted. There is a separate ioremap_nocache() variant for cases where the author wants to be explicit, but use of that interface has always been rare.

In 2.6.26, ioremap() was changed to map the memory uncached at all times. That created a couple of surprises in cases where, as it happens, the memory range involved had been cachable before and that was what the code needed. As of 2.6.26, such code will break until the call is changed to use the new ioremap_cache() interface instead. There is also a ioremap_wc() function for cases where a write-combined mapping is needed.

It is also possible to manipulate the PAT entries for an address range explicitly:

    int set_memory_uc(unsigned long addr, int numpages);
    int set_memory_wc(unsigned long addr, int numpages);
    int set_memory_wb(unsigned long addr, int numpages);

These functions will set the given pages to uncachable, write-combining, or writeback (cachable), respectively. Needless to say, anybody using these functions should have a firm grasp of exactly what they are doing or unpleasant results are certain.

Comments (12 posted)

Extending system calls

Getting interfaces right is a hard, but necessary, task, especially when that interface has to be supported "forever". Such is the case with the system call interface that the kernel presents to user space, so adding features to it must be done very carefully. Even so, when Ulrich Drepper set out to remove a hole that could lead to a race condition, he probably did not expect all the different paths that would need to be tried before closing in on an acceptable solution.

The problem stems from wanting to be able to create file descriptors with new properties—things like close-on-exec, non-blocking, or non-sequential descriptors. Those features were not considered when the system call interface was developed. After all, many of those system calls are essentially unchanged from early Unix implementations of the 1970s. The open() call is the most obvious way to request a file descriptor from the kernel, but there are plenty of others.

In fact, open() is one of the easiest to extend with new features because of its flags argument. Calls like pipe(), socket(), accept(), epoll_create() and others produce file descriptors as well, but don't have a flags argument available. Something different would have to be done to support additional features for the file descriptors resulting from those calls.

The close-on-exec functionality is especially important to close a security hole for multi-threaded programs. Currently, programs can use fcntl() to change an open file descriptor to have the close-on-exec property, but there is always a window in time between the creation of the descriptor and changing its behavior. Another thread could do an exec() call in that window, leaking a potentially sensitive file descriptor into the newly run program. Closing that window requires an in-kernel solution.

Back in June of last year, after some false starts, Linus Torvalds suggested adding an indirect() system call, as a way to pass flags to system calls that don't currently support them. The indirect() call would apply a set of flags to the invocation of an existing system call. This would allow existing calls to remain unchanged, with only new uses calling indirect(). User space programs would be unlikely to call the new function directly, instead they would call glibc functions that handled any necessary indirect() calls.

Davide Libenzi created a sys_indirect() patch in July, but Drepper saw it as "more complex than warranted". So Drepper created his own "trivial" implementation, that was described on this page in November. It was met with a less than enthusiastic response on linux-kernel for being, amongst other things, an exceedingly ugly interface.

The alternative to sys_indirect() is to create a new system call for each existing call that needed a flags argument. This was seen as messy by some, including Torvalds, leading some kernel hackers into looking for alternatives. The indirect approach also had some other potential benefits, though, because it was seen as something that could be used by syslets to allow asynchronous system calls. No decision seemed to be forthcoming, leading Drepper to ask Torvalds for one:

To bolster his argument that sys_indirect() was the way to go, Drepper also created a patch to add some of the required system calls. He started with the socket() family, by adding socket4(), socketpair5(), and accept4()—tacking the number of arguments onto the function name a la wait3() and wait4(). Drepper's intent may not have been well served by choosing those calls as Alan Cox immediately noted that the type argument could be overloaded:

        socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC, ...)

Michael Kerrisk looked over the set of system calls that generate file descriptors, categorizing them based on whether they needed a flag argument added. He observed that roughly half of the file descriptor producing calls need not change because they could either use an overloading trick like the socket calls, the glibc API already added a flags argument, or there were alternatives available to provide the same functionality along with flags.

In response, Drepper made one last attempt to push the indirect approach, saying:

Even though the indirect approach has some good points, Torvalds liked the approach advocated by Cox, saying:

Ultimately, developers will only use these new interfaces if they can easily test for the existence of the new code. Torvalds gives an example of how that might be done using the O_NOATIME flag to open(), which has only been available since 2.6.8. It is this testability issue that makes him believe the flags-based approach is the right one:

This new approach, with a scaled down number of new system calls rather than adding a general-purpose system call extension mechanism like sys_indirect(), is now being pursued by Drepper. In the explanatory patch at the start of the series, he lays out which of the system calls will require a new user space interface: paccept(), epoll_create2(), dup3(), pipe2(), and inotify_init1(), as well as those that do not: signalfd4(), eventfd2(), timerfd_create(), socket(), and socketpair().

Drepper has already made several iterations of patches addressing most of the concerns expressed by the kernel developers along the way. There have been some architecture specific problems, but Drepper has been knocking those down as well. If no further roadblocks appear, it would seem a likely candidate for inclusion in 2.6.27.

Comments (8 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.26-rc2. (May 12, 2008)

• Andrew Morton: 2.6.26-rc2-mm1. (May 14, 2008)

• Greg Kroah-Hartman: Linux 2.6.25.3. (May 12, 2008)

• Oliver Pinter: v2.6.22.24-op1. (May 12, 2008)

Core kernel code

• Gregory Haskins: sched: core balancer. (May 13, 2008)

• Huang, Ying: [PATCH] kexec based hibernation: a prototype of kexec multi-stage load. (May 13, 2008)

• Hirofumi Nakagawa: MAZE: Mazed processes monitor. (May 13, 2008)

• Ingo Molnar: "kill the Big Kernel Lock (BKL)" tree. (May 14, 2008)

Device drivers

• Darrick J. Wong: [PATCH v3] ibmaem: New driver for power/energy/temp meters in IBM System X hardware. (May 13, 2008)

• Greg KH: HSO: add option hso driver. (May 14, 2008)

Filesystems and block I/O

• Sage Weil: Ceph distributed file system v0.2. (May 8, 2008)

• Andrew Patterson: detect online disk resize. (May 12, 2008)

• Miklos Szeredi: file locking fixes + fuse nfs export support. (May 13, 2008)

• Barry Naujok: XFS: Case-insensitive support - ASCII only. (May 13, 2008)

• Evgeniy Polyakov: POHMELFS high performance network filesystem. Transactions, failover, performance.. (May 13, 2008)

Memory management

• Mel Gorman: Guarantee faults for processes that call mmap(MAP_PRIVATE) on hugetlbfs v2. (May 8, 2008)

• Andrea Arcangeli: mmu-notifier-core v17. (May 9, 2008)

• Ross Biro: MM: Make Page Tables Relocatable -- Conditional Flush. (May 13, 2008)

• Christoph Lameter: Slab Fragmentation Reduction V12. (May 13, 2008)

• Timur Tabi: Add alloc_pages_exact() and free_pages_exact(). (May 14, 2008)

Architecture-specific

• Suresh Siddha: x86: xsave/xrstor support, ucontext_t extensions. (May 13, 2008)

• Jon Tollefson: 16G and multi size hugetlb page support on powerpc. (May 13, 2008)

Virtualization and containers

• KAMEZAWA Hiroyuki: memcg: better migration handling. (May 13, 2008)

• Balbir Singh: Add memrlimit controller (v4). (May 14, 2008)

Benchmarks and bugs

• Rafael J. Wysocki: 2.6.26-rc1-git9: Reported regressions from 2.6.25. (May 12, 2008)

Miscellaneous

• Kay Sievers: udev 121 release. (May 13, 2008)

• Alex Tsariounov: Cpuset management utility 0.9.1 release announcement. (May 13, 2008)

Page editor: Jonathan Corbet

Distributions

News and Editorials

A Talk with Fedora Project Leader Paul Frields

One thing Paul is passionate about is getting people to volunteer. There are many ways to get involved with the Fedora Project, lots of sub-projects and Special Interest Groups (SIGs) that people can join depending on their interests and talents. The Fedora Project wiki is a good starting point for finding out more. The Join Fedora page also goes into the various roles that a Fedora contributor might be suited for, with easy links to setting up a Fedora account and using the Fedora Account system. You don't have to be a programmer or a computer expert to contribute to the project.

Joining the Fedora Project is easier now than it ever was during Fedora's five year history. As a result Fedora now has over 2000 registered account holders. That includes about 350 ambassadors who promote Fedora in their local area. In addition to making it easier to become a Fedora contributor, a variety of new web applications/collaborative tools are now available for contributors. Of course all Fedora infrastructure is Free Software, available in the Fedora repository, and running on Fedora.

All registered account holders may vote in Fedora elections, which is worth noting because there is an election coming up in June.

The composition of the Fedora board was recently changed to five elected members of the nine board seats. Four of those seats will be voted on in the next election. The other board seats are appointed by Red Hat, but are not necessarily Red Hat employees. Red Hat retains some control by employing and appointing the Project Leader. Paul took a job with Red Hat when he was offered the position of Project Leader.

Paul mentioned that former Fedora Project Leader Max Spevack is moving to the Netherlands to organize and manage Fedora volunteers in Europe. Paul also mentioned that Fedora has many Brazilian contributors. Of course Red Hat employs some Fedora engineers. There are fourteen Red Hat employees working full time on Fedora, mostly acting as team leaders and organizing the volunteers. In addition all Red Hat engineers will spend some fraction of their time working on Fedora in areas where Red Hat Enterprise Linux in involved.

Some people think of Fedora as a beta for Red Hat Enterprise Linux, but its more realistic to think of Fedora as the upstream source for its enterprising cousin and spin-offs such as CentOS. So even though Fedora is a community project, Red Hat is still very involved in its development.

FUDCon (Fedora User & Developer Conference) is an event held on an irregular schedule several times per year. Some are smaller events held in conjunction with a larger event, such as the May 30, 2008 FUDCon, which will be held at LinuxTag in Berlin, Germany. Further out, there is some talk of having a mini-FUDCon at the 2009 linux.conf.au. The Boston FUDCon coming up in June, will run for several days. Co-located with the Red Hat Summit, the Boston FUDCon will feature hackfests, a barcamp and technical talks.

The Red Hat Summit will bring in Red Hat customers, and include talks about actual use cases. These talks should be interesting for Fedora developers, who will have a chance to see what people are doing with their work downstream. FUDCon is open to anyone, so stop by if there is a FUDCon in your area.

On to the just released Fedora 9 and the upcoming Fedora 10. Fedora 9 is one of the first major releases to feature KDE 4 by default. To make this work, the KDE SIG has built a compatibility library to keep KDE 3 applications running properly. For Fedora 10 Casey Dahlin is working on replacing the init system with upstart, the system developed for Ubuntu.

Some other items that we touched on briefly: Fedora maintains an open build system and works at getting patches upstream. The project also strives to cooperate with other distributions. From what I've seen, Fedora 9 looks very good, attractive and functional. Now that rawhide has moved on to Fedora 10 it will be a rough ride for at least a few days. So stick with Fedora 9, or get it from a mirror near you.

Fedora 9 is Paul's first release as Project Leader and he had a few words to add. "It's been less than five years since the first release of Fedora (back when it was called Fedora Core), and in that time Fedora has become not just a vibrant, innovative, and extremely popular Linux distribution, but also a thriving community. A community that believes that free and open source software is not just something you *use*, it's something you *do* -- something to which you *contribute*."

Comments (2 posted)

New Releases

Fedora 9 released

Full Story (comments: 10)

BLFS-6.3-rc1 has been released!

Full Story (comments: none)

Distribution News

Debian GNU/Linux

Surveying the Debian community!

Comments (none posted)

Fedora

Red Hat Board Appointments

Full Story (comments: none)

Rawhide moving on to Fedora 10

Full Story (comments: 1)

rpm.livna.org repositories for Fedora 9 (Sulphur) now available

Full Story (comments: none)

Gentoo Linux

Gentoo council meeting summary for 8 May 2008

Full Story (comments: none)

Ubuntu family

Xubuntu Meeting

Full Story (comments: none)

Distribution Newsletters

OpenSUSE Weekly News/21

Comments (none posted)

PCLinuxOS Magazine May 2008 Released

Comments (none posted)

Ubuntu Weekly Newsletter, Issue 90

Full Story (comments: none)

DistroWatch Weekly, Issue 252

Comments (none posted)

Miscellaneous Articles

Hats off to Fedora 9 (DesktopLinux)

Comments (none posted)

Fedora 9 Released with KDE 4.0.3 (KDE.News)

Comments (none posted)

CNR supports Linux Mint, adds Weatherbug (DesktopLinux)

Comments (none posted)

Distribution reviews

Top 5 Tiny Distros (TuxMachines)

Comments (none posted)

Fedora 9 - an OS that even the Linux challenged can love (The Register)

Comments (none posted)

Page editor: Rebecca Sobol

Development

The Freedom of Fork

One of the importa