Kernel development [LWN.net]

Brief items

Kernel release status

The current 2.6 development kernel is 2.6.27-rc9, released on October 6. Says Linus: "I know, I know, I said that -rc8 was supposed to be the last -rc, and that I'd release 2.6.27 this weekend. I lied. Sue me. I merged two subtle regression fixes today, and while both looked perfectly fine and had been tested by the people involved in the regressions, I just couldn't bring myself to then just slap a 'v2.6.27' on it without some more testing." Expect the final 2.6.27 release in the near future.

It is worth noting that, as of this writing, 2.6.27 does not contain a fix for the e1000e hardware corruption bug. What it does contain, though, is a series of patches which will prevent that bug from actually damaging the hardware. That makes the kernel safer to run, which is an important step in the right direction.

No stable kernel releases have been made over the last week. As of this writing, though, there were large updates for the 2.6.25 and 2.6.26 kernels in the review process.

Comments (2 posted)

Kernel development news

Quotes of the week

Of greatest interest to me were the descriptions given by Patrick McHardy for his new filtering framework, where all the complexity is in userspace and the kernel just runs filtering scripts and lookup datastructures fed to it by the user tools. In short, I think this stuff is great, and unlike some folks I don't think this will decrease netfilter participation by other developers at all.

And frankly, iptables was absolutely too accessible to contributors. Look at how much stinking poo is in the patch-o-matic, oft called "crap-o-matic".

-- David Miller

But then voting season comes and reminds you that all those Americans that are individually sane and normal tend to be collectively crazy and very odd. And that's when you really notice that you're not in Finland any more.

-- Linus Torvalds starts a blog

Comments (none posted)

Some development statistics for 2.6.27

By Jonathan Corbet
October 7, 2008

It's that time of the development cycle again: the 2.6.27 kernel, if not yet released by the time you read this, will be shortly. Various other LWN articles have looked at features found in this release; here we will look at where that code came from.

As of 2.6.27-rc9, a total of 10,604 non-merge changesets had been added to the mainline for the 2.6.27 kernel; those patches added a total of 826,000 lines of code while removing 608,000, for a net growth of 217,000 lines. There were 1,109 developers who contributed to 2.6.27, representing over 150 employers. 376 of those developers contributed a single patch during this development cycle.

The most active developers for 2.6.27 were:

Most active 2.6.27 developers

By changesets

Ingo Molnar 238 2.2%

Bartlomiej Zolnierkiewicz 235 2.2%

Adrian Bunk 221 2.1%

David S. Miller 206 1.9%

Alan Cox 196 1.8%

Yinghai Lu 192 1.8%

Jeremy Fitzhardinge 162 1.5%

Tomas Winkler 128 1.2%

Ben Dooks 120 1.1%

Jean Delvare 113 1.1%

Steven Rostedt 108 1.0%

Harvey Harrison 105 1.0%

Pavel Emelyanov 103 1.0%

Thomas Gleixner 101 1.0%

Jean-Francois Moine 89 0.8%

Lennert Buytenhek 88 0.8%

Hans Verkuil 81 0.8%

Joerg Roedel 81 0.8%

Arnd Bergmann 76 0.7%

David Brownell 75 0.7%

By changed lines

Paul Mackerras 138374 12.1%

David Woodhouse 44759 3.9%

Jean-Francois Moine 41157 3.6%

Adrian Bunk 35160 3.1%

Artem Bityutskiy 34545 3.0%

Luis R. Rodriguez 31825 2.8%

Sam Ravnborg 27443 2.4%

Karsten Keil 24674 2.2%

Russell King 22861 2.0%

Eilon Greenstein 19470 1.7%

Alan Cox 16957 1.5%

Felipe Balbi 16287 1.4%

Kumar Gala 14490 1.3%

David Brownell 12551 1.1%

Ralf Baechle 11057 1.0%

Lennert Buytenhek 9735 0.9%

David S. Miller 8621 0.8%

Juergen Beisert 8516 0.7%

Steven Rostedt 8455 0.7%

Ben Dooks 8399 0.7%

On the changeset side, Ingo Molnar ended up on top by virtue of the creation of large numbers of mostly x86-related changes, including a big subarchitecture reorganization; Ingo's count also includes the addition of ftrace, though much of that code was written by others. Bartlomiej Zolnierkiewicz continues to rework the old IDE layer, and Adrian Bunk, as always, energetically cleans up code all over the tree. David Miller's total includes the multiqueue networking code and a lot of other changes; Alan Cox did a lot of TTY work and big kernel lock removal.

Your editor was disappointed to come in at #23, and, thus, off the bottom of the table. Time to send in some quick white space fixes. More seriously, though, it's worth noting that there are relatively few patches of the "trivial change" variety in the mix this time around.

If we look at changed lines, Paul Mackerras comes out on top as the result of a single patch removing the obsolete ppc architecture. David Woodhouse reworked the management of firmware throughout the driver tree. Jean-François Moine brought the GSPCA webcam drivers into the tree, then put vast amounts of effort into cleaning them up. Artem Bityutskiy added the UBIFS flash filesystem, and Luis Rodriguez merged the ath9k wireless driver.

If we look at the companies behind this work, we get the following results (note that, as always, these results are somewhat approximate):

Most active 2.6.27 employers

By changesets

(None) 1925 18.2%

Red Hat 1405 13.2%

(Unknown) 921 8.7%

IBM 791 7.5%

Intel 605 5.7%

Novell 586 5.5%

Movial 234 2.2%

SGI 197 1.9%

(Consultant) 193 1.8%

Sun 184 1.7%

XenSource 165 1.6%

Parallels 157 1.5%

Oracle 148 1.4%

Marvell 143 1.3%

Fujitsu 138 1.3%

AMD 129 1.2%

Renesas Technology 125 1.2%

linutronix 121 1.1%

Simtec 119 1.1%

(Academia) 108 1.0%

By lines changed

IBM 207215 18.1%

(None) 129998 11.4%

Red Hat 109970 9.6%

(Unknown) 108878 9.5%

Nokia 52022 4.5%

Novell 49944 4.4%

(Consultant) 46529 4.1%

Broadcom 43438 3.8%

Atheros 38212 3.3%

Movial 35439 3.1%

Intel 32887 2.9%

Freescale 25511 2.2%

SGI 23444 2.0%

Marvell 20967 1.8%

Renesas Technology 15723 1.4%

MIPS Technologies 15701 1.4%

Pengutronix 13334 1.2%

Atmel 10786 0.9%

Analog Devices 10725 0.9%

Sun 9176 0.8%

There are not too many surprises in this table - in particular, the list of companies at the top tends not to change very much. That said, a few things are worthy of note. One is that Sun Microsystems has made its first appearance on this list. People complain about this company, but Sun's engineers have been quietly fixing things all over the tree. Broadcom is another company with a mixed reputation in the Linux community, but Broadcom is happy to provide support for some of its network adapters. Nokia's strong showing in the lines-changed table results primarily from the contribution of the UBIFS filesystem.

The most welcome change, though, is the first appearance of Atheros on this list. Atheros is a company which has quickly moved from a position of complete non-cooperation to one of supporting all of its hardware in the mainline kernel. To say that this is an encouraging development would be an understatement.

All told, the 2.6.27 development cycle shows that the process continues at full pace in a seemingly healthy state. Developers from all over the industry are all working together to make the kernel better for all. The number of companies which see participation in the process as being in their interest is growing, as is the number of developers who contribute patches. The Linux kernel, it seems, is in good shape.

Comments (22 posted)

Btrfs to the mainline?

By Jake Edge
October 8, 2008

One of the kernel projects that seems to be attracting a fair amount of attention these days is the new, copy-on-write filesystem, Btrfs. While still rather immature—the disk format is slated to be finalized by the end of the year—Btrfs has reached a point where lead developer Chris Mason wants to start talking about when to merge it into the mainline. Some are advocating moving quickly, while others are a bit more skeptical that merging it will lead to faster development.

Merging Btrfs would have a number of advantages, but more eyes is what Mason is seeking:

But, the code is very actively developed, and I believe the best way to develop Btrfs from here is to get it into the mainline kernel (with a large warning label about the disk format) and attract more extensive review of both the disk format and underlying code.

The Btrfs developers are committed to making the FS work and to working well within the kernel community. I think everyone will be happier with the final result if I am able to attract eyeballs as early as possible.

Typically, kernel code is not merged until it is ready, but an argument can be made that filesystems, like device drivers, are sufficiently isolated from the rest of the kernel that an early inclusion will do little harm. Also, a kind of precedent was set by the early "merge" of ext4, though that was an evolution of the existing ext3 filesystem, while Btrfs is entirely new. Andrew Morton has been encouraging Mason to get Btrfs "into linux-next asap and merge it into 2.6.29." He describes his reasoning:

My thinking here is that btrfs probably has a future, and that an early merge will accelerate its development and will broaden its developer base. If it ends up failing for some reason, well, we can just delete it again.

For various reasons this approach often isn't appropriate as a general policy thing, but I do think that Linux has needed a new local filesystem for some time, and btrfs might be The One, and hence is worth a bit of special-case treatment.

Adrian Bunk is not convinced that an early merge will bring the benefits that Morton is touting. He points to an early ext4 development plan, noting that the timelines outlined in that message were, perhaps, overly optimistic. "When comparing with what happened in reality it kinda disproves your 'acceleration' point."

There is a difference, though, between ext4 and Btrfs, that Serge Hallyn points out:

OTOH, maybe it's just me, but I think there is more excitement around btrfs. Myself I'm dying for snapshot support, and can't wait to try btrfs on a separate data/scratch partition (where i don't mind losing data). btrfs and nilfs - yay. Ext4? <yawn> That can make all the difference.

The original timeline showed mid-2007 as a target for a stable ext4 filesystem, but the project overshot that by a year or so. A recent patch proposes renaming ext4dev to ext4 because it "is getting stable enough that it's time to drop the 'dev' prefix." Unexpected difficulties led to ext4 development taking longer, as Mason describes:

Ext4 has always had to deal with the ghost of ext3. Both from a compatibility point of view and everyone's expectations of stability. I believe that most of us underestimated how difficult it would be to move ext4 forward.

Many seem to think that Btrfs is different, but it still has a ways to go. Currently, it does not handle I/O errors very well, while running out of space on the disk can be fatal. But it is getting close to usable—at least for testing and benchmarking. Getting the code into the mainline would cause more folks to look at it, as well as test various filesystem changes against it. Mason gives an example of how that can work:

For example, see the streaming write patches I sent to fsdevel last week. I wouldn't test against ext4 as often if I had to hunt down external repos just to get something consistent with the current development kernels. ext4 in mainline makes it much easier for me to kick the tires.

Btrfs has an aggressive schedule that targets a 1.0 release this year. The focus of that release is to nail down the on-disk format so that changes after that point will be backward compatible. Given that 2.6.29 will likely be released in early to mid-2009, it seems quite possible that Btrfs will be "merge-worthy" by then, which means that it really is not premature to start considering it now.

Comments (6 posted)

Moving interrupts to threads

By Jake Edge
October 8, 2008

Processing interrupts from the hardware is a major source of latency in the kernel, because other interrupts are blocked while doing that processing. For this reason, the realtime tree has a feature, called threaded interrupt handlers, that seeks to reduce the time spent with interrupts disabled to a bare minimum—pushing the rest of the processing out into kernel threads. But it is not just realtime kernels that are interested in lower latencies, so threaded handlers are being proposed for addition to the mainline.

Reducing latency in the kernel is one of the benefits, but there are other advantages as well. The biggest is probably reducing complexity by simplifying or avoiding locking between the "hard" and "soft" parts of interrupt handling. Threaded handlers will also help the debuggability of the kernel and may eventually lead to the removal of tasklets from Linux. For these reasons, and a few others as well, Thomas Gleixner has posted a set of patches and a "request for comments" to add threaded interrupt handlers.

Traditionally, interrupt handling has been done with top half (i.e. the "hard" irq) that actually responds to the hardware interrupt and a bottom half (or "soft" irq) that is scheduled by the top half to do additional processing. The top half executes with interrupts disabled, so it is imperative that it do as little as possible to keep the system responsive. Threaded interrupt handlers reduce that work even further, so the top half would consist of a "quick check handler" that just ensures the interrupt is from the device; if so, it simply acknowledges the interrupt to the hardware and tells the kernel to wake the interrupt handler thread.

In the realtime tree, nearly all drivers were mass converted to use threads, but the patch Gleixner proposes makes it optional—driver maintainers can switch if they wish to. Automatically converting drivers is not necessarily popular with all maintainers, but it has an additional downside as Gleixner notes: "Converting an interrupt to threaded makes only sense when the handler code takes advantage of it by integrating tasklet/softirq functionality and simplifying the locking."

A driver that wishes to request a threaded interrupt handler will use:

    int request_threaded_irq(unsigned int irq, irq_handler_t handler,
	    		     irq_handler_t quick_check_handler,
			     unsigned long flags, const char *name, void *dev)

This is essentially the same as request_irq() with the addition of the quick_check_handler. As requested by Linus Torvalds at this year's Kernel Summit, a new function was introduced rather than changing countless drivers to use a new request_irq().

The quick_check_handler checks to see if the interrupt was from the device, returning IRQ_NONE if it isn't. It can also return IRQ_HANDLED if no further processing is required or IRQ_WAKE_THREAD to wake the handler thread. One other return code was added to simplify converting to a threaded handler. A quick_check_handler can be developed prior to the handler being converted; in that case, it returns IRQ_NEEDS_HANDLING (instead of IRQ_WAKE_THREAD) which will call the handler in the usual way.

request_threaded_irq() will create a thread for the interrupt and put a pointer to it in the struct irqaction. In addition, a pointer to the struct irqaction has been added to the task_struct so that handlers can check the action flags for newly arrived interrupts. That reference is also used to prevent thread crashes from causing an oops. One of the few complaints seen so far about the proposal was a concern about wasting four or eight bytes in each task_struct that was not an interrupt handler (i.e. the vast majority). That structure could be split into two types, one for the kernel and one for user space, but it is unclear whether that will be necessary.

Andi Kleen has a more general concern that threaded interrupt handlers will lead to bad code: "to be honest my opinion is that it will encourage badly written interrupt code longer term," but he seems to be in the minority. There were relatively few comments, but most seemed in favor—perhaps many are waiting to see the converted driver as Gleixner promises to deliver "real soon". If major obstacles don't materialize, one would guess the linux-next tree would be a logical next step, possibly followed by mainline merging for 2.6.29.

Comments (14 posted)

Patches and updates

Kernel trees