Kernel development [LWN.net]

Brief items

Kernel release status

The current 2.6 development kernel remains 2.6.26-rc4, released on May 26. An -rc5 release would seem to be due this week. If past history is any guide, look for it right after the LWN weekly edition is published.

Comments (3 posted)

Kernel development news

Quotes of the week

This is really irritating me, because this is one huge case of frickin Animal Farm. First a little was taken away, then a little bit more, and by the end you have something absolutely nobody would have agreed to from the beginning.

-- David Miller on removing firmware (see below)

I'd probably make a few grammatical changes too. When you're happy with the content and your document is in the tree, I'll submit a patch :-)

-- Nick Andrew

Comments (none posted)

An interview with the new embedded maintainers

By Jake Edge
June 4, 2008

Embedded Linux is getting a lot of attention these days. A new kernel.org mailing list, linux-embedded—archived here—has been set up, with discussions and patches already being posted. In addition, Paul Gortmaker and David Woodhouse have volunteered to be the "embedded maintainers" for the kernel to help coordinate the embedded Linux community. They graciously agreed to a joint email interview to shed some light on their new roles.

LWN: What is your background with Linux, especially with embedded Linux?

David: I got involved in Linux while I was at University, and ended up working at Nortel during one of the summer vacations, on a project for networking over mains power lines. It involved Linux boxes as routers, and I was working on solid state storage for that. From that, and from the basic support we had for similar devices in the PCMCIA code base, the MTD [Memory Technology Device] subsystem grew.

After a while, I ended up working for Red Hat's engineering services division, doing board ports, drivers and other work. That's when JFFS2 was written, as part of a customer contract.

I've been at Red Hat since 2000, in various rôles including spending most of the last couple of years on OLPC. Due to HR misconduct, I handed in my notice on Monday and will be going elsewhere. I spoke to my new boss before volunteering for the 'embedded maintainer' rôle, and he was happy with that—it's another Linux-friendly company where I'll be doing kernel development, and community interaction will continue to be part of my day job.

Paul: I started using Linux back in the pre 1.0 days, and having always been one to take things apart and see how it works, being able to do that with the OS appealed to me. I put together various documents to help people back when the entry level into Linux was quite high, started fixing and writing drivers, and on it went from there. In 2005, I joined Wind River, where I've been primarily focused on kernel and board specific kernel patches, and this has given me the opportunity to be exposed to all the different architectures and lots of board variants within each architecture family.

LWN: What is the role you see for the embedded Linux maintainers for the kernel?

David: A bunch of things really. It's not like a normal maintainer rôle where we take ownership of a certain section of code; it's a bit more fluid.

To start with, one of the things we really need to do is work with the various people who are using Linux in "embedded" situations, and help them to work better with the community. That isn't just the vendors of consumer equipment—it's communities like OpenWRT, handhelds.org, OLPC too. In no other field is the development of the Linux kernel so balkanised, with people all over the place carrying their own patches or even full trees of code.

Another part of the job, which is actually something I've been doing for years anyway, is reviewing general changes in the kernel with a particular mind to how they affect embedded systems. That's not just bloatwatch, although obviously that's a part of it. It also covers things like watching the IBM zSeries folks provide execute-in-place support for block devices under z/VM, and saying "hey, how can we use the same memory management for XIP from flash?".

The other main part of it is implementing features in the core kernel which are motivated by "embedded" requirements. Like the tricks for compiling parts of the kernel with "-fwhole-program --combine" to let GCC optimise better and reduce code size, for example.

A certain amount of it, especially the new linux-embedded@vger.kernel.org list, I expect to be a kind of targeted kernelnewbies—but obviously with a more specific focus on embedded issues, and to a certain extent on professional developers rather than having such a high proportion of hobbyists. Although I certainly wouldn't want to discourage the hobbyists and students from getting involved with embedded. It's a good way to get people to send you cute toys, after all!

I was trying to avoid having a 'linux-embedded' git tree, but for small things like the patch Tim Bird just sent to the linux-embedded list to introduce CONFIG_CONSOLE_TRANSLATIONS, I suppose it makes sense—so I've created that at git://git.infradead.org/embedded-2.6.git.

Paul: There are several things that can be done here that will all benefit Linux and its users in the end. To start with, I'm hoping that we can close some of the entry level gap between people who don't necessarily track kernel development but yet have decided to develop on Linux with a specific embedded use case in mind, and those people who are long time Linux developers. We can also improve the linkage between people writing feature changes and some of the users of those features who are likely to be impacted, but otherwise would probably go unheard from. We can also look at externally maintained features of interest to embedded users, and try and determine what is the blocking factor that is stopping it (or parts of it) from being merged upstream, and then assist in removing those barriers where possible.

LWN: What are the specific problems that are faced by embedded developers trying to use Linux? What can you do to make that situation better?

David: I think the biggest single problem has always been the same—it's that people are too focused on getting their stuff out the door as quickly as possible without much thought to working with upstream. Managers aren't budgeting the time to get things merged, and engineers aren't talking about their design early enough that it can be improved before it's a fait accompli. That extra time isn't just about being a good citizen—failing to do it almost always comes back to bite you personally, when you come to do a new product, a product update, or even need to merge in changes from upstream to fix bugs. But everybody seems to need to learn that the hard way, it seems.

Paul: A lot of times, you get the situation where a group who is developing for an embedded platform is focused 100% on getting their product up, running and deployed. The developers involved aren't necessarily hard core Linux folks, and it usually plays out by them picking a kernel version, getting their stuff in their local tree, and that is it. They may not know git, they probably don't have insight into who the respective subsystem maintainers are, they may perceive LKML as too hostile, or they may not have management buy-in on trying to push stuff upstream. But inevitably, some time passes, and then they have a carry forward task where they try and do a big jump uprev of all their changes, and this repeats forever.

Most people who have had to endure the jump uprev vs. a continual tracking and carrying of changes will tell you the jump is not the way to go for a multitude of reasons, but it seems a lesson that everybody ends up having to learn on their own. So, I'm hoping we can get some of these people more aligned with the typical Linux developer workflow—i.e. work from the latest codebase, create logical changesets that can be submission candidates etc. I've been in a couple of meetings recently where we've had the opportunity to educate embedded developers on the advantages of doing this, and the feedback has been positive so far.

LWN: The size of the kernel is getting larger in general, is it getting too big for some embedded applications? What, if anything, should be done to remedy that situation?

David: I know there are people who'll want to take me out back and shoot me for this, but I think a large part of the solution to that is knowing when Linux is the answer, and accepting that sometimes it isn't. I've always been a bit dubious about implementing XIP support in Linux, for example, on the basis that if you care that much, you should probably have been using something like eCos anyway.

Getting back to the real question, though, there are things we can do. The smaller, more efficient "slub" memory allocator is an example, as is the --combine thing I mentioned above. The trick is to find ways to improve matters without just littering the whole thing with ifdefs.

Paul: There will always be some hardware or some use case where Linux isn't the right choice. It only makes sense to use the right tool for the job. However we do want to make sure that Linux is that right tool in as many cases as possible. On the plus side, the resources that are found on a typical embedded target today are a lot more rich than they were years ago. We just need to make sure that in optimizing for the general x86 use case, we don't inadvertently hinder these more fringe use cases coming from the embedded world.

LWN: What do you see as the priorities for kernel work to better support embedded Linux?

David: One important priority right now is replacing JFFS2. I wrote it, so I'm allowed to say that—it was good for its time, with NOR flash devices on the order of 32MiB. But having made it work on 1GiB of NAND flash in OLPC, I certainly agree with the observation that it's being pushed past its design limits. I'm very keen to get LogFS and/or UBIFS merged into the kernel and stabilised to the point where we can really start moving to them.

We need to revamp the MTD API fairly urgently too. It was derived from the PCMCIA code we had at the time without much planning, and we really need to improve on it now.

There may be a certain amount of bias in the items I've picked out, I suppose.

Paul: The embedded community as a whole is probably the biggest user of all the architectures outside of the x86 based platforms. Sometimes the functionality of certain things don't get much testing outside of the basic x86 family. For example, one of the features that there is considerable interest in is the full preempt_rt patch set. Yet once you stray outside of the x86 family, you are pretty much guaranteed to run into drivers specific to embedded targets that don't play nice once this patch set is in place. This isn't such a surprise, simply because the intersection of the two hasn't been explored yet. I think there is value here in getting these types of intersections explored sooner rather than later, by reducing some of the gap between the people working on these sorts of features, and those intending to use them on embedded platforms.

LWN: Do you have any specific goals for timelines of getting various features merged?

David: Other than "ASAP" for LogFS and UBIFS, not particularly. Stuff is merged when it's ready.

Paul: At this point in time, no. I'm not really interested in hijacking anyone's project or feature and trying to drive it towards some self-imposed merge deadline. I'd rather work with them to try and find out what the problem areas are, help with those where possible, be they logistical or technical and get them to a point where they feel that they can offer up merge candidates.

LWN: What problems do you foresee in working with other kernel developers who may have less (or no) interest in the concerns of the embedded community? Are there specific features that may be difficult or impossible to get merged?

David: I know it's fashionable to claim there's a big disconnect between embedded and big-iron users, but actually there's a lot more overlap than many people seem to realise. I mentioned XIP earlier; can you also guess who was first to implement tickless support?

A lot of the problem has been people who show up and throw their code over the wall, then run away. Or worse, those who don't even throw it over the wall at all. People seem to have forgotten how long it took us to educate the enterprise vendors and get them to work nicely with us; we're a bit behind the curve on the embedded side but we're getting there. And organisations like CELF are doing good work on that front, too.

Paul: We have to be realistic. There will always be some features that either are too invasive to be sensible merge candidates, or the particular feature has such a small user base, that it may not make sense from a carrying cost point of view to target it for inclusion in the standard kernel. Fortunately, I think the Linux developer community at large has generally been flexible in accommodating most things, while at the same time excluding things where the best interest of the kernel as a whole needed to come first.

In such cases where a feature doesn't look to be a probable merge candidate, not all is lost. We have to capitalize on the remaining value adds that come with still working with it as if it was a merge candidate. Things like cherry-picking parts of it that are of global value and thus reducing the carrying cost. Or being able to voice an opinion at the appropriate time if the maintainer of the feature notices that a proposed change somewhere else in the kernel will impact the feature that they have been maintaining independently. So I think we still want to work towards getting the people handling these "harder" features of interest to the embedded community working more in parallel with the main kernel community.

LWN: The term "embedded Linux" covers a huge spectrum of devices and uses of Linux, everything from devices where the OS is completely invisible up through internet tablets and UMPC devices that are essentially desktops squeezed into a smaller package. Where on that spectrum do your interests lie? What do you think the challenges of trying to support all of those different uses will be?

David: My interest is everywhere in that spectrum—and beyond. Too much focus on one small area is the way to ensure that you solve your own problems while pessimising things for other people. I think it's important to keep a certain amount of holistic focus, because that's how we can make sure that Linux scales well both up and down.

Paul: Absolutely. It seems that people naturally associate embedded with the small and resource constrained end of the scale. But the reality is that there are people who are wanting to use Linux in embedded applications where the baseline hardware has 16 cores and gigabytes of memory. On the one end of the scale you are interested in things like efficiency of resource usage, quick boot times, and on the other end of the scale, your interests are more likely around features relating to specific high availability features that may not be present in the standard kernel tree.

These are clearly separate problem spaces, but the common thing they both share is that you've got a group using a specific piece of hardware with a specific use case in mind. This tends to bring out the "works for us, lets get it done and shipping" mentality, and the work tends to never make it out to where others can review it and look at merging bits that make sense. I'm hoping this is where we can make a difference.

We would like to thank David and Paul for taking time to answer these questions.

Comments (2 posted)

Profiling kernel code coverage

By Jake Edge
June 4, 2008

Measuring which lines of code get executed and how often can be a useful tool for debugging or testing. That capability has long been available for user space programs in the form of gcov. A recent patch seeks to allow kernel hackers access to the same tool.

There are three main components to making gcov work with the kernel: changing the build to add the -fprofile-arcs -ftest-coverage gcc flags, hooking up the gcc-generated code to record the coverage information, and providing a way for the kernel to output the data to user space. The GCOV_PROFILE kconfig option governs whether to include gcov into the build, while GCOV_PROFILE_ALL activates profiling for the entire kernel. If desired, individual directories and files can be selectively included or excluded from being instrumented.

The new kernel/gcov directory contains the necessary functions to support the gcc-generated profiling code. This includes handling statically linked kernel code as well as kernel modules that are loaded. Information gathered from code in modules can be either preserved or discarded when they are unloaded. This will allow analysis of the module unloading path that could be useful for detecting resource leaks or other problems in that process.

A user space program compiled for gcov will write a binary file to the filesystem for each source file that contains the data corresponding to the execution path through that file. The kernel needs to do that differently, so instead it writes to a file in debugfs. Each source file that is compiled for gcov will store its information in /sys/kernel/debug/gcov/path/file.gcda, where /sys/kernel/debug is the debugfs mount point and path is the path to the file in the kernel tree. The individual .gcda files can also be written to, which will result in setting the accumulated data for that source file back to zero.

Once the data has been gathered, gcov can be invoked to produce a file that annotates the source showing each line with the number of times it has been executed. LCOV is a graphical tool that can also be used to examine the coverage information. LCOV and the gcov kernel patches both come from the Linux Test Project which has an extensive kernel test suite and is using gcov to expand the coverage of their tests.

As part of the patch set, the seq_file interface has been extended to allow writing of arbitrary binary data to a virtual file. Currently, the seq_file interface is somewhat character oriented, so a function has been added to fs/seq_file.c to provide that ability:

    int seq_write(struct seq_file *seq, const void *data, size_t len)

As the prototype implies, it writes len bytes from data to the seq_file seq.

Efforts to get gcov support into the kernel have been around since 2002, but the code was recently rewritten to be a better fit for recent kernels. In the patch, Peter Oberparleiter says "due to regular requests, I rewrote the gcov-kernel patch from scratch so that it would (hopefully) be fit for inclusion into the upstream kernel." One of the bigger changes is to move the user space interface for gcov from /proc into debugfs.

It seems that the technical issues have largely been addressed in the third version of the gcov patch. It can provide useful information, especially for increasing the reach of test coverage—something that can only help reduce kernel bugs—so it could make for a nice kernel addition. Whether it will be picked up into linux-next or -mm and pushed towards an eventual mainline merge remains to be seen.

Comments (3 posted)

Moving the firmware out

By Jake Edge
June 4, 2008

It seems that David Woodhouse had a bit of an ulterior motive when he recently reworked the kernel firmware loader. That is not to say the work is not useful in its own right, but one of his goals is more apparent now: removing all of the firmware from the kernel source tree. By making it easy to separate the firmware blobs—while still allowing them to be statically built into kernels—he has provided a possible path for all firmware needed by any Linux driver to live in a single place.

The firmware issue is somewhat contentious, with licensing and political issues that tend to annoy the kernel developers. Arguments about the "legality" of distributing firmware with the kernel flare up from time to time. Separate from that, there are some good reasons why it makes sense to keep the firmware in its own place: some distributions need or want to distribute their kernels without firmware blobs and some hardware manufacturers will not allow their firmware to be distributed with the kernel because of concerns about the GPL. The current situation makes it harder for both users and distributors.

Woodhouse brought up the idea of pulling the firmware out of the kernel in a post to linux-kernel and ksummit-2008-discuss. The agenda for this year's Kernel Summit is under discussion, so he proposed that it be discussed there. He is clearly trying to anticipate the technical concerns that others might have:

By the time the kernel summit comes around, we should have made decent progress on moving _all_ the firmware blobs to the firmware/ directory. And at that point I'd like to remove them completely, to a separate git tree and tarball. Those who really want to build them in to their static kernel would still be able to, but it wouldn't be the default behaviour.

Unsurprisingly, there are some fairly strenuous objections. David Miller is quite annoyed:

Sorry, that's taking things too far. I've fought, like, forever, to keep the tg3 driver with it's firmware in-tree. I refuse to let the driver get broken like that, it's staying working, and that means in-tree and linked into the driver.

If debian or whoever else have these concerns and want to rip the firmware out, it is one hundred percent their problem to patch things out of the kernel tree they use.

But there are other reasons to collect firmware in one single place, as Arjan van de Ven notes:

Right now it's a royal pain for users to get all the right pieces of firmware.... having ONE place to put all that would go a long way of making that side of things easier.

If you want to argue that that should be in the kernel tarball itself, you won't hear me complain. But others will... and for that a 2nd tarball might well be the answer. Just we shouldn't need 100 tarballs.

There is a very real concern, though, that putting firmware without source into the kernel is a GPL violation. It is impossible to know for sure without a court decision, which is something that no one wants to have to deal with. Companies—and their lawyers—tend to be very conservative when it comes to inviting lawsuits, so removing unrelated, possibly actionable code from the kernel sources is of great benefit to them. As Woodhouse says:

And it isn't just the nutters. Fedora also wants to ship the firmware in a separate package from the kernel -- since the alleged GPL violation is such a _gratuitous_ risk given that we always use an initrd anyway, and because people want to be able to do 'Free' spins which don't feature the firmware at all, even in the source packages.

By making it easier to put all of the firmware in one non-GPL tree, hardware vendors—and their lawyers—may be willing to allow the firmware to be distributed. If Woodhouse's plan for supporting both compile-time and runtime loading of the firmware is successful and reasonably transparent, there should be little difference for kernel developers, but big improvements for users and distributors. It is unclear whether this is something that will be resolved in email, as Woodhouse hopes, or will require a discussion at the Kernel Summit in September, but it's an idea with a lot of merit that may find its way into the mainline at some point.

Comments (22 posted)

Patches and updates

Kernel trees