Kernel development
Brief items
Kernel release status
The current 2.6 development kernel remains 2.6.26-rc4, released on May 26. An -rc5 release would seem to be due this week. If past history is any guide, look for it right after the LWN weekly edition is published.
Kernel development news
Quotes of the week
An interview with the new embedded maintainers
Embedded Linux is getting a lot of attention these days. A new kernel.org mailing list, linux-embedded—archived here—has been set up, with discussions and patches already being posted. In addition, Paul Gortmaker and David Woodhouse have volunteered to be the "embedded maintainers" for the kernel to help coordinate the embedded Linux community. They graciously agreed to a joint email interview to shed some light on their new roles.
LWN: What is your background with Linux, especially with embedded Linux?
After a while, I ended up working for Red Hat's engineering services division, doing board ports, drivers and other work. That's when JFFS2 was written, as part of a customer contract.
I've been at Red Hat since 2000, in various rôles including spending most of the last couple of years on OLPC. Due to HR misconduct, I handed in my notice on Monday and will be going elsewhere. I spoke to my new boss before volunteering for the 'embedded maintainer' rôle, and he was happy with that—it's another Linux-friendly company where I'll be doing kernel development, and community interaction will continue to be part of my day job.
LWN: What is the role you see for the embedded Linux maintainers for the kernel?
To start with, one of the things we really need to do is work with the various people who are using Linux in "embedded" situations, and help them to work better with the community. That isn't just the vendors of consumer equipment—it's communities like OpenWRT, handhelds.org, OLPC too. In no other field is the development of the Linux kernel so balkanised, with people all over the place carrying their own patches or even full trees of code.
Another part of the job, which is actually something I've been doing for years anyway, is reviewing general changes in the kernel with a particular mind to how they affect embedded systems. That's not just bloatwatch, although obviously that's a part of it. It also covers things like watching the IBM zSeries folks provide execute-in-place support for block devices under z/VM, and saying "hey, how can we use the same memory management for XIP from flash?".
The other main part of it is implementing features in the core kernel which are motivated by "embedded" requirements. Like the tricks for compiling parts of the kernel with "-fwhole-program --combine" to let GCC optimise better and reduce code size, for example.
A certain amount of it, especially the new linux-embedded@vger.kernel.org list, I expect to be a kind of targeted kernelnewbies—but obviously with a more specific focus on embedded issues, and to a certain extent on professional developers rather than having such a high proportion of hobbyists. Although I certainly wouldn't want to discourage the hobbyists and students from getting involved with embedded. It's a good way to get people to send you cute toys, after all!
I was trying to avoid having a 'linux-embedded' git tree, but for small things like the patch Tim Bird just sent to the linux-embedded list to introduce CONFIG_CONSOLE_TRANSLATIONS, I suppose it makes sense—so I've created that at git://git.infradead.org/embedded-2.6.git.
LWN: What are the specific problems that are faced by embedded developers trying to use Linux? What can you do to make that situation better?
Most people who have had to endure the jump uprev vs. a continual tracking and carrying of changes will tell you the jump is not the way to go for a multitude of reasons, but it seems a lesson that everybody ends up having to learn on their own. So, I'm hoping we can get some of these people more aligned with the typical Linux developer workflow—i.e. work from the latest codebase, create logical changesets that can be submission candidates etc. I've been in a couple of meetings recently where we've had the opportunity to educate embedded developers on the advantages of doing this, and the feedback has been positive so far.
LWN: The size of the kernel is getting larger in general, is it getting too big for some embedded applications? What, if anything, should be done to remedy that situation?
Getting back to the real question, though, there are things we can do. The smaller, more efficient "slub" memory allocator is an example, as is the --combine thing I mentioned above. The trick is to find ways to improve matters without just littering the whole thing with ifdefs.
LWN: What do you see as the priorities for kernel work to better support embedded Linux?
We need to revamp the MTD API fairly urgently too. It was derived from the PCMCIA code we had at the time without much planning, and we really need to improve on it now.
There may be a certain amount of bias in the items I've picked out, I suppose.
LWN: Do you have any specific goals for timelines of getting various features merged?
LWN: What problems do you foresee in working with other kernel developers who may have less (or no) interest in the concerns of the embedded community? Are there specific features that may be difficult or impossible to get merged?
A lot of the problem has been people who show up and throw their code over the wall, then run away. Or worse, those who don't even throw it over the wall at all. People seem to have forgotten how long it took us to educate the enterprise vendors and get them to work nicely with us; we're a bit behind the curve on the embedded side but we're getting there. And organisations like CELF are doing good work on that front, too.
In such cases where a feature doesn't look to be a probable merge candidate, not all is lost. We have to capitalize on the remaining value adds that come with still working with it as if it was a merge candidate. Things like cherry-picking parts of it that are of global value and thus reducing the carrying cost. Or being able to voice an opinion at the appropriate time if the maintainer of the feature notices that a proposed change somewhere else in the kernel will impact the feature that they have been maintaining independently. So I think we still want to work towards getting the people handling these "harder" features of interest to the embedded community working more in parallel with the main kernel community.
LWN: The term "embedded Linux" covers a huge spectrum of devices and uses of Linux, everything from devices where the OS is completely invisible up through internet tablets and UMPC devices that are essentially desktops squeezed into a smaller package. Where on that spectrum do your interests lie? What do you think the challenges of trying to support all of those different uses will be?
These are clearly separate problem spaces, but the common thing they both share is that you've got a group using a specific piece of hardware with a specific use case in mind. This tends to bring out the "works for us, lets get it done and shipping" mentality, and the work tends to never make it out to where others can review it and look at merging bits that make sense. I'm hoping this is where we can make a difference.
We would like to thank David and Paul for taking time to answer these questions.
Profiling kernel code coverage
Measuring which lines of code get executed and how often can be a useful tool for debugging or testing. That capability has long been available for user space programs in the form of gcov. A recent patch seeks to allow kernel hackers access to the same tool.
There are three main components to making gcov work with the kernel: changing the build to add the -fprofile-arcs -ftest-coverage gcc flags, hooking up the gcc-generated code to record the coverage information, and providing a way for the kernel to output the data to user space. The GCOV_PROFILE kconfig option governs whether to include gcov into the build, while GCOV_PROFILE_ALL activates profiling for the entire kernel. If desired, individual directories and files can be selectively included or excluded from being instrumented.
The new kernel/gcov directory contains the necessary functions to support the gcc-generated profiling code. This includes handling statically linked kernel code as well as kernel modules that are loaded. Information gathered from code in modules can be either preserved or discarded when they are unloaded. This will allow analysis of the module unloading path that could be useful for detecting resource leaks or other problems in that process.
A user space program compiled for gcov will write a binary file to the filesystem for each source file that contains the data corresponding to the execution path through that file. The kernel needs to do that differently, so instead it writes to a file in debugfs. Each source file that is compiled for gcov will store its information in /sys/kernel/debug/gcov/path/file.gcda, where /sys/kernel/debug is the debugfs mount point and path is the path to the file in the kernel tree. The individual .gcda files can also be written to, which will result in setting the accumulated data for that source file back to zero.
Once the data has been gathered, gcov can be invoked to produce a file that annotates the source showing each line with the number of times it has been executed. LCOV is a graphical tool that can also be used to examine the coverage information. LCOV and the gcov kernel patches both come from the Linux Test Project which has an extensive kernel test suite and is using gcov to expand the coverage of their tests.
As part of the patch set, the seq_file interface has been extended to allow writing of arbitrary binary data to a virtual file. Currently, the seq_file interface is somewhat character oriented, so a function has been added to fs/seq_file.c to provide that ability:
int seq_write(struct seq_file *seq, const void *data, size_t len)
As the prototype implies, it writes len bytes from
data to the seq_file seq.
Efforts to get gcov support into the kernel have been around since
2002,
but the code was recently rewritten to be a better fit for recent
kernels. In the patch, Peter Oberparleiter says "due to regular
requests, I rewrote the gcov-kernel patch from scratch so that it
would (hopefully) be fit for inclusion into the upstream kernel.
"
One of the bigger changes is to move the user space interface for gcov from
/proc into debugfs.
It seems that the technical issues have largely been addressed in the third version of the gcov patch. It can provide useful information, especially for increasing the reach of test coverage—something that can only help reduce kernel bugs—so it could make for a nice kernel addition. Whether it will be picked up into linux-next or -mm and pushed towards an eventual mainline merge remains to be seen.
Moving the firmware out
It seems that David Woodhouse had a bit of an ulterior motive when he recently reworked the kernel firmware loader. That is not to say the work is not useful in its own right, but one of his goals is more apparent now: removing all of the firmware from the kernel source tree. By making it easy to separate the firmware blobs—while still allowing them to be statically built into kernels—he has provided a possible path for all firmware needed by any Linux driver to live in a single place.
The firmware issue is somewhat contentious, with licensing and political issues that tend to annoy the kernel developers. Arguments about the "legality" of distributing firmware with the kernel flare up from time to time. Separate from that, there are some good reasons why it makes sense to keep the firmware in its own place: some distributions need or want to distribute their kernels without firmware blobs and some hardware manufacturers will not allow their firmware to be distributed with the kernel because of concerns about the GPL. The current situation makes it harder for both users and distributors.
Woodhouse brought up the idea of pulling the firmware out of the kernel in a post to linux-kernel and ksummit-2008-discuss. The agenda for this year's Kernel Summit is under discussion, so he proposed that it be discussed there. He is clearly trying to anticipate the technical concerns that others might have:
Unsurprisingly, there are some fairly strenuous objections. David Miller is quite annoyed:
If debian or whoever else have these concerns and want to rip the firmware out, it is one hundred percent their problem to patch things out of the kernel tree they use.
But there are other reasons to collect firmware in one single place, as Arjan van de Ven notes:
If you want to argue that that should be in the kernel tarball itself, you won't hear me complain. But others will... and for that a 2nd tarball might well be the answer. Just we shouldn't need 100 tarballs.
There is a very real concern, though, that putting firmware without source into the kernel is a GPL violation. It is impossible to know for sure without a court decision, which is something that no one wants to have to deal with. Companies—and their lawyers—tend to be very conservative when it comes to inviting lawsuits, so removing unrelated, possibly actionable code from the kernel sources is of great benefit to them. As Woodhouse says:
By making it easier to put all of the firmware in one non-GPL tree, hardware vendors—and their lawyers—may be willing to allow the firmware to be distributed. If Woodhouse's plan for supporting both compile-time and runtime loading of the firmware is successful and reasonably transparent, there should be little difference for kernel developers, but big improvements for users and distributors. It is unclear whether this is something that will be resolved in email, as Woodhouse hopes, or will require a discussion at the Kernel Summit in September, but it's an idea with a lot of merit that may find its way into the mainline at some point.
Patches and updates
Kernel trees
Architecture-specific
Build system
Core kernel code
Development tools
Device drivers
Documentation
Filesystems and block I/O
Memory management
Networking
Security-related
Virtualization and containers
Page editor: Jake Edge
Next page:
Distributions>>