The current 2.6 development kernel remains 2.6.26-rc4, released on May 26. An -rc5
release would seem to be due this week. If past history is any guide, look
for it right after the LWN
weekly edition is published.
Comments (3 posted)
Kernel development news
This is really irritating me, because this is one huge case of frickin
Animal Farm. First a little was taken away, then a little bit more,
and by the end you have something absolutely nobody would have agreed
to from the beginning.
-- David Miller
on removing firmware (see
I'd probably make a few grammatical changes too. When you're happy
with the content and your document is in the tree, I'll submit a
-- Nick Andrew
Comments (none posted)
Embedded Linux is getting a lot of attention these days. A new kernel.org
mailing list, linux-embedded—archived
here—has been set up, with discussions and patches already being
posted. In addition, Paul Gortmaker and David Woodhouse have volunteered
to be the "embedded maintainers" for the kernel to help coordinate the embedded
Linux community. They graciously agreed to a joint email interview to shed
some light on their new roles.
LWN: What is your background with Linux, especially with embedded
: I got involved in Linux while I was at University, and ended
at Nortel during one of the summer vacations, on a project for
networking over mains power lines. It involved Linux boxes as routers,
and I was working on solid state storage for that. From that, and from
the basic support we had for similar devices in the PCMCIA code base,
the MTD [Memory Technology Device] subsystem grew.
After a while, I ended up working for Red Hat's engineering services
division, doing board ports, drivers and other work. That's when JFFS2
was written, as part of a customer contract.
I've been at Red Hat since 2000, in various rôles including spending
most of the last couple of years on OLPC. Due to HR misconduct, I
handed in my notice on Monday and will be going elsewhere. I spoke to my
new boss before volunteering for the 'embedded maintainer' rôle, and he
was happy with that—it's another Linux-friendly company where I'll be
doing kernel development, and community interaction will continue to be
part of my day job.
Paul: I started using Linux back in the pre 1.0 days, and having
always been one to
take things apart and see how it works, being able to do that with the OS
appealed to me. I put together various documents to help people back when
entry level into Linux was quite high, started fixing and writing drivers,
on it went from there. In 2005, I joined Wind River, where I've been
focused on kernel and board specific kernel patches, and this has given me
opportunity to be exposed to all the different architectures and lots of
board variants within each architecture family.
LWN: What is the role you see for the embedded Linux maintainers for
: A bunch of things really. It's not like a normal maintainer
we take ownership of a certain section of code; it's a bit more fluid.
To start with, one of the things we really need to do is work with the
various people who are using Linux in "embedded" situations, and help
them to work better with the community. That isn't just the vendors of
consumer equipment—it's communities like OpenWRT, handhelds.org, OLPC
too. In no other field is the development of the Linux kernel so
balkanised, with people all over the place carrying their own patches or
even full trees of code.
Another part of the job, which is actually something I've been doing for
years anyway, is reviewing general changes in the kernel with a
particular mind to how they affect embedded systems. That's not just
bloatwatch, although obviously that's a part of it. It also covers
things like watching the IBM zSeries folks provide execute-in-place
support for block devices under z/VM, and saying "hey, how can we use the
same memory management for XIP from flash?".
The other main part of it is implementing features in the core kernel
which are motivated by "embedded" requirements. Like the tricks for
compiling parts of the kernel with "-fwhole-program --combine" to let
GCC optimise better and reduce code size, for example.
A certain amount of it, especially the new
email@example.com list, I expect to be a kind of targeted
kernelnewbies—but obviously with a more specific focus on embedded
issues, and to a certain extent on professional developers rather than
having such a high proportion of hobbyists. Although I certainly
wouldn't want to discourage the hobbyists and students from getting
involved with embedded. It's a good way to get people to send you cute
toys, after all!
I was trying to avoid having a 'linux-embedded' git tree, but for small
things like the patch Tim Bird just sent to the linux-embedded list to
introduce CONFIG_CONSOLE_TRANSLATIONS, I suppose it makes sense—so
I've created that at git://git.infradead.org/embedded-2.6.git.
Paul: There are several things that can be done here that will all
benefit Linux and
its users in the end. To start with, I'm hoping that we can close some of
entry level gap between people who don't necessarily track kernel
but yet have decided to develop on Linux with a specific embedded use case
mind, and those people who are long time Linux developers. We can also
the linkage between people writing feature changes and some of the users of
those features who are likely to be impacted, but otherwise would probably
unheard from. We can also look at externally maintained features of
to embedded users, and try and determine what is the blocking factor that
stopping it (or parts of it) from being merged upstream, and then assist in
removing those barriers where possible.
LWN: What are the specific problems that are faced by embedded
trying to use Linux? What can you do to make that situation better?
David: I think the biggest single problem has always been the same—it's that
people are too focused on getting their stuff out the door as quickly as
possible without much thought to working with upstream. Managers aren't
budgeting the time to get things merged, and engineers aren't talking
about their design early enough that it can be improved before it's a
That extra time isn't just about being a good citizen—failing to do
it almost always comes back to bite you personally, when you come to
do a new product, a product update, or even need to merge in changes
from upstream to fix bugs. But everybody seems to need to learn that
the hard way, it seems.
: A lot of times, you get the situation where a group who is
developing for an
embedded platform is focused 100% on getting their product up, running and
deployed. The developers involved aren't necessarily hard core Linux
and it usually plays out by them picking a kernel version, getting their
in their local tree, and that is it. They may not know git, they probably
don't have insight into who the respective subsystem maintainers are, they
perceive LKML as too hostile, or they may not have management buy-in on
to push stuff upstream. But inevitably, some time passes, and then they
carry forward task where they try and do a big jump uprev of all their
and this repeats forever.
Most people who have had to endure the jump uprev vs. a continual tracking
carrying of changes will tell you the jump is not the way to go for a
of reasons, but it seems a lesson that everybody ends up having to learn on
their own. So, I'm hoping we can get some of these people more aligned
the typical Linux developer workflow—i.e. work from the latest codebase,
create logical changesets that can be submission candidates etc. I've been
a couple of meetings recently where we've had the opportunity to educate
embedded developers on the advantages of doing this, and the feedback has
positive so far.
LWN: The size of the kernel is getting larger in general, is it
too big for some embedded applications? What, if anything, should
be done to remedy that situation?
: I know there are people who'll want to take me out back and
shoot me for
this, but I think a large part of the solution to that is knowing when
Linux is the answer, and accepting that sometimes it isn't. I've always
been a bit dubious about implementing XIP support in Linux, for example,
on the basis that if you care that
much, you should probably have been
using something like eCos anyway.
Getting back to the real question, though, there are things we can do.
The smaller, more efficient "slub" memory allocator is an example, as is the
--combine thing I mentioned above. The trick is to find ways to improve
matters without just littering the whole thing with ifdefs.
Paul: There will always be some hardware or some use case where
the right choice. It only makes sense to use the right tool for the
job. However we do want to make sure that Linux is that right tool in
as many cases as possible. On the plus side, the resources that are
found on a typical embedded target today are a lot more rich than they
were years ago. We just need to make sure that in optimizing for the
general x86 use case, we don't inadvertently hinder these more fringe
use cases coming from the embedded world.
LWN: What do you see as the priorities for kernel work to better
: One important priority right now is replacing JFFS2. I wrote
it, so I'm
allowed to say that—it was good for its time, with NOR flash devices
on the order of 32MiB. But having made it work on 1GiB of NAND flash in
OLPC, I certainly agree with the observation that it's being pushed past
its design limits. I'm very keen to get LogFS and/or UBIFS merged into
the kernel and stabilised to the point where we can really start moving
We need to revamp the MTD API fairly urgently too. It was derived from
the PCMCIA code we had at the time without much planning, and we really
need to improve on it now.
There may be a certain amount of bias in the items I've picked out, I
Paul: The embedded community as a whole is probably the biggest user
of all the
architectures outside of the x86 based platforms. Sometimes the
of certain things don't get much testing outside of the basic x86 family.
example, one of the features that there is considerable interest in is the
preempt_rt patch set. Yet once you stray outside of the x86 family, you
pretty much guaranteed to run into drivers specific to embedded targets
don't play nice once this patch set is in place. This isn't such a
simply because the intersection of the two hasn't been explored yet. I
there is value here in getting these types of intersections explored sooner
rather than later, by reducing some of the gap between the people working
these sorts of features, and those intending to use them on embedded platforms.
LWN: Do you have any specific goals for timelines of getting various
David: Other than "ASAP" for LogFS and UBIFS, not particularly.
Stuff is merged when it's ready.
Paul: At this point in time, no. I'm not really interested in
project or feature and trying to drive it towards some self-imposed merge
deadline. I'd rather work with them to try and find out what the problem
are, help with those where possible, be they logistical or technical and
them to a point where they feel that they can offer up merge candidates.
LWN: What problems do you foresee in working with other kernel
who may have less (or no) interest in the concerns of the embedded
community? Are there specific features that may be difficult or
impossible to get merged?
: I know it's fashionable to claim there's a big disconnect
embedded and big-iron users, but actually there's a lot more overlap
than many people seem to realise. I mentioned XIP earlier; can you also
guess who was first to implement tickless support?
A lot of the problem has been people who show up and throw their code
over the wall, then run away. Or worse, those who don't even throw it
over the wall at all. People seem to have forgotten how long it took us
to educate the enterprise vendors and get them to work nicely with us;
we're a bit behind the curve on the embedded side but we're getting
there. And organisations like CELF are doing good work on that front,
: We have to be realistic. There will always be some features
are too invasive to be sensible merge candidates, or the particular
feature has such a small user base, that it may not make sense from
a carrying cost point of view to target it for inclusion in the standard
kernel. Fortunately, I think the Linux developer community at large has
generally been flexible in accommodating most things, while at the same
excluding things where the best interest of the kernel as a whole needed
to come first.
In such cases where a feature doesn't look to be a probable merge
not all is lost. We have to capitalize on the remaining value adds that
with still working with it as if it was a merge candidate. Things like
cherry-picking parts of it that are of global value and thus reducing the
carrying cost. Or being able to voice an opinion at the appropriate time
the maintainer of the feature notices that a proposed change somewhere else
the kernel will impact the feature that they have been maintaining
independently. So I think we still want to work towards getting the people
handling these "harder" features of interest to the embedded community
more in parallel with the main kernel community.
LWN: The term "embedded Linux" covers a huge spectrum of devices and
of Linux, everything from devices where the OS is completely invisible
up through internet tablets and UMPC devices that are essentially
desktops squeezed into a smaller package. Where on that spectrum do
your interests lie? What do you think the challenges of trying to
support all of those different uses will be?
David: My interest is everywhere in that spectrum—and beyond. Too
on one small area is the way to ensure that you solve your own problems
while pessimising things for other people. I think it's important to
keep a certain amount of holistic focus, because that's how we can make
sure that Linux scales well both up and down.
: Absolutely. It seems that people naturally associate embedded
small and resource constrained end of the scale. But the reality is that
there are people who are wanting to use Linux in embedded applications
the baseline hardware has 16 cores and gigabytes of memory. On the one
end of the scale you are interested in things like efficiency of resource
usage, quick boot times, and on the other end of the scale, your interests
are more likely around features relating to specific high availability
features that may not be present in the standard kernel tree.
These are clearly separate problem spaces, but the common thing they both
share is that you've got a group using a specific piece of hardware with a
specific use case in mind. This tends to bring out the "works for us, lets
it done and shipping" mentality, and the work tends to never make it out to
where others can review it and look at merging bits that make sense. I'm
hoping this is where we can make a difference.
We would like to thank David and Paul for taking time to answer these
Comments (2 posted)
Measuring which lines of code get executed and how often can be a useful
tool for debugging or testing. That capability has long been
available for user space programs in the form of gcov. A recent
patch seeks to allow kernel
hackers access to the same tool.
There are three main components to making gcov work with the kernel: changing
the build to add the -fprofile-arcs -ftest-coverage gcc flags,
hooking up the gcc-generated code to record the coverage information, and
providing a way for the kernel to output the data to user space.
The GCOV_PROFILE kconfig option governs whether to include gcov
into the build, while GCOV_PROFILE_ALL activates profiling for the
entire kernel. If desired, individual directories and files can be
selectively included or excluded from being instrumented.
The new kernel/gcov directory contains the necessary functions to
support the gcc-generated profiling code. This includes handling
statically linked kernel code as well as kernel modules that are loaded.
Information gathered from code in modules can be either preserved or
discarded when they are unloaded. This will allow analysis of the module
unloading path that could be useful for detecting resource leaks or other
problems in that process.
A user space program compiled for gcov
will write a binary file to the filesystem for each source file that contains the
data corresponding to the execution path through that file. The kernel
needs to do that differently, so instead it writes to a file in debugfs.
Each source file that is compiled for gcov will store its information in
/sys/kernel/debug is the debugfs mount point and path is
the path to the file in the kernel tree. The individual .gcda
files can also be written to, which will result in setting the
accumulated data for that source file back to zero.
Once the data has been gathered,
gcov can be invoked to
produce a file that annotates the source showing each line with the number
of times it
has been executed. LCOV is a graphical
tool that can also be used to examine the coverage information. LCOV and
the gcov kernel patches both come from the Linux Test Project which has an
extensive kernel test suite and is using gcov to expand the coverage of
As part of the patch set, the seq_file interface has been
extended to allow writing of arbitrary binary data to a virtual file.
the seq_file interface is somewhat character oriented, so a function has
been added to fs/seq_file.c to provide that ability:
int seq_write(struct seq_file *seq, const void *data, size_t len)
As the prototype implies, it writes len
to the seq_file seq
Efforts to get gcov support into the kernel have been around since
but the code was recently rewritten to be a better fit for recent
kernels. In the patch, Peter Oberparleiter says "due to regular
requests, I rewrote the gcov-kernel patch from scratch so that it
would (hopefully) be fit for inclusion into the upstream kernel."
One of the bigger changes is to move the user space interface for gcov from
/proc into debugfs.
It seems that the technical issues have largely been addressed in the third
version of the gcov patch. It can provide useful information, especially for
increasing the reach of test coverage—something that can only help
reduce kernel bugs—so it could make for a nice kernel addition.
Whether it will be picked up into linux-next or
-mm and pushed towards an eventual mainline merge remains to be seen.
Comments (3 posted)
It seems that David Woodhouse had a bit of an ulterior motive when he recently
reworked the kernel firmware
loader. That is not to say the work is not useful in its own right,
but one of his goals is more apparent now: removing all of the firmware
kernel source tree. By making it easy to separate the firmware
blobs—while still allowing them to be statically built into
kernels—he has provided a possible path for all firmware needed by
any Linux driver to live in a single place.
The firmware issue is somewhat contentious, with licensing and political
issues that tend to annoy the kernel developers. Arguments about the
"legality" of distributing firmware with the kernel flare up from time to
time. Separate from that, there are some good reasons why it makes sense
to keep the firmware in its own place: some distributions need or want to
distribute their kernels without firmware blobs and some hardware
manufacturers will not allow their firmware to be distributed with the
kernel because of concerns about the GPL. The current situation makes it
harder for both users and distributors.
Woodhouse brought up the idea of pulling the firmware out of the kernel in
a post to linux-kernel and
ksummit-2008-discuss. The agenda for this year's Kernel Summit is
under discussion, so he proposed that it be discussed there. He is clearly
trying to anticipate the technical concerns that others might have:
By the time the kernel summit comes around, we should have made decent
progress on moving _all_ the firmware blobs to the firmware/ directory.
And at that point I'd like to remove them completely, to a separate git
tree and tarball. Those who really want to build them in to their static
kernel would still be able to, but it wouldn't be the default behaviour.
Unsurprisingly, there are some fairly strenuous objections. David Miller
is quite annoyed:
Sorry, that's taking things too far. I've fought, like, forever, to
keep the tg3 driver with it's firmware in-tree. I refuse to let the
driver get broken like that, it's staying working, and that means
in-tree and linked into the driver.
If debian or whoever else have these concerns and want to rip the
firmware out, it is one hundred percent their problem to patch things
out of the kernel tree they use.
But there are other reasons to collect firmware in one single place, as
Arjan van de Ven notes:
Right now it's a royal pain for users to get all the right pieces of
firmware.... having ONE place to put all that would go a long way of making
side of things easier.
If you want to argue that that should be in the kernel tarball itself, you
hear me complain. But others will... and for that a 2nd tarball might well
be the answer.
Just we shouldn't need 100 tarballs.
There is a very real concern, though, that putting firmware without source
into the kernel is a GPL violation. It is impossible to know for sure
without a court decision, which is something that no one wants to have to
deal with. Companies—and their lawyers—tend to be very
conservative when it comes to inviting lawsuits, so removing unrelated,
possibly actionable code from the kernel sources is of great benefit to
them. As Woodhouse says:
And it isn't just the nutters. Fedora also wants to ship the firmware in
a separate package from the kernel -- since the alleged GPL violation is
such a _gratuitous_ risk given that we always use an initrd anyway, and
because people want to be able to do 'Free' spins which don't feature
the firmware at all, even in the source packages.
By making it easier to put all of the firmware in one non-GPL tree,
hardware vendors—and their lawyers—may be willing to
allow the firmware to be distributed. If Woodhouse's plan for supporting
both compile-time and runtime loading of the firmware is successful and
reasonably transparent, there
should be little difference for kernel developers, but big improvements for
users and distributors. It is unclear whether this is something that will
be resolved in email, as Woodhouse hopes, or will require a discussion at
the Kernel Summit in September, but it's an idea with a lot of merit that
may find its way into the mainline at some point.
Comments (22 posted)
Patches and updates
Core kernel code
Filesystems and block I/O
Virtualization and containers
Page editor: Jake Edge
Next page: Distributions>>