Two kernel panel sessions were held last week in San Francisco, one for
each of two conferences sharing facilities—and participants. In
both cases, the kernel developers making up the panel were asked about
various kernel features and developments, both from a historical and future
perspective, but each had its own focus as well. The Embedded
Linux Conference (ELC) panel was, unsurprisingly, focused on topics of most
interest to the embedded community, while the Linux
Summit (LFCS) panel looked at more mainstream kernel concerns.
ELC: Embedded Linux Kernel Features and Developments
venue, the panel sessions also had another thing in common: LWN Executive
Corbet, who moderated the LFCS panel and was a member of the ELC version.
The ELC panel was moderated by CE Linux Forum (CELF) architecture group
Bird, while embedded maintainer David Woodhouse and Matt Mackall, developer
of the SLOB memory allocator (along with various other kernel tasks),
rounded out the panel. Bird asked most of the questions, but the audience also
got the opportunity to ask a few too.
One of the themes of the discussion—as well as Woodhouse's earlier
keynote—was the convergence of features between so-called "big iron"
(servers and mainframes) and embedded devices. Corbet was amused to see
"highmem" support recently added for ARM processors, noting that it was a
controversial feature at the time it was added for servers; supporting a
full 1GB of RAM on a 32-bit processor was once a "big iron" problem.
Mackall also pointed to SMP and
NUMA support moving into the embedded architectures. But things are not
only moving in that direction, Mackall said, there is recognition from the
big iron developers that there is value for their systems in some of the
embedded features too.
Bird asked the panel about the proliferation of embedded distributions and
whether that was a good or bad thing. Woodhouse said "fragmentation
doesn't have to be bad"; it's only bad when a distribution doesn't
work well with the various upstreams and goes off and does its own
"weird things". Multiple distributions are one of the
"great strengths of Linux", Corbet said, as it provides playgrounds
where folks can experiment with different approaches.
Mackall pointed to a lack of community involvement in the various embedded
Linux distributions, noting that the most successful desktop distributions
were those with a strong community. In the mobile space, the distributions
are "coming from the top down", he said, for any of them to be
successful, they need to get community feedback.
The impact and usefulness of new "social networking" sites for Linux
developers—like MontaVista's meld and the LF's relaunch of the Linux.com community—was another question Bird
put to the panel. Woodhouse didn't really see the need, but
"communication is always good". Mackall was concerned that
these other services not become a "substitute for talking to the
Linux kernel community through its normal channels". Corbet noted
that there is value in "small town environments", but there is
a risk that they can become inbred. "It rarely leads to good
things" when a small community gets headed off in their own
direction, he said.
One of the more interesting exchanges centered around the question of what
a developer who just has a small amount of time can do to assist the larger
community. The discussion spread out from there, though. Woodhouse stated
that every developer needed to make sure that what they are working on can
go upstream even if their managers "need to be whipped to allow you
to do that". But Mackall wanted to "back up a step"
and ensure that developers are running Linux on their desktop.
Mackall said that developers should be running Linux at home as well; if
they are going to work with Linux, they should "live it".
Making it work on a laptop is a good exercise; if it doesn't work, figure
out why and fix it. He has seen too many embedded Linux developers with
Windows desktops who don't understand Linux well enough to properly
develop on it. "They don't have the Linux mindset", he said.
Those thoughts were echoed by Woodhouse as he related an anecdote about
some embedded developers who would FTP a file to the Windows box, edit it
using Notepad, then FTP it back to the Linux machine. It is not
efficient to do things that way, he said. Doing the development on Linux
will lead to a better
result, Mackall said. Doing everything on a Linux desktop will help that,
Mackall pointed out, "you should read your mail on it too".
Towards the end of the hour-long session, Bird asked "have we
won?", is embedded Linux unstoppable or "is it possible to
lose?". Mackall and Corbet had similar thoughts, worrying about
the proliferation of devices running Linux that could not be modified by
their users. "We haven't won until I can put my code on my
phone", Mackall said. Corbet echoed that: "If we end up
populating the world with locked-down Linux systems, then we've lost".
In closing, Bird noted that embedded Linux has made an "awful lot of
progress". This is the fifth year for ELC and he has been working
on embedded Linux for 17 years, over that time, "things have gotten
way better", he said.
LFCS: The Linux Kernel: What's Next
Corbet opened the panel by having the participants introduce
themselves to an audience of around 400 people. The panel consisted of
X.org project lead Keith Packard of Intel, Andrew Morton the kernel "odd
job man" from Google, USB maintainer Greg Kroah-Hartman of Novell,
and Ted Ts'o of IBM who is currently on loan to the LF as its CTO. After
that, Corbet got started by asking Kroah-Hartman about the -staging tree.
Approximately one-third of the code that was merged as part of the 2.6.30
merge window came in via the -staging tree, which Kroah-Hartman maintains.
Corbet said there was a lot of confusion about the tree and asked for an
explanation. Basically, it is a collection of drivers that used to live
outside of the tree, Kroah-Hartman said, consisting of bad code with bad API
usage and other major problems barring their acceptance into the mainline,
in other words, "crap". But there is a lot of hardware in use
that requires those drivers and the code was not getting improved out of
the tree, so moving them gives a centralized location where people can get
them and hopefully improve them.
Kroah-Hartman said that there were several drivers that had graduated from
-staging and into the mainline, so the process seemed to be
working. "If you want to get involved in the kernel, that's a good
place to start", he said. He noted that there are TODO files in
each driver's directory listing the kinds of changes needed before the
driver will be accepted into the mainline.
Corbet mentioned that he had been going to conferences for years hearing
about all the great things that were going to be done in the Linux graphics
area, but that we had now reached a point where much of it had actually
been done. He asked Packard to fill the audience in on what had been done
and "why it's cool". Packard described how X.org had
"turned the graphics stack upside down" by moving the device
configuration out of user space and into the kernel.
By doing that, X becomes just an API for existing applications, and other
APIs such as OpenGL or Wayland can be considered, he said. Support for
Intel graphics is good, and there is lots of work going on for Radeon (ATI)
chipsets, but NVIDIA is "not helping at all". He pointed out
that Fedora 11 will be shipping with the Nouveau driver for NVIDIA hardware
because it has surpassed the free 'nv' driver in
capabilities. He also noted that moving the configuration and
initialization into the kernel allows people to experiment with graphics
acceleration without spending an inordinate amount of time figuring out how
to initialize the hardware.
Next, Corbet asked Ts'o about the status of the ext4 filesystem. Ts'o
reported that Fedora and Ubuntu would be shipping it in their next releases
that are coming within a few months. He said that the user community was
"to be brutally honest, that will sometimes find bugs". He
said one goal is to get it into the next round of enterprise
distributions. He also noted that ext4 is a temporary solution, based on
BSD FFS, which is technology from the 70s. Btrfs, nilfs, and others were
where the interesting filesystem development is happening. All of those
make it an "exciting time" to be a filesystem developer.
Morton responded to a query about the linux-next tree by saying that it is
working out well, overall, as a place for integration and testing. But, he
said that he
was "a bit disappointed with the uptake it has", especially
from a testing perspective. Fewer
maintainers are taking advantage of the opportunity to integrate and test
using linux-next than he would like to see. It is often
the case that when there is a problem that shows up in Linus Torvalds's
tree, it is because the code never made into linux-next.
From the audience, ftrace developer Steven Rostedt asked about the pressure
to merge new code upstream into the mainline, but that there is major resistance
to certain things—he mentioned SystemTap and utrace—being
merged. He wanted to know what can be done to resolve that. Morton
responded that for device drivers or supporting new architectures, the path
is easier, but that the two examples Rostedt gave touch core kernel code.
Morton likened the utrace battle to an "incestuous family
struggle", but noted that the code needed improvement before it
could go in.
One of the reasons that utrace didn't make it into the kernel was a lack of
an in-kernel user of the code, Rostedt noted. Morton responded that
having an in-kernel user for a feature is a "nice checkbox",
because it gives the kernel community a means to test the code. But,
Kroah-Hartman pointed out that "changing core kernel code is hard, and it
should be". Ts'o also pointed out that several core kernel
developers are helping out with utrace, which should significantly smooth its
path into the mainline.
That discussion led Corbet to ask about tracing, noting that there were
several tracing solutions that were still out of the tree, but that ftrace
got new tracers added for each kernel release. Morton would like to
"see evidence that people are using them and getting good
results". Both he and Ts'o pointed at the lack of documentation for
various tracers, saying that adding that and making the tracing more usable
would help get more of that code into the mainline kernel.
The recently proposed nftables packet
filtering subsystem was raised by Corbet as an example of a place where a
user-space interface—the existing iptables—might be
supplanted. He asked how that transition could be accomplished. Morton
called it a "pretty traumatic transition" that would require a
compatible set of tools, with several years of warning along with buy-in
distributions. That takes three to four years according to Kroah-Hartman.
Ts'o called the packet filtering interface more of an administrative
interface that didn't have to be kept as stable as others, but that the
iptables command does need to be stable.
All of that led Packard to complain about the difficulties of keeping the
current user-space interface for X servers while moving modesetting into
the kernel. According to Packard, there are exactly two users of the
interface, both of which are under his control, so why does he need to
provide backward compatibility? Ts'o said that the problem would be for
distribution users who wanted to upgrade their kernel. Because the
distribution might use an old X server, that interface—which Packard
described as "open /dev/mem"—needs to be
maintained. Kernel hackers want as much testing of new kernels as they can
get, so any barrier to that testing is problematic.
At the end of the session, LF Executive Director Jim Zemlin announced the
first ever LF "Unsung Hero" award, which he then presented to
Morton. He explained that Morton is an avid car racer, so the LF
arranged for him to have a day at the track as a reward. It was no
surprise that there
was much applause for Morton—one of the few people actually
able to follow the linux-kernel mailing list. He also reviews an incredible
amount of the code that ends up in the kernel.
These sessions provide an interesting view into the thinking of the members
of the panel—one not easily derived from just keeping up with the
technical side of Linux development via the LWN Kernel page or even by
sifting through linux-kernel. They also give attendees a look at what's
coming in the future that can be hard to discern, though Corbet's
Forecast is helpful there. In the end analysis, though, the biggest
benefit may just be putting kernel developers and users together in a
fairly informal setting so that both sides get a better feel for the
other. Faces and personalities don't necessarily jump out of the normal
communication channels, so panel sessions like those that went on in San
Francisco are useful well beyond their technical content.
to post comments)