In what is becoming a tradition at Linux Foundation conferences, a handful
of kernel developers
participated in a hour-long discussion of various topics at the first
Europe (LCE), which is being held in Prague on October 26-28. The format
typically doesn't change, with four kernel hackers and a moderator, but
the participants do regularly change. The LCE edition featured Linus
Torvalds, Paul McKenney, Alan Cox, Thomas Gleixner, and Lennart Poettering
as the moderator. A number of timely topics were discussed (security, ARM,
control groups), as well as a few timeless ones (user-space interfaces,
aging kernel hackers, future challenges).
While Torvalds really doesn't need to introduce himself, he did so by
saying that he writes very little code anymore and just acts as a
central point to gather up the work of the other kernel hackers. He is
"sadly, mainly just a manager these days", he said. McKenney
maintains the read-copy-update (RCU) code in the kernel, Cox has done many
things over the years but mostly works on System-on-Chip (SoC) support for
Intel these days,
while Gleixner works on "parts of the kernel nobody wants to
touch" and maintains the realtime patch set.
Poettering said that he wanted to start things off with something a bit
controversial, so he asked about user space compatibility. Clearly trying
to evoke a response, he asked if it was hypocritical that certain user
space interfaces have been broken in the past without being reverted. One
of his examples was
the recent version number change (from 2.6.x to 3.y) which broke a number
of user space tools yet wasn't reverted.
version number change is a perfect example, Torvalds said. Because it
(badly written) user-space programs, code was added to the kernel to
the version as 2.6.x, rather than 3.y (where x = 40 + y). It is "stupid of us to add extra code to lie about our
version", but it is important to do so because breaking the user
experience is the biggest mistake that a program can make, he said.
From the very beginning, Torvalds's mantra has been that "breaking the
user experience is a big no-no". The most important aspect of a
program is how useful it is to users and "no project is more
important than the user". Sometimes user space does get broken by
mistake because someone writes code they think is making an improvement but
breaks something. It is unavoidable and he feels bad about it when it
Not only does Torvalds feel bad, but he makes sure that the other kernel
developers feel bad about it
as well, Gleixner said. Torvalds readily agreed that he is quite vocal
about telling other kernel developers that they messed up when there is
breakage of user space. "There is a saying that 'on the internet, no one can hear you being
subtle'", so he doesn't even try, he said. Cox noted that the
current unhappiness with GNOME 3 is a demonstration of why you don't
suddenly change how things work for users.
There is pressure from hardware
and software makers to constantly upgrade, but Cox said that if you ask
users, they just want what worked before to continue working. Torvalds noted that
he used to have some old a.out binaries that he would periodically
test on new kernels to ensure they worked. One of them was an old shell
that used deprecated system calls, but he wanted to ensure that even though
he had moved on to newer shells, the old stuff would still work. It's
fine to add new things, he said, as long you "make sure that the old
ways of working still work".
The sysfs interface has been changed many times in ways that could have
broken user-space programs, according to Poettering, and he wondered why
that was allowed to happen. Cox said that it comes down to a question of
whether users care. If no one reports the problem, then no one really sees
it as one.
McKenney noted that his approach is to "work on things so
far from user space that I can't even see it", but that he was
concerned about adding tracepoints to the RCU code. Maintaining the exact
same tracepoints could make it difficult to change RCU down the road.
An audience member asked if it was worth it to continue maintaining
backward compatibility "forever", and wondered if it doesn't lead to more
kernel complexity. Torvalds was unconcerned about that and said that the
complexity problems in the kernel rarely arise because of backward
compatibility. Open source means that the code can be fixed, he said. Cox
echoed that, saying that if the kernel developers give user space better
interfaces, the programs will eventually switch, and any compatibility code
can be deprecated and eventually removed.
After noting that many of the Kernel Summit (KS) participants were in bed
by 9 or 10, Poettering asked whether the kernel community is
getting too old or, as he put it, whether it has become an old
man's club. Cox said that he saw a lot of fresh blood in the community,
more than enough to sustain it. Torvalds noted, though, that the average
age at the
summit has risen by one year every year. It's not quite that bad, he
said, but he also believes that the KS attendees are not an
accurate reflection of the community as a whole. It tends to be
maintainers that attend the summit, while many of the younger developers
have not yet become maintainers.
Part of the problem is one of perception, according to Torvalds. The Linux kernel crowd used to be notably young, because the older people
ignored what those crazy Linux folks were doing, he said. The kernel
hackers had a
reputation of being ridiculously young, but many of those same people are
still around, and are just older now. Cox noted that the kernel is now a stable
project and that it may be that some younger folks are gravitating to other
projects that are more exciting. Those projects will eventually suffer the
same fate, he said.
In response to an audience question, Torvalds wanted to clear something up:
"everyone seems to think I hate ARM", but he doesn't, he said.
He likes the architecture and instruction set of the ARM processor, but he
doesn't like to see the fragmentation that happens because there is no
standard platform for ARM like there is for x86. There are no standard interrupt
controllers or timers, which is a mistake, he said. Because of the lack of
a standard platform, it takes ten times the code to support ARM vs. x86.
ARM is clearly the most important architecture other than x86, he said, and
some would argue that the order should be reversed. The good news is that ARM Linux is
getting better, and the ARM community seems to be making progress, so he is
much happier with ARM today than he was six months ago. It's not perfect,
and he would like see more standardization, but things are much better.
Torvalds said that he doesn't necessarily think that the PC platform is
wonderful, but "supporting only a few ways to handle timers rather
than hundreds is wonderful".
Poettering asked if the panel would feel comfortable putting their private
key on a public Linux machine that had other users. In general, the clear
consensus was that it would be "stupid" (in Torvalds's words) to put
private keys on a system like
that. What you want, Torvalds said, is layers of security, and the kernel
does a reasonable job there. "But there will always be
problems", he said, so it is prudent to have multiple layers and
realize that some of the layers will fail. He noted that there are three
firewalls between his development
machine and the internet, none of which accept incoming ssh.
Cox agreed, saying that in security, depth is important. You might have
really good locks on the doors of your house, but you still keep your money
in the bank, he said. His primary worry is other local users on the
system, rather than remote attackers.
Torvalds and Cox disagreed a bit about the biggest challenge facing the kernel in
the coming years; Torvalds worries about complexity, while Cox is concerned
about code quality. The kernel has always been complicated, Torvalds said,
but it is not getting less complicated over time. There is a balancing act
between the right level of complexity to do the job right and
Cox believes that the complexity problem is self-correcting for the most
part. Once a subsystem becomes too complicated, it eventually gets replaced
with something simpler. But code quality is a problem our industry has
faced for sixty years. It is not just an open source problem, but the
whole industry struggles with it. When better solutions are found, it may well
require language changes which would have an enormous impact on Linux.
Poettering next asked about control groups: are they irretrievably broken
or can they be fixed? Cox thought that the concept behind cgroups was
good, but that some of the controller implementations are terrible. The
his mind is whether or not the controllers can be fixed and still continue
to manage the resource they are meant to. To Torvalds, controllers are a
necessary evil because they provide a needed way to manage resources like
memory, CPU, and networking across processes. But it is a
"fundamentally hard problem to dole out resources when everyone wants
everything", he said.
Cox agreed, noting that managing resources at a large scale will introduce
inefficiencies, and that's the overhead that some are complaining about.
Those who don't use the controllers hate the overhead, but those who need
them don't care about the impact, Torvalds said. Gleixner agreed that some
kind of resource management is needed.
According to Torvalds, part of the problem is that the more heavily used a
feature is, the more its impact is noticed. At one point, people said that
caches were bad because they impose some costs, but we got past that, he
said. Cox pointed to SMP as another area where the changes needed added
costs that some were not interested in paying. Torvalds noted that when
Cox started working on SMP for Linux, he thought it was an interesting
project but didn't have any personal interest as he couldn't even buy the
machines that Cox was using to develop the feature.
It is always nice to get a glimpse inside of the thinking of Torvalds and
some of his lieutenants, and these kernel developer panels give people just
that opportunity. It is one of the few forums where folks outside of the
kernel community can interact with Torvalds and the others in a more or less
[ I would like to thank the Linux Foundation for supporting my travel to
to post comments)