Your editor, having actually managed to spend a few weeks at home, once
again succumbed to the allure of long-distance travel. What is life, after
all, without jet lag, economy-class seats, and airline meals? The excuse
this time was the combination of the Linux Foundation's Japan Linux Symposium
and the Consumer Electronics Linux Forum's Korea
. Both events are intended to increase
communications with the Asian technical community and encourage
participation in the development process. They are also an opportunity for
developers from other parts of the world to learn more about what their
colleagues are thinking.
This trip was your editor's second Japanese adventure, so it is interesting
to look at what has changed over the intervening 16 months. The
organization of the event remains about the same, down to the
pizza-and-sushi party at the end of the first day. The agenda was more
heavily oriented toward filesystems this time around, along with an
overview of control group resource controllers by Hiroyuki Kamezawa. There
was a big difference, though, in how the discussions went. Japanese
audiences are notoriously quiet and unwilling to ask questions, but the
attendees at the Japan Linux Symposium have gotten over this constraint.
Questions and discussion abounded - and this is a good thing. Free
software development does not work well if people are unwilling to ask
questions or raise concerns. The fact that Japanese developers seem to be
becoming more willing to participate in this way bodes well for their
participation in the process as a whole.
How much are these developers participating now? Your editor did a quick
and unscientific pass over the changes merged for the 2.6.28 kernel. It
appears that a full 5% of those patches came from Japanese developers. If
we exclude the work of one prolific developer who currently lives in Europe, it
can be said that about 4% of 2.6.28 came from Japan itself. There has been a
distinct increase in the amount of kernel code coming from that part of the
world, and that can only be a good thing. The Linux Foundation's events in
Japan (which began in the OSDL days and have been occurring regularly for a
few years now) are, perhaps, producing the intended result.
Partly in recognition of the larger role now played by Japan in the free
software community, the Japan Symposium will be taken to a higher level
next year. The 2009 Kernel Summit will be held in Tokyo in October,
followed by an expanded, three-day Symposium hosting talks by developers
from all over the world. Planning for this event is just getting underway;
expect the call
for papers to come out early next year. It should be an interesting
gathering in a fun city; your editor is already looking
forward to attending.
The Korea Technical Jamboree was a lower-key gathering, held for a single
afternoon on the 25th floor of a Seoul skyscraper. It lacked some of the
infrastructure of the Japan Symposium (simultaneous translation, for
example), but made up for it in enthusiasm. Your editor found a
highly-engaged group of developers interested in talking about the
technology. While much of the discussion was, surprisingly enough, in
Korean, your editor was able to figure out that virtualization is high on
the list of topics that this group was interested in.
There was also talk of business models and more. What there was less of,
though, was talk of working with the community. From this brief encounter,
your editor can guess that the Korean community is still working through
the stage of figuring out what it can get from free software. Developers
there seem to have, for the most part, not yet reached the point of
sharing control of our free operating system and driving it
in directions which better suit their needs. By their own admission,
Korean developers are a little behind their Japanese counterparts in this
regard, but that situation may not last for long.
One event your editor was not able to attend was FreedomHEC Taipei, held at
the same time. Harald Welte was there, though, and posted a
I was really happy about FreedomHEC. It is really about time that
the Linux world and the Taiwan-based chipset vendors and system
integrators start much more interaction. It is a simple economic
fact that a lot of hardware development, both in the PC mainboard,
Laptop as well as the embedded device space happens in Taiwan. It
is also very true, that for whatever reason the gradual Linux
revolution in the server and desktop market in the EU, the US and
other markets such as Southern America has not really reached
Harald concludes that a higher Linux awareness in Taiwan should lead to
better hardware support worldwide. With any luck at all, events like
FreedomHEC, like those in neighboring regions, will help to create that
awareness and expand our global development community.
Your editor was also unable to attend FOSS.in
this year, despite a desire to return to that part of the world. FOSS.in
is experimenting with a new event plan which is strongly oriented toward
the production of tangible results; it has clearly been influenced by the
success of the Linux Plumbers Conference. India has vast numbers of
capable developers, relatively few of whom actively participate in our
That number has been growing, though, and events like FOSS.in have a lot to
do with that change.
Finally: while your editor saw a lot of people expressing enthusiasm
for Linux, many of them seemed to be doing it with Windows laptops. It
seems that the value of Linux has not yet made itself felt in the desktop
setting, even among those whose job it is to develop for or promote Linux.
It would be interesting to know why more of this work can't move off of
Some of the answer may be related to episodes like this: your editor had
rashly upgraded his laptop to a new stable distribution release (we'll call
it Incredibly Irritating for the purposes of this discussion) just prior to
obligatory check to ensure that video projection still worked got forgotten
this time; it had always worked before, what could go wrong this time? But
it seems that this "upgrade" moved the tools needed to interface with
RandR into a separate package, which it did not bother to install. So it
was not possible to tell the laptop to send video out the external port.
Suffice to say that, five minutes prior to giving a talk, while
disconnected from the network, one does not want to hear "you need to
install this package before I'll turn on your external video port" from
one's computer. Your editor will accept the blame for not having verified
this functionality before traveling, but, still: things like this should
Just Work, especially with a distribution which claims to have invested
much energy into making such things Just Work. The presenters using
Windows laptops were not having to contend with this kind of challenge.
That little glitch notwithstanding, this trip was a big success. The
hospitality was amazing, interest was high, and there is always value in
seeing how other groups are approaching free software. Our community
continues to grow; many good things will come from that.
Comments (12 posted)
Shane Coughlan, legal coordinator for the Free Software Foundation Europe
(FSFE), spoke about the advantages of free software from a business
perspective at the recent Embedded
Linux Conference Europe. His talk was not necessarily directed at his
audience—as most were already free software users—but, instead,
at the bosses of his audience, the
management of companies using or considering using free software. His
approach was to use the language that management understands while making a
strong case for the value that free software can bring.
Coughlan noted the obligatory analyst projections, including 4% of
European GDP coming from free software by 2010 as well as 80% of
commercial software projected to contain free software by 2011. These are
eye-opening numbers, so Coughlan went on to explain why those numbers are
Businesses are created to deliver value to their investors; in order to
succeed, they will need to "deliver value now and deliver more value
later and that's how you are going
to run a successful business". A short-term outlook is not going to
deliver real success. Paraphrasing Bill Clinton, he said "it's for
the long term, stupid".
Proprietary software allows businesses to "do some stuff", but
free software allows them to "do more stuff". As
Coughlan describes it, the correct approach is for a business to "do
more and keep doing it"; using free software makes that easier.
"From a business perspective, free software rocks."
The key to free software is not in the cost nor is it in the availability of
source code, he said, as those do not embody the freedoms that are
important. The ability to "use, study, share, and improve",
known as the four freedoms, are what gives free software its edge. They
allow for more
flexibility and growth than other kinds of software, he said.
If free software has so many upsides, what's the
catch? "Free software is powered by licenses", so businesses
need to understand those licenses and, just as importantly, the reasoning
behind those licenses. This is no different than any other license, but a
common problem is that people don't read the licenses or follow the terms.
If they do, there is no problem, though. So, there is a catch, but
"the catch isn't too big".
A business must apply some management science to determine its strategy:
whether to use an existing solution or work on building a new one. If it
decides to build something new, does it foster some kind of community model
or not? These are the kinds of questions that need to be answered as part
of determining a free software strategy.
Communication with people in the community is important as is choosing
licenses that are popular and compatible. There are ways to reduce any
risk associated with free software by using existing best practices. That
means pro-actively resolving issues, not just putting free software into a
product, then "pray, and be upset when someone tells us we were
One of the resources available to help management is the FSFE's Freedom Task Force (FTF) which is
set up to assist everyone in understanding free software licensing. The
FTF does training and consulting for businesses to help with
licensing or other issues. If one is having trouble getting management
on-board, refer them to FTF, "we won't actually lock them up and
brainwash them", Coughlan said.
While companies are resistant to releasing their code, "if you're
doing your marketing right and you're not relying on temporary monopolies,
you can probably release quite a lot" of code without any business
harm. It has been estimated that the body of free software is "worth" $12
billion, so a company can reimplement it, "at an estimated cost of
$12 billion, or you can share your $2-3 million [investment] and use the
code". It's a matter of recognizing the immense benefits that come
with free software.
Coughlan also described a legal network that the FTF is fostering in
Europe, where lawyers and legal experts can discuss issues of importance to
free software, especially across jurisdictional boundaries. That network
can help provide businesses with legal information to help reduce risks.
There is, as
yet, no US equivalent, though some US lawyers are participating with the
European network. "Still, I'm confident that eventually the US will
catch up with us", he said.
He wrapped up with some thoughts on the GPLv3, noting that "adoption
in the first year has been very, very promising". In fact, it has
been adopted faster than he expected. He did note that there are some
problems with license incompatibilities, but that those are probably
unavoidable. The ideal situation would be for every license to be able to
work with every other, but it doesn't work that way, which is a bit of an
inconvenience, but not really a problem at this point.
Coughlan did not really say very much that LWN readers won't have
heard before, but he did put it together in a way that should resonate with
businesspeople. It was also interesting to get a look at what FSFE, and
particularly FTF, are up to. There is a lot of important free software
work, completely separate from development, going on in Europe.
Because I am US-based—hopefully not too US-biased—that
sometimes gets overlooked, so it was very nice to have a chance to hear about
Comments (1 posted)
[Editor's note: the following article may look like a message to a
specific kernel developer, but it is really about the development process
in general. Over the years, your editor has seen too many worthy hackers
run into development process problems; the end result is often that we lose
that person's contributions. We are not so rich that we can afford that
sort of loss. The desire to prevent such problems was the motivation
behind your editor's recently-written development
process document - and this letter.
Your editor has chosen to write to you in a public manner because he hates
to see talented developers get frustrated with the kernel process and storm
off. We do not have an excess of capable hackers, especially those who can
work at your level. Losing one hurts. Your editor hopes that this
eventuality can be avoided in this case - for you, and for others who may
be encountering the same sort of frustrations you are. Getting code into
the kernel can be a pain, sometimes. That said, some 1160
developers have managed it since the opening of the 2.6.28 merge window in
October. It is possible to get code merged with sufficient care.
You first posted your distributed storage (DST) patch back in 2007; LWN took a look at it at that time.
Since then, this code has come a long way. Beyond the basic task of
exporting (and accessing) storage volumes across the net, this code claims
"bullet-proof memory allocations," zero-copy transport, failover recovery
with full transaction support, support for IPv6 and beyond, and a number of
features including encrypted data channels. And, it is said, this code is
fast. In general, it looks like good stuff.
You have posted the DST code on the mailing lists a number of times - too many,
apparently, for your tastes. Frustration with the process appears to have
led to the behavior described in your recent weblog post:
To understand the roots of this issue, I made a simple experiment with the
previous DST release. I added following lines into the patch to catch
static char dst_name = "Successful erackliss screwing into";
As you may expect, this does not compile and thus was never read by the
people who are subscribed to the appropriate mail lists. I got one private
mail about this fact for the whole week. The same DST code (without above
lines) was sent public first time more than month ago and was resent 3
times after that.
That's why I do not care about DST inclusion anymore. I do not care about
its linux-kernel@ feedback.
So, because the fourth posting of identical code in one month received
DST now risks joining Kevents, network channels, network tree memory
management, asynchronous crypto, and more in that place where dusty,
stuff lives. This would not be a good outcome. So let us look at what can
be done to avoid that - for your sake, for DST users' sake, and for the
sake of other developers who may follow.
One way to get more reviews for your code is to pay attention to what those
reviewers are saying. Andrew Morton spent some
time on DST back in October. He had a number of concrete requests -
such as documenting the user-space ABI and the network protocol - which
have not been satisfied. He also asked for better code documentation in
So please. Go through all the code and make it tell a story. Ask
yourself "how would I explain all this to a kernel developer who is
sitting next to me". It's important, and it's an important skill.
The November 25, 2008 version
of DST still does not tell that story, and that makes it very hard for
other developers to understand. Code review, as you know, is in critically
short supply in most free software projects. Getting reviews for
difficult-to-understand code is hard, especially when it is a large body of
complex code which occupies a niche in which relatively few developers have
expertise. So it's not surprising that your most recent comment involved
white space - anybody can make that kind of review without any need to
actually understand what's going on.
Not only does your patch not tell a story, but the individual pieces of it
do not even contain changelogs. For a patch set marked "consider for
inclusion," that is a fatal error. Playing along with the system on things
like that can seem like a waste of time, especially if you hold out no real
hope of the patch being merged, but it is a necessary sign of respect for
the people you are asking to consider the patch. No maintainer will accept
a patch without a changelog.
While we're on the topic of documentation, your kernel configuration help
text reads, in its entirety:
This driver allows to create a distributed storage block device.
You owe your users a little bit more than that. Why might they want to use
DST? Where can they get the associated tools? This, too, is a fatal error
for any substantive kernel change.
And, while we're still somewhat on the subject of reviews: Andrew naturally
called out the generic-looking thread pool implementation buried deep
within DST; shouldn't it pulled out and made more generic? Your response can be paraphrased as "I can't be
bothered to get the API past the review process, which, in any case, is
biased toward those who are 'closer to the high end'." But pulling out
this code and
merging it separately might be the ideal starting point for getting the
larger patch set into the kernel. A generic thread pool hiding within a
storage device driver, instead, will be an ongoing impediment to
Then there is the issue of motivation: why should the kernel developers
want to merge this patch? Who are the users of it - do you have users now?
How does it compare to other distributed storage technologies already in
the kernel? What's the performance like - can you post some benchmark
results? As it stands, DST looks like a nice piece of technology, but its
benefits are still unclear. Tell that story, and the level of interest may
well go up.
Finally, your editor would like to counsel patience. Some patches just
take longer than others to find their way in the kernel. That is
especially true of complex patches which touch on issues like memory
management and which add new user-space ABIs. As a close-to-home example,
look at David Howells's FS-Cache
code, recently reposted for
consideration. The first LWN
article on this code was published more than four years ago. David is
probably getting a little tired of maintaining this code out-of-tree, but
he sticks with it, responds to reviews, and appears to be getting closer to
Evgeniy, you appear to be a brilliant and productive hacker. You charge
into places that scare off most kernel developers, and you always come back
out with something interesting. We need developers like you. But
we need developers like you who can work with the process - no matter how
frustrating it gets. The kernel process is certainly far from perfect, but
it is built around a set of principles which have served us well for many
years. You could easily rise up through that process to become one of the
"high end" developers who, you say, have an easier time getting code
merged. Or you could take your marbles and storm home, making snide
comments about reviewers on the way. But that would not be good
for anybody involved.
(See also: Evgeniy's response
to this article.)
Comments (21 posted)
Page editor: Jake Edge
Next page: Security>>