LWN.net Weekly Edition for November 27, 2008

The Grumpy Editor's Asian Tour

This trip was your editor's second Japanese adventure, so it is interesting to look at what has changed over the intervening 16 months. The organization of the event remains about the same, down to the pizza-and-sushi party at the end of the first day. The agenda was more heavily oriented toward filesystems this time around, along with an overview of control group resource controllers by Hiroyuki Kamezawa. There was a big difference, though, in how the discussions went. Japanese audiences are notoriously quiet and unwilling to ask questions, but the attendees at the Japan Linux Symposium have gotten over this constraint. Questions and discussion abounded - and this is a good thing. Free software development does not work well if people are unwilling to ask questions or raise concerns. The fact that Japanese developers seem to be becoming more willing to participate in this way bodes well for their participation in the process as a whole.

How much are these developers participating now? Your editor did a quick and unscientific pass over the changes merged for the 2.6.28 kernel. It appears that a full 5% of those patches came from Japanese developers. If we exclude the work of one prolific developer who currently lives in Europe, it can be said that about 4% of 2.6.28 came from Japan itself. There has been a distinct increase in the amount of kernel code coming from that part of the world, and that can only be a good thing. The Linux Foundation's events in Japan (which began in the OSDL days and have been occurring regularly for a few years now) are, perhaps, producing the intended result.

Partly in recognition of the larger role now played by Japan in the free software community, the Japan Symposium will be taken to a higher level next year. The 2009 Kernel Summit will be held in Tokyo in October, followed by an expanded, three-day Symposium hosting talks by developers from all over the world. Planning for this event is just getting underway; expect the call for papers to come out early next year. It should be an interesting gathering in a fun city; your editor is already looking forward to attending.

The Korea Technical Jamboree was a lower-key gathering, held for a single afternoon on the 25th floor of a Seoul skyscraper. It lacked some of the infrastructure of the Japan Symposium (simultaneous translation, for example), but made up for it in enthusiasm. Your editor found a highly-engaged group of developers interested in talking about the technology. While much of the discussion was, surprisingly enough, in Korean, your editor was able to figure out that virtualization is high on the list of topics that this group was interested in.

There was also talk of business models and more. What there was less of, though, was talk of working with the community. From this brief encounter, your editor can guess that the Korean community is still working through the stage of figuring out what it can get from free software. Developers there seem to have, for the most part, not yet reached the point of sharing control of our free operating system and driving it in directions which better suit their needs. By their own admission, Korean developers are a little behind their Japanese counterparts in this regard, but that situation may not last for long.

One event your editor was not able to attend was FreedomHEC Taipei, held at the same time. Harald Welte was there, though, and posted a brief report:

Harald concludes that a higher Linux awareness in Taiwan should lead to better hardware support worldwide. With any luck at all, events like FreedomHEC, like those in neighboring regions, will help to create that awareness and expand our global development community.

Your editor was also unable to attend FOSS.in this year, despite a desire to return to that part of the world. FOSS.in is experimenting with a new event plan which is strongly oriented toward the production of tangible results; it has clearly been influenced by the success of the Linux Plumbers Conference. India has vast numbers of capable developers, relatively few of whom actively participate in our community now. That number has been growing, though, and events like FOSS.in have a lot to do with that change.

Finally: while your editor saw a lot of people expressing enthusiasm for Linux, many of them seemed to be doing it with Windows laptops. It seems that the value of Linux has not yet made itself felt in the desktop setting, even among those whose job it is to develop for or promote Linux. It would be interesting to know why more of this work can't move off of proprietary platforms.

Some of the answer may be related to episodes like this: your editor had rashly upgraded his laptop to a new stable distribution release (we'll call it Incredibly Irritating for the purposes of this discussion) just prior to traveling. The obligatory check to ensure that video projection still worked got forgotten this time; it had always worked before, what could go wrong this time? But it seems that this "upgrade" moved the tools needed to interface with RandR into a separate package, which it did not bother to install. So it was not possible to tell the laptop to send video out the external port.

Suffice to say that, five minutes prior to giving a talk, while disconnected from the network, one does not want to hear "you need to install this package before I'll turn on your external video port" from one's computer. Your editor will accept the blame for not having verified this functionality before traveling, but, still: things like this should Just Work, especially with a distribution which claims to have invested much energy into making such things Just Work. The presenters using Windows laptops were not having to contend with this kind of challenge.

That little glitch notwithstanding, this trip was a big success. The hospitality was amazing, interest was high, and there is always value in seeing how other groups are approaching free software. Our community continues to grow; many good things will come from that.

Comments (12 posted)

ELCE: Free software strategies for business

Shane Coughlan, legal coordinator for the Free Software Foundation Europe (FSFE), spoke about the advantages of free software from a business perspective at the recent Embedded Linux Conference Europe. His talk was not necessarily directed at his audience—as most were already free software users—but, instead, at the bosses of his audience, the management of companies using or considering using free software. His approach was to use the language that management understands while making a strong case for the value that free software can bring.

Coughlan noted the obligatory analyst projections, including 4% of European GDP coming from free software by 2010 as well as 80% of commercial software projected to contain free software by 2011. These are eye-opening numbers, so Coughlan went on to explain why those numbers are that high. Businesses are created to deliver value to their investors; in order to succeed, they will need to "deliver value now and deliver more value later and that's how you are going to run a successful business". A short-term outlook is not going to deliver real success. Paraphrasing Bill Clinton, he said "it's for the long term, stupid".

Proprietary software allows businesses to "do some stuff", but free software allows them to "do more stuff". As Coughlan describes it, the correct approach is for a business to "do more and keep doing it"; using free software makes that easier. "From a business perspective, free software rocks."

The key to free software is not in the cost nor is it in the availability of source code, he said, as those do not embody the freedoms that are important. The ability to "use, study, share, and improve", known as the four freedoms, are what gives free software its edge. They allow for more flexibility and growth than other kinds of software, he said.

If free software has so many upsides, what's the catch? "Free software is powered by licenses", so businesses need to understand those licenses and, just as importantly, the reasoning behind those licenses. This is no different than any other license, but a common problem is that people don't read the licenses or follow the terms. If they do, there is no problem, though. So, there is a catch, but "the catch isn't too big".

A business must apply some management science to determine its strategy: whether to use an existing solution or work on building a new one. If it decides to build something new, does it foster some kind of community model or not? These are the kinds of questions that need to be answered as part of determining a free software strategy.

Communication with people in the community is important as is choosing licenses that are popular and compatible. There are ways to reduce any risk associated with free software by using existing best practices. That means pro-actively resolving issues, not just putting free software into a product, then "pray, and be upset when someone tells us we were naughty".

One of the resources available to help management is the FSFE's Freedom Task Force (FTF) which is set up to assist everyone in understanding free software licensing. The FTF does training and consulting for businesses to help with licensing or other issues. If one is having trouble getting management on-board, refer them to FTF, "we won't actually lock them up and brainwash them", Coughlan said.

While companies are resistant to releasing their code, "if you're doing your marketing right and you're not relying on temporary monopolies, you can probably release quite a lot" of code without any business harm. It has been estimated that the body of free software is "worth" $12 billion, so a company can reimplement it, "at an estimated cost of $12 billion, or you can share your $2-3 million [investment] and use the code". It's a matter of recognizing the immense benefits that come with free software.

Coughlan also described a legal network that the FTF is fostering in Europe, where lawyers and legal experts can discuss issues of importance to free software, especially across jurisdictional boundaries. That network can help provide businesses with legal information to help reduce risks. There is, as yet, no US equivalent, though some US lawyers are participating with the European network. "Still, I'm confident that eventually the US will catch up with us", he said.

He wrapped up with some thoughts on the GPLv3, noting that "adoption in the first year has been very, very promising". In fact, it has been adopted faster than he expected. He did note that there are some problems with license incompatibilities, but that those are probably unavoidable. The ideal situation would be for every license to be able to work with every other, but it doesn't work that way, which is a bit of an inconvenience, but not really a problem at this point.

Coughlan did not really say very much that LWN readers won't have heard before, but he did put it together in a way that should resonate with businesspeople. It was also interesting to get a look at what FSFE, and particularly FTF, are up to. There is a lot of important free software work, completely separate from development, going on in Europe. Because I am US-based—hopefully not too US-biased—that sometimes gets overlooked, so it was very nice to have a chance to hear about that work.

Comments (1 posted)

An open letter to Evgeniy Polyakov

Dear Evgeniy,

Your editor has chosen to write to you in a public manner because he hates to see talented developers get frustrated with the kernel process and storm off. We do not have an excess of capable hackers, especially those who can work at your level. Losing one hurts. Your editor hopes that this eventuality can be avoided in this case - for you, and for others who may be encountering the same sort of frustrations you are. Getting code into the kernel can be a pain, sometimes. That said, some 1160 developers have managed it since the opening of the 2.6.28 merge window in October. It is possible to get code merged with sufficient care.

You first posted your distributed storage (DST) patch back in 2007; LWN took a look at it at that time. Since then, this code has come a long way. Beyond the basic task of exporting (and accessing) storage volumes across the net, this code claims "bullet-proof memory allocations," zero-copy transport, failover recovery with full transaction support, support for IPv6 and beyond, and a number of features including encrypted data channels. And, it is said, this code is fast. In general, it looks like good stuff.

You have posted the DST code on the mailing lists a number of times - too many, apparently, for your tastes. Frustration with the process appears to have led to the behavior described in your recent weblog post:

    ass licker
    static char dst_name[] = "Successful erackliss screwing into";

So, because the fourth posting of identical code in one month received little attention, DST now risks joining Kevents, network channels, network tree memory management, asynchronous crypto, and more in that place where dusty, out-of-tree stuff lives. This would not be a good outcome. So let us look at what can be done to avoid that - for your sake, for DST users' sake, and for the sake of other developers who may follow.

One way to get more reviews for your code is to pay attention to what those reviewers are saying. Andrew Morton spent some time on DST back in October. He had a number of concrete requests - such as documenting the user-space ABI and the network protocol - which have not been satisfied. He also asked for better code documentation in general:

The November 25, 2008 version of DST still does not tell that story, and that makes it very hard for other developers to understand. Code review, as you know, is in critically short supply in most free software projects. Getting reviews for difficult-to-understand code is hard, especially when it is a large body of complex code which occupies a niche in which relatively few developers have expertise. So it's not surprising that your most recent comment involved white space - anybody can make that kind of review without any need to actually understand what's going on.

Not only does your patch not tell a story, but the individual pieces of it do not even contain changelogs. For a patch set marked "consider for inclusion," that is a fatal error. Playing along with the system on things like that can seem like a waste of time, especially if you hold out no real hope of the patch being merged, but it is a necessary sign of respect for the people you are asking to consider the patch. No maintainer will accept a patch without a changelog.

While we're on the topic of documentation, your kernel configuration help text reads, in its entirety:

You owe your users a little bit more than that. Why might they want to use DST? Where can they get the associated tools? This, too, is a fatal error for any substantive kernel change.

And, while we're still somewhat on the subject of reviews: Andrew naturally called out the generic-looking thread pool implementation buried deep within DST; shouldn't it pulled out and made more generic? Your response can be paraphrased as "I can't be bothered to get the API past the review process, which, in any case, is biased toward those who are 'closer to the high end'." But pulling out this code and merging it separately might be the ideal starting point for getting the larger patch set into the kernel. A generic thread pool hiding within a storage device driver, instead, will be an ongoing impediment to inclusion.

Then there is the issue of motivation: why should the kernel developers want to merge this patch? Who are the users of it - do you have users now? How does it compare to other distributed storage technologies already in the kernel? What's the performance like - can you post some benchmark results? As it stands, DST looks like a nice piece of technology, but its benefits are still unclear. Tell that story, and the level of interest may well go up.

Finally, your editor would like to counsel patience. Some patches just take longer than others to find their way in the kernel. That is especially true of complex patches which touch on issues like memory management and which add new user-space ABIs. As a close-to-home example, look at David Howells's FS-Cache code, recently reposted for consideration. The first LWN article on this code was published more than four years ago. David is probably getting a little tired of maintaining this code out-of-tree, but he sticks with it, responds to reviews, and appears to be getting closer to inclusion.

Evgeniy, you appear to be a brilliant and productive hacker. You charge into places that scare off most kernel developers, and you always come back out with something interesting. We need developers like you. But we need developers like you who can work with the process - no matter how frustrating it gets. The kernel process is certainly far from perfect, but it is built around a set of principles which have served us well for many years. You could easily rise up through that process to become one of the "high end" developers who, you say, have an easier time getting code merged. Or you could take your marbles and storm home, making snide comments about reviewers on the way. But that would not be good for anybody involved.

(See also: Evgeniy's response to this article.)

Comments (21 posted)