Notes from the Collaboration Summit

One of the first events was the kernel developers' panel, moderated by your (normally rather immoderate) editor. Panelists James Bottomley, Matt Domsch, Dave Jones, Christoph Lameter, Ted Ts'o, Arjan van de Ven, and Chris Wright discussed a variety of topics ranging from kernel quality (getting better), code review, development process participation, hardware support, and more. Your editor was not able to take notes from the panel; perhaps the best report which has come up so far can be found in this InformationWeek article by Charles Babcock.

IDC analyst Al Gillen spent half an hour going through a bunch of chart-heavy slides on the future of Linux in the marketplace. Overall, things look good, in that a market worth $20 billion in 2007 is expected to go up to $50 billion in 2011. There were lots of associated details which have been reported elsewhere. One interesting aspect was watching how the analyst trade copes with "non-paid" Linux deployments - which, according to Mr. Gillen, is 43% of the total. There was talk about how "monetizing" these deployments is a challenge for those looking to make money in the Linux marketplace. He expressed surprise at just how many companies are confident in their ability to support Linux deployments on their own. But he also talked about just how important that non-paid base is for the support of the entire ecosystem. Non-paid deployments may be a "challenge" to those who would prefer to be paid, but their absence would be a rather larger challenge.

There was an echo of this insight when Red Hat CTO Brian Stevens talked. One of Red Hat's goals, he says, is to give customers the immense value that goes with a "zero cost to exit" offering. There is no RHEL lock-in. To that end, he says, the folks at CentOS have done Red Hat a great favor. Brian also talked about the difference between the old "selling the distribution" business model, which gave Red Hat an incentive to put lots of shiny new things into each release, and the current model, which puts the focus on continuity instead. Since Red Hat's customers have already paid for the next release, Red Hat doesn't need to add lots of cool new features to encourage them all to upgrade.

He then spent the rest of his talk on the various cool new features the company is working on, including messaging, realtime support, and more.

Marten Mickos, once CEO of MySQL and now a vice president at Sun Microsystems, gave a talk which was intended to make listeners feel good about Sun and its plans for free software. It bothers him, he says, when people ask whether MySQL will remain committed to Linux; it strikes him as a demonstration of uncertainty about the future of Linux in general. That uncertainty is unnecessary; Linux's future is strong, regardless of what MySQL does. But MySQL (and Sun) do remain committed to Linux as a platform; the era of monolithic computing platforms is over, and companies have to support customers who will make their own choices at each level in the stack. So LAMP as an "architecture of participation" will remain supported by Sun well into the future.

An industry panel on "the state of Linux" was a useful view into how some large companies see the platform. They are all seeing growth in Linux; Bdale Garbee (representing HP) noted that Linux is "showing up in everything" that customers are planning. IBM's Dan Frye said that Linux is ready for any kind of workload. Oracle's Wim Coekaerts did note, though, that Oracle's revenue from Linux, at a mere $2 billion, is "still lagging."

There was a fair amount of discussion on how to work with the development community; NetApp's Brian Pawlowski asserted that "money helps." By that, he means employing developers to work within the community and advance the platform. Bdale noted that HP tries to work "in" the community, not "with" it. Dan Frye echoed that thought, saying that it's important to have people with credibility in the community and to allow them to work inside the community for long periods of time. Motorola's Christy Wyatt, instead, worried that her company still doesn't have the necessary wisdom to work effectively with the development community; Linux and the mobile industry, she says, are still relatively new to each other.

Wim related a story from the first kernel summit wherein an Oracle representative presented a laundry list of desired features. That is, he says, not the right way to do things; the community tends not to react well to wishlists with no development effort behind them. Oracle now has a Linux development team which is entirely separate from the normal product teams; among other things, it has a blanket approval to contribute the code it develops, avoiding the lengthy and tiresome internal legal review process. The company has also adopted a policy of making projects open from the beginning, getting much-needed review early in the process.

Other participants noted that working with a company's legal department can often be the hardest part of community participation. Dan suggested bringing in the legal department at the beginning of a project and keeping them around; sticking with a single counsel who can slowly be educated in free software ways is also important. Bdale said that we were likely to need "legal domain experts" for some time yet, but that the situation is getting better; most lawyers now have at least some understanding of how free software licensing works. A couple of panelists discussed the legal headaches that come with mixing components with different licenses; they would certainly like to see fewer licenses going into the future.

The final session from the first day covered the state of mobile Linux. It was about the only contentious panel on a day where the majority of the sessions were mostly educational in nature. One area of disagreement was over security models. Some platforms (such as ACCESS) work with a fine-grained set of privileges, while Google's Android uses sandboxing and controlled access to resources determined by asking the user. The fine-grained approach is seen by some as an ideal way for carriers to lock down handsets and exert firm control over what handset owners can do - not the desired outcome. On the other hand, asking users is seen as insecure; it's not usually too hard to get users to agree to almost anything.

Perhaps the lowest moment in this panel came when Google's Eric Chu was asked about participation with the community as opposed to developing everything as a private fork. He replied that the Android code was open, it sits in a repository somewhere. But there will be no effort to engage with (for example) the kernel community and merge this code until it is "done." That approach runs against what others had been saying since the kernel panel that morning: one must get code out there as early as possible. When the Android developers finally decide that their code is ready, they are likely to have a nasty surprise when they try to merge it into the kernel and are told that much of it is unsuitable by design. Google came off looking somewhat bad here, but the truth of the matter is that most of the (many) mobile Linux projects are operating in similar ways. Getting these projects to really work with the communities whose code they are using is, as with many embedded applications, a challenge. One can hope that the suggestions given to these projects at the summit will be taken to heart.

That sort of communication is what makes this event worthwhile; it is often hard for this particular mixture of people to come together in other contexts. The Collaboration Summit was heavy on conversation in general, often to great effect. One well-known developer commented to your editor that the Summit had the biggest disparity between the official content and the "hallway track" that he had ever seen. The hallway track was good, with, hopefully, lots of good things to come from it in the coming months.

I've been bitten by the embedded problem many times. For starters it's
very hard to develop patches for new hardware or ASICs on your
platform and submit them to the mainline while you are in development.
Aside from the general problem of going "here is a driver for
something you won't see or be able to test for 6 months" companies are
usually very wary of tipping their hand before product release, even
if they aren't that worried about the code being open.

Of course we pay for it in the end. Embedded platforms typically end
up with a relatively old platform, heavily patched, which they have to
maintain for the product lifetime. I'm currently porting some apps
that run with quite old libraries on an embedded machine to a modern
desktop system, a process that has more than a few hurdles if you need
to stay running on your embedded system as well. The kernel for the
platform is still a heavily patched 2.4 with RT extensions. It would
be nice to run on a modern 2.6 kernel with it's in-built RT support,
but it would require a fair amount of up-front engineering effort to
reach something as functional as we already have, even if it would be
more maintainable going forward.

I wonder about the cooperation between Linux kernel development and mobile device developers.
How well do the OLPC and OpenMoko projects in this?  AFAIK it seems it was helpful that
pre-production OLPC machines were regularly made available to important developers, that OLPC
employed people known in the Linux area (at least Marcelo's name was familiar to me, and X11
people), and the close cooperation with Redhat / Fedora.  OpenMoko had the Neo1973 instead,
which probably got into the "right hands" too, and from the mailing list I somewhere saw they
do what they can to get their Linux patches merged upstream (... sometimes painfully).

Are there (already) lessons that e.g. Motorola and friends (traditional companies in contrast
to OLPC/OpenMoko) could learn from?  Are some of those lessons not applicable to them and why
not?

A couple of articles about the summit 
(such as this one http://news.yahoo.com/s/cmp/20080410/tc_cmp/207100528)
mention a patch from Nortel for restarting a crashed system.

Do anyone have any links to more info about this?

Thanks

the observation is correct - automatic reboot upon lockups is not possible in Linux yet.

The soft-lockup detector looks like the logical place to put this feature. I have whipped up a
patch for this and have queued it up for v2.6.26:

http://redhat.com/~mingo/softlockup-patches/softlockup-al...

it adds softlockup_panic=1 boot option, a /proc/sys/kernel/softlockup_panic sysctl, and a
CONFIG_SOFTLOCKUP_PANIC=y .config option - whichever is the most convenient to use.

this feature should be used in conjunction with panic_timeout=2, so that every panic reboots
the box after two seconds. Probably /proc/sys/kernel/softlockup_thresh should be decreased as
well to 10 seconds (down from the default 60 seconds) - so the combined reboot delay should be
12 seconds.

(Note: the feature will be default-available on all default kernel builds, but the sysctl will
default to off. I.e. the config option influences the value of the default, not the
availability of the feature.)

Just a note of warning based on my personal experience with Sun and interoperability ...

When AOL acquired Netscape and assigned many of its software engineers to work on iPlanet, a
joint venture between Sun and AOL, I was assigned to work on that disaster. During that time
Sun found some performance problems in one of its software server products and isolated it to
one of the components I worked on. Our tean looked into it and found that because of the way
the component had been designed in the first place, fixing it for Sun meant fixing it for all
platforms: about a dozen *nix variants including Linux and AIX as well as all flavors of
Windows (32bit). And, we did just that. Sun was pleased with the performance improvement but
had a sh?t hemorrhage when they discovered we had fixed it for every platform(!) They wanted
it fixed for Sun only. Nevermind that the component was MPL, GPL dual licensed and available
publicly for checkout via CVS.

To this day I'm very thankful for the management team at Netscape who wrestled Sun to the
ground over this and kept my team out of the politics of this. The whole ugly incident left a
very bad taste with me and forever poisoned my opinion of Sun's intentions with open-source.

The message is: Beware of Sun's stated support for open-source anything. They speak with
forked tongue.

JMHO. YMMV. IANAL. HAND. HTH.