|
|
Log in / Subscribe / Register

LPC: Fitting into the kernel ecosystem

By Jonathan Corbet
September 17, 2008
The first Linux Plumbers Conference started on September 17, 2008; the opening talk was a keynote by Greg Kroah-Hartman. He got the conference going with with a provocative sermon on how the development ecosystem works and the niche we all occupy within it. It was a fun talk - unless you happen to work for Canonical.

He started with an apology to Canonical, though. In earlier talks, he had said that only eight kernel patches had ever come from Canonical. In fact, he has been corrected; the proper number is 100.

So, Greg asked, why is he picking on Canonical? His answer came in the form of a table of contributors to the kernel. It looked like this:

DistributorChangesets
Red Hat11,846
Novell7,222
MontaVista1,074
Debian288
Gentoo229
Mandriva237
Wind River207
rPath186
Canonical100

Then Greg asked: does anybody from Canonical want to say anything? Nobody did.

[GregKH] Moving on to the Linux ecosystem. Greg put up a slide showing the larger components of this ecosystem - the low-level stuff that makes Linux what it is. Some of the largest components, beyond the kernel, were GCC, binutils, X.org, and the man pages distribution. Looking at lines of code, the kernel amounts to about 40% of the total. Other large components are all significantly smaller.

It turns out that Greg has been doing repository data mining in a number of projects beyond the kernel. So, for projects like GCC, X.org, and binutils, he was able to put up tables listing the top contributors. The results varied somewhat, but there were a number recurring themes. Red Hat tends to be toward the top of the list on all of these projects; companies like IBM and Novell also appear regularly. CodeSourcery is a significant contributor to GCC and binutils. The U.S. National Security Agency contributes 2.1% of the patches into X.org; why is not clear. In all of these projects there are significant contributions from unpaid developers, but those contributions are overshadowed by those from paid developers.

And Canonical is always at the bottom of the chart - if it is there at all.

At this point Greg moved to a whiteboard to present his view of how the community works. At the development level, you have developers contributing to projects, which then release the code. There may be a few users at that level who feed back information (and maybe patches), but, in general, the biggest consumers of the project's releases are the distributors.

Distributors package everything and provide it to their users. At this point, another feedback loop comes into play: users feed their experiences and problems back to the distributor. Those distributors will respond to the user feedback, improving their products. The amount of feedback from the distributors to the upstream projects varies, but it tends to be small. For enterprise distributions, it is quite small; they are running ancient versions of everything and have little to do with current upstream. The community-oriented distributions, such as Fedora or openSUSE, tend to feed more changes back to their upstream sources.

Then, there is the matter of redistributors who base their products on another distributor's work; these are distributors like Ubuntu or CentOS. There are no contributions back to the community from that kind of distributor at all. They are not functioning as a part of the Linux ecosystem.

Greg finished up with what appears to be the message he came to the Linux Plumbers Conference to deliver: if you are a developer, if you want to be a part of the ecosystem, and if you work for a non-contributing company: quit. There are plenty of companies that understand the ecosystem and which need good people; at least one company, it seems, had wanted to set up a recruiting table at the conference. It is a very good time for people with community participation skills; there is no reason for anybody who wants to work in the community to stay on the outside.

[As a postscript, it is amusing to note that, while the conference did not allow companies to set up recruiting tables, nobody has prevented prospective employers from filling a prominently-placed whiteboard with information about available positions.]

Index entries for this article
ConferenceLinux Plumbers Conference/2008

to post comments

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 0:58 UTC (Thu) by Burgundavia (guest, #25172) [Link] (39 responses)

Ugh, this is pretty poor reporting by LWN. It is overtly biased and mentions nothing that what Greg is saying is controversial. I hope there is a followup based on mdz's blog post: http://mdzlog.wordpress.com/2008/09/17/greg-kh-linux-ecos...

(Full disclosure: I have been an Ubuntu user since 2004 and sit on the Community Council)

Controversial

Posted Sep 18, 2008 1:06 UTC (Thu) by corbet (editor, #1) [Link] (18 responses)

Of course it's controversial. I was just reporting on what he said, though. What else do you think I should have done? I have expressed other opinions about Ubuntu elsewhere; I didn't feel the need to inject that into Greg's talk.

Controversial

Posted Sep 18, 2008 1:52 UTC (Thu) by jengelh (subscriber, #33263) [Link] (1 responses)

What timeframe does the statistic table span?

Controversial

Posted Sep 18, 2008 2:00 UTC (Thu) by willy (subscriber, #9762) [Link]

The analysis is for the past 3 years

Controversial

Posted Sep 18, 2008 2:03 UTC (Thu) by tseaver (guest, #1544) [Link] (3 responses)

Perhaps it would have been useful to note Greg's employer, as a tool to help readers evaluate his possible bias (I didn't know offhand that he was a Novell employee, for instance, untill reading mdz's followup).

First slide

Posted Sep 18, 2008 7:30 UTC (Thu) by alex (subscriber, #1355) [Link] (2 responses)

I'd of thought his email on the first slide gave it away. And anyway he's showing the numbers even when Novell doesn't come up as high as certain distros.

First slide

Posted Sep 21, 2008 0:27 UTC (Sun) by frazier (guest, #3060) [Link] (1 responses)

If you saw the slides you'd seen the suse.de email on the first slide, but if you were an LWN.net subscriber who read the article and didn't read the comments, you wouldn't have known the rest of the story. I like to think that the comments section is additional info, and not a place to be visited for a notable piece of the story that is missing.

First slide

Posted Sep 22, 2008 0:50 UTC (Mon) by dlang (guest, #313) [Link]

I agree, it's unusual for a LWN piece to be this one-sided.

Re: Controversial

Posted Sep 18, 2008 2:03 UTC (Thu) by mmcgrath (guest, #44906) [Link] (6 responses)

No worries corbet, you did right by the write up as far as I can see. Though it is natural that the Canonical folk would want to save face. One day they'll learn, patches speak louder then words.

Re: Controversial

Posted Sep 18, 2008 11:40 UTC (Thu) by mkflint (guest, #50223) [Link]

> patches speak louder than words

Yes they do! But so does artwork, integration, innovation, documentation, marketing, blah blah blah.

The primary goals, as I see them, are to create a great quality product and to increase usage of Free Software. And the kernel is just one part of that whole experience.

While each individual kernel developer has a preferred distro, the project as a whole should be distro-agnostic. I hope Greg KH made it clear that he was speaking as "Greg KH", and not as "Linux Kernel Representative".

Re: Controversial

Posted Sep 24, 2008 0:02 UTC (Wed) by daniel (guest, #3181) [Link] (2 responses)

I find GregKH's statement overtly offensive with no possible excuse. And just to be clear, I am not aligned with any Linux Vendor. However, such ill considered public behavior reflects poorly on Suse, not just GregKH.

Regards,

Daniel

Re: Controversial

Posted Sep 24, 2008 1:46 UTC (Wed) by mmcgrath (guest, #44906) [Link] (1 responses)

So you're saying he's right, he's wrong or that he just shouldn't be expressing his opinions / organizing metrics?

Re: Controversial

Posted Sep 24, 2008 3:46 UTC (Wed) by daniel (guest, #3181) [Link]

I am saying you are wrong to applaud that sort of antisocial behavior.

Regards,

Daniel

Re: Controversial

Posted Sep 25, 2008 14:05 UTC (Thu) by SEMW (guest, #52697) [Link] (1 responses)

> "patches speak louder then words"

True, but I do wonder how many patches Canonical GregKH feels would speak loud enough.

Canonical has ~130 employees; Novell has ~4100. So, considering the table for kernel contributions at the top, this works out at ~0.77 patches per employee for Canonical, and ~1.77 for Novell; a touch under 2.5 times as many.

But now consider that SuSE Linux has been around since 1994*, and Ubuntu, since 2004. That's around... Well, 2.5 times as long.

So it seems to me that Canonical doesn't actually do too badly out of the comparison.

* I am assuming that the table at the top doesn't distinguish between contributions from Novell SUSE and S.u.S.E.

(I admit that that's a slightly dodgy calculation, in that neither Novell nor S.u.S.E will have had anything like 4100 employees in 1994 -- but then, neither will Canonical have had 130 in 2004. I'm not aiming for a scientific comparison, only pointing out that presenting the raw numbers with no context of company size, as Greg did, is rather disingenuous).

Re: Controversial

Posted Feb 1, 2011 14:09 UTC (Tue) by jospoortvliet (guest, #33164) [Link]

Those numbers don't make much sense - most of Novell doesn't have anything to do with SUSE. The SUSE labs have about 500 employees, I believe - then there is some marketing and sales. That is surely 5-6 times what Canonical has, in total - but far from 130 vs 4100.

Disclaimer: I work for SUSE/Novell.

Controversial

Posted Sep 18, 2008 3:31 UTC (Thu) by Burgundavia (guest, #25172) [Link] (3 responses)

My comment was that other people have pointed out that Greg's analysis is flawed well before he made this speech. I suggest Ben Collin's (head of the Canonical server team) post: http://blog.phunnypharm.org/2008/07/canonical-and-linux-k...

And for the record, I don't work for Canonical.

Controversial

Posted Sep 18, 2008 6:51 UTC (Thu) by mdz@debian.org (guest, #14112) [Link] (2 responses)

Factual correction: Ben Collins is not the head of the Canonical server team.

Controversial

Posted Sep 19, 2008 6:15 UTC (Fri) by nealmcb (guest, #20740) [Link] (1 responses)

Correct. Rather, Ben Collins leads the Canonical Kernel Team.

Controversial

Posted Sep 19, 2008 16:52 UTC (Fri) by mdz@debian.org (guest, #14112) [Link]

That's not quite correct either. Pete Graner now leads the Canonical Kernel Team.

Controversial

Posted Sep 18, 2008 15:41 UTC (Thu) by frazier (guest, #3060) [Link]

> What else do you think I should have done?

I completely agree with another response to your comment, that it should have been obvious that the speaker works for a different distribution. In the future, please note who they're working for!

Also, I don't know if they were available at the time of article publication, but a link to presentation slides would be handy and appreciated:
http://www.kroah.com/log/linux/lpc_2008_keynote.html

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 3:36 UTC (Thu) by mcopple (guest, #2920) [Link] (12 responses)

His bias does not change the ultimate fact, which the referenced blog acknowledges:

"Canonical is primarily a consumer of the Linux kernel. It is one of the building blocks we need in order to fulfill our primary mission, which is to provide an operating system that end users want to use. It is, on the whole, a good piece of software which meets our needs well. We routinely backport patches from newer kernels, and fix bugs which are particularly relevant to us, but our kernel consists almost entirely of code we receive from upstream."

The Linux "ecosystem," whether that contains just a few pieces or every piece of software ever made for Linux, only works if everyone contributes. Eric Raymond spent quite a bit of time making this point in Cathedral and Bazaar. We know that in practice, only a small proportion of the user base actually contributes back to the community; therefore, it is even more critical that major distributors, of which Canonical is undeniably one, need to do their part to give as much back to the community as they take away from it.

Greg is saying, with a lot of justification, that Canonical is letting other distributors, especially RedHat and Novell, carry its water. The referenced blog post admits as much, but then goes on to try to cast it as a virtue, to opine that Ubuntu is about the whole operating system, not the kernel alone. Fair enough. But RedHat and Novell are in the business of putting together operating systems that "...end users want to use..." as well. What happens if, on Monday morning, they decide to follow the Canonical model, and become "...primarily a consumer of the Linux kernel...?"

Greg made it clear that, whatever your opinion on RedHat et al vs. Canonical, the majority of kernel hackers are doing this for a paycheck. If RedHat and Novell decide to pay their hackers to do something else, then the kernel process gets a lot slower and probably, a lot less functional.

It is time for Canonical to step up. I see claims all over the 'net that ubuntu is the most popular Linux desktop; it is time for Canonical to put some money behind that reputation. Step up, or shut up.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:06 UTC (Thu) by gregwilkins (guest, #515) [Link] (2 responses)

I think this criticism of canonical represents a pretty one-eye view of how what constitutes a contribution to open source. Patches are not the only way to contribute.

Canonical has put a huge effort into ubuntu and have made it one of the most easily consumable distributions of a vast amount of open source software. This exposure to users is the lifeblood of many projects.

It would be a fool who would try argue that the open source eco system would be better off without Canonical and ubuntu. If we are better off with them, then they must be contributing in somehow! So is the criticism that they are obliged to contribute more because they benefit from support contracts with users?

All users of open source software benefit from using it. We don't demand payment in cash or kind from our users, so why should we hold those that contribute in other ways to a higher standard?

To say that anybody is obliged to contribute back just devalues the gift that all contributions (code or otherwise) are!

To say that canonical is under some moral obligation to contribute developer time, is paramount to saying that we license our software freely so long as you don't get too big a benefit from it, in which case you are obliged to give in cash or kind! This is like the old ext.js license which was LGPL (unless you mades lots of money or were competative to ext.js)

That's not the deal! it's free! FREE F R E E !

Free as in freedom and that includes being freedom from moral obligations or guilt trips at the hands of those that think that only the kernel commits counts!

Canonical do contribute and even if they didn't it is wrong to say that they should.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:44 UTC (Thu) by davidw (guest, #947) [Link] (1 responses)

"Free riding happens with open source, news at 11". It would be great if Canonical poured even more of Mark Shuttleworth's money into various projects, but I think they're doing a pretty fine job in terms of making a product that is winning a lot of converts to Linux and open source. If they do less kernel work and more work on the distribution, isn't that simply taking advantage of the open source model? It's the concept of "comparative advantage": concentrate on what you do best.

Comparative advantage

Posted Sep 19, 2008 18:34 UTC (Fri) by giraffedata (guest, #1954) [Link]

Clarification of the economic principle of comparative advantage: it isn't "concentrate on what you do best"; it's "concentrate on what you do most better (than others do it)"

So even if Canonical can submit kernel patches better than it can distribute Linux, and even if Canonical can submit kernel patches better than anyone else, and even if Canonical can't distribute Linux as well as others, it may still be best for everyone if Canonical concentrates on distributing Linux.

What you compare is the difference between Canonical's and others' patch-submitting ability and the difference between Canonical's and others Linux-distributing abilities.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 12:53 UTC (Thu) by nix (subscriber, #2304) [Link] (8 responses)

Possibly Canonical prefers to spend its effort developing components which aren't in the rather small set Greg checked? Things the users can actually see? Things like, oh, GNOME? (Or KDE, I suppose, although Ubuntu isn't KDE-focussed.)

The Linux kernel and infrastructural stuff is very nice, but not everyone who uses it as a basis needs to enhance it, nor is there anything immoral or unethical about choosing to enhance non-infrastructural components instead.

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 14:38 UTC (Thu) by mmcgrath (guest, #44906) [Link] (7 responses)

"Possibly Canonical prefers to spend its effort developing components which aren't in the rather small set Greg checked? Things the users can actually see? Things like, oh, GNOME?"

Yes, surely thats the ticket! Tell yourself whatever lies you have to, the work Ubuntu does, is for Ubuntu. If you think not... Go get the source and prove me wrong. Greg mentioned he looked at more then just the kernel but left it open enough that someone could tell themselves "Oh, they must just not work on the kernel."

I'd challenge anyone with Ubuntu/Canonical to post their own numbers _from an upstream repo_ and show where Ubuntu/Canonical commit count is at vs other distributions.

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 16:43 UTC (Thu) by jschrod (subscriber, #1646) [Link]

I'm not a Ubuntu user and are not connected to Canonical in any ways. I have it tried once and deleted it a few weeks after. (It's not my style of distribution; I guess I'm too used to the old-fashioned Unix way, being a (literal) grey-beard Unix user since 1984 and a Linux user since 1994.)

But even I can see the value that Ubuntu (and Canonical, by paying the salaries of Ubuntu developers) provides to the Linux community: Packaging code into a coherent and *nice looking* whole, targeted towards desktop usage, that appeals to new users that have never used Linux before. And for that target audience their distribution is better than Debian or Red Hat, and arguably better than SUSE or Mandrake. In my book, this is a big win and a big contribution for the Linux community.

If they don't contribute code upstream to core infrastructure projects in addition, too bad. It mostly irks the developers in that projects; but other members of the rest of our community might see that providing packaging and user attraction (and also marketing) is a value in itself. At least, I do.

(To put that into context where I'm coming from: I use Debian, Red Hat, Slackware, and SUSE (in alphabetical order :), and almost all other Unix systems. As a developer, I belong to the TeX development community, and am active there since 1982.)

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 19:14 UTC (Thu) by lysse (guest, #3190) [Link] (5 responses)

> Tell yourself whatever lies you have to

I appreciate that your feelings are running strongly here, but after that little outburst I can't take you seriously.

Back on topic: Slackware contribution count, anyone? And unlike Canonical, Slackware sell copies of their distribution. Yet nobody's slagging off Pat Volkerding for freeloading (have they ever done so, even when Slackware was the most popular distro out there?).

This is "tall poppy syndrome" at its worst; if I'm alone in finding it kind of disgusting, then so be it, but I do.

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 19:44 UTC (Thu) by mmcgrath (guest, #44906) [Link] (4 responses)

"I appreciate that your feelings are running strongly here, but after that little outburst I can't take you seriously."

And why would you. I've asked for someone to provide numbers, facts. Why do that when they can just attack the person requesting them. ad hominem anyone?

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 22:37 UTC (Thu) by lysse (guest, #3190) [Link] (3 responses)

> I've asked for someone to provide numbers, facts.

Maybe so, but the bit I quoted and responded to was

> Tell yourself whatever lies you have to

which is aggressive and rude, and makes your response of

> Why do that when they can just attack the person requesting them. ad hominem anyone?

amusingly ironic (although a little sad).

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 23:18 UTC (Thu) by JoeBuck (subscriber, #2330) [Link] (2 responses)

OK. I will then politely ask: can anyone provide statistics on how much Ubuntu employees have contributed to Gnome, as compared to other contributors like Red Hat, Novell, and Sun?

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 23:48 UTC (Thu) by chromatic (guest, #26207) [Link]

I'm not sure that information would be useful or interesting either, unless there should be some metric of goodness based on an amount of contribution upstream.

In my mind, the question is *does* a project contribute upstream, not *how much* does a project contribute upstream.

RE: LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 11:19 UTC (Fri) by hppnq (guest, #14462) [Link]

Ubuntu employees

The difference between a company and a distribution is, well, key.

Maybe this comes close to answering your question, maybe it is better to go here for the datamining software, download the latest Gnome tarball and answer it yourself.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:18 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (3 responses)

Greg is above all a kernel contributor. Ubuntu may have won the hearths of many users but Canonical's insistence on directing the entities that do the work while contributing little didn't earn it a lot of goodwill from people like him. After Launchpad's "let us manage your projects for you" Canonical's hubris has reached such ridiculous levels it seriously proposed that other distributions synch their releases with its own (and Greg's numbers show clearly it had precious little to contribute to such a project).

Thus, of course there is a backlash. Canonical is behaving like it had lots of credits to spend, while doing precious little to earn them.

PS. On the exact count subject please remember that all the volunteers loosely affiliated to Fedora or OpenSuse don't contribute to Red Hat's or Novell's numbers either.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 17:09 UTC (Fri) by talisein (subscriber, #31829) [Link] (2 responses)

Canonical's hubris has reached such ridiculous levels it seriously proposed that other distributions synch their releases with its own...
I mention this only for the sake of the facts, but I'm afraid you have this backwards. The offer was to change Ubuntu's own release schedule to match up with others.

LPC: Fitting into the kernel ecosystem

Posted Sep 22, 2008 8:17 UTC (Mon) by niner (guest, #26151) [Link] (1 responses)

Well the real "offer" comes into sight when reading the full original:

"There’s one thing that could convince me to change the date of the next Ubuntu LTS:
the opportunity to collaborate with the other, large distributions on a coordinated major /
minor release cycle. If two out of three of Red Hat (RHEL), Novell (SLES) and Debian
are willing to agree in advance on a date to the nearest month, and thereby on a
combination of kernel, compiler toolchain, GNOME/KDE, X and OpenOffice versions,
and agree to a six-month and 2-3 year long term cycle, then I would happily realign
Ubuntu’s short and long-term cycles around that."

For me this sounds like: if the others do the coordination work of picking a date and the
corner stones of the distribution, Canonical may want to use the results. In other words:
offloading their core work. I wouldn't call that a nice offer.

LPC: Fitting into the kernel ecosystem

Posted Sep 26, 2008 5:36 UTC (Fri) by turpie (guest, #5219) [Link]

"For me this sounds like: if the others do the coordination work of picking a date and the
corner stones of the distribution, Canonical may want to use the results. In other words:
offloading their core work. I wouldn't call that a nice offer."
Be fair, that is a rather biased way of interpreting Shuttleworth's statement. What he said is that if the other major players agreed on a schedule, he would be willing to change Canonicals to match theirs. I'm sure that if the other distributions expressed an interest in his idea he would be willing to work with them on organising the synchronised schedule.

It is fair enough to criticise the amount of code Ubunutu contribute to the wider community, but I dont think it is fair to twist Mark's words that way.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:23 UTC (Thu) by tzafrir (subscriber, #11501) [Link] (2 responses)

Here's one point that I think was missed from the debate.

In this article, he writes:

> our [Ubuntu's] kernel consists almost entirely of
> code we receive from upstream.
>
> Why, then, does Greg feel that Canonical should be
> expected to make more changes to the Linux kernel?
>
> Is it because Ubuntu is a very popular system, with a
> lot of users? It is that, but most people who use Linux
> aren’t kernel developers, so a large user population
> doesn’t translate to a lot of Linux kernel patches.

(He goes on to list other potential reasons that he considers improbable)

I can't speak for Greg or anyone else. But I think that this reason is valid.

Ubuntu's users surely run into problems with, say, the kernel. They use some newer hardware, they do some unpredictable things. Being the "non-kernel-developers", they surely don't work out those problems directly with upstream. This is why you have a distribution. Some of them will report bugs. Ubuntu developers respond to those bugs and fix them. Some of them result in code fixes.

So if we see very few contributions from Ubuntu in the upstream kernel, what can it mean. I don't really know which of those is actually responsible for the problems.

(And note that Ubuntu != Cannonical)

1. Ubuntu's users run into relatively few bugs. Can't be. Where there are users, there are bugs :-)

2. Ubuntu's users don't report enough bugs. This would mean Ubuntu is very buggy. This is a problem for Ubuntu.

3. Users report issues, but the Ubuntu developers fail to solve most of them. If so, this is a problem of the level of support Ubuntu's users get from their developers.

4. Ubuntu's developers fix problems, but don't forward enough of the fixes upstream. If so, Ubuntu is not a good community member, and its developers create themselves maintenance issues.

We all agree on the impact of (1) and (2). We all do our best to increase the number of bugs (develop software) and help users report bugs. So let's focus on potential reasons (3) and (4).

If (3) is an issue, it impacts Ubuntu's name. Specifically, it impacts Cannonical's ability to provide professional support services on top of Ubuntu: if they do a bad job maintaining a distribution, why would I bother paying them for it?

(Note: that's an *if* there)

LPC: Fitting into the kernel ecosystem

Posted Sep 21, 2008 9:40 UTC (Sun) by Jonno (guest, #49613) [Link] (1 responses)

You are missing one possibility (which is the most common one in my limited
experience).

5) Ubuntu users report bugs, the ubuntu developers wait for upstream to
solve them, and then backport the fixes to the ubuntu packages.

In this senario, bugs are reported and fixed, but few patches goes from
ubuntu to upstream.

LPC: Fitting into the kernel ecosystem

Posted Sep 22, 2008 4:00 UTC (Mon) by vonbrand (guest, #4458) [Link]

That would just mean that they haven't got the manpower to give support to a "enterprise" distribution... You need engineers that know intimately what is going on upstream (i.e., are inmersed in its development) if you want to be able to fix problems in a reasonable timeframe (least of all, in the time your support contract promises).

Why the NSA contributes to X

Posted Sep 18, 2008 2:40 UTC (Thu) by nwnk (guest, #52271) [Link] (2 responses)

The U.S. National Security Agency contributes 2.1% of the patches into X.org; why is not clear.

SELinux. Well, Flask really, since the goal is to be portable to other OSes too.

X's security model is pretty much wide open by default. Once you get an authorized connection to the server, you can touch any X object you want. Flask labelling lets you fix this: you can prevent copy-and-paste from Top Secret apps to mere Classified apps, and so forth.

It also turns out that to do this you have to modernize huge swaths of X's internal object model. The subsystem private data infrastructure got a complete rewrite, and we have an extremely powerful hook system around basically every interesting point in the server's execution. Beyond that, Eamon's been an absolute machine in terms of code quality, smashing compiler warnings and bad APIs all over the place.

The hope, as I understand it, is to have a basic security policy available in the next six months or so that fixes some of the obvious flaws in the X security model, with development on a serious lockdown mode to follow. Don't quote me on timeframe or anything, I'm not doing any of that work, but it's really good stuff and they're definitely one of my favorite contributors to X right now.

Why the NSA contributes to X

Posted Sep 18, 2008 3:03 UTC (Thu) by bronson (subscriber, #4806) [Link] (1 responses)

That's fantastic news! I had no idea this sort of thing was going on behind the scenes. If the spooks want to improve X's object model, I couldn't be happier.

(And, of course, cue rumors about sending all X events to a small room in Langley...)

Why the NSA contributes to X

Posted Sep 18, 2008 8:50 UTC (Thu) by james (guest, #1325) [Link]

Come on, it's still X!

A large room at Langley...

</tongue-in-cheek>

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 3:01 UTC (Thu) by flewellyn (subscriber, #5047) [Link] (6 responses)

To be fair to Canonical, I imagine much of their work goes into userspace stuff, more than the kernel. And why not? A good kernel's only part of what makes a good OS, after all.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 3:36 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (5 responses)

Kernel is strategically, the most important part however especially for a vendor relying on services as the revenue stream. Red Hat is the leading contributor to the core components like Kernel, Xorg etc and that is much more compared to other niche upstream projects even among those included in Red Hat Enterprise Linux. However I don't think a user space comparison would look very different either considering that some of them are covered by Greg KH already. If someone wants to run similar stats from the desktop perspective against say the whole of GNOME, that would be pretty interesting. Maybe LWN can do that?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 6:34 UTC (Thu) by dlang (guest, #313) [Link] (4 responses)

Greg only looked at the biggest projects.

frankly, those projects are not where most of the work that Ubuntu has done shows up. they have been doing far more with tweaking configs and scripts to make a linux system do things that it could have done before, but nobody bothered to make it do.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 6:56 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link] (3 responses)

Looking at the biggest projects is only natural. If a distribution did "tweak" things to do stuff better and did not send it upstream, I would consider it as a net negative really and not a benefit to the community on the whole. If any considerable work has been done, it should show up *somewhere*. Lets see those stats.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:17 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

the upstram projects do not ship the slackware/gentoo/fedora/Suse config files (and nobody expects those distros to submit their default config files upstream), so why should you expect Ubunto to do so?

as for their other administrative and config scripts, where do you suggest that they submit them? there is no 'upstream' (other then possibly debian) for them to deal with. they do like every other distro does and provide the source of the scripts with the distro.

If one of the other posters is correct and that for their kernel they are almost completely upstream + backported fixes from upstream, then the main thing that they are doing is tuning the kernel and selecting config options, who do you suggest that they submit these configs to?

where they do have real development work going on I expect them to push things back upstream, if only in self defense (reducing their workload), but it takes people experienced in distro maintenance to really believe that. (just like it takes a experienced developer to really believe in code re-use, and some people never do learn and insist on always re-inventing the wheel)

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:48 UTC (Thu) by filipjoelsson (guest, #2622) [Link]

If you do work on configs to make packages harmonize better with each other (or whatever), it's perfectly legitimate for the communities to expect you to contribute these new improved default configs back to the originating projects. Pretty much the only thing that upstream wouldn't be interested in, is trademarked artwork.

Are the pesky Ubuntu developers twiddling their thumbs all day, or actually doing work? If they are doing work, it'd be a whole lot nicer of them to contribute. And since they are a Debian derivative, there is an upstream for exactly everything they do (including little scripts and exotic configs) - except inserting trademarked logotypes.

Then again, they may be very few, and already contributing exactly everything they do - but if that was the case; why the difference between Debian and Ubuntu?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:22 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

> the upstram projects do not ship the slackware/gentoo/fedora/Suse config
> files (and nobody expects those distros to submit their default config
> files upstream),

Of course they do. If you had followed any major software projects you'd have seen default config fixes from Red Hat, Mandriva, Novell, etc

> so why should you expect Ubunto to do so?

That's Ubuntu's opinion and why it's getting a black eye now.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 3:17 UTC (Thu) by pj (subscriber, #4506) [Link] (15 responses)

Clearly his reasoning is bogus: the downstream of a downstream contributes to its upstream, which is your downstream, so you can't tell its contributions from your downstream. In other words, Ubuntu is probably contributing back to Debian (though there may be gripes there too - but that's a separate issue), and the metrics he's gathered have limited visibility into such contributions.

Another interesting metric might be changesets per group-member. 'per capita' if you will. How may employees to Red Hat, Novell, Wind River and Canonical each have? How many developers do Debian, Gentoo, Mandriva have?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 5:01 UTC (Thu) by dowdle (subscriber, #659) [Link] (11 responses)

If Ubuntu/Canonical wants to modify the source code of a software package and get their patch upstream so they don't have to continuously carry it from version to version... they should be dealing with the direct upstream software and NOT Debian. Debian is not going to be a patch middleman between Ubuntu and an upstream software project.

If you are talking Debian produced software packages or packaging related changes that are distro specific... then it would be appropriate for Ubuntu to go through Debian.

So the claim that you can't see Ubuntu because they are go through Debian is just wrong. I don't think anyone from Ubuntu/Canonical nor Debian would claim that.

It is my understanding that LWN started doing this type of analysis on the Linux kernel some time ago, scripts that I believe GKH is using and has perhaps improved upon. I bring this up because LWN's findings have been exactly the same so far as I can tell. When Jon started publishing articles with his results I emailed asking why Ubuntu/Canonical and Debian didn't show up anywhere. If there was a flaw in the way he was gathering the data that was somehow excluding Ubuntu/Canonical and Debian... or if it was basically harder to identify them if some developers were using non-organizational email addresses. I believe Jon's answer was basically... no, I don't think I have a flaw and I don't think that their developers are being attributed to other catagories (like unknown or independent)... and that it just appears they do not contribute much to upstream kernel development. Now, I don't want to speak for Jon... so he can certainly clarify if he wants to. :)

Here is a clarification I want to make though... the blog posting protesting GKH's presentation says that his data is flawed and inaccurate with regards to Ubuntu/Canonical. It isn't that it is somehow flawed JUST toward them... but that it isn't perfect and is flawed in general. Measuring what he is trying to measure is quite difficult. Jon has pointed out near the beginning when he started reporting on the kernel... that there was no perfect way to measure... but that it was worth trying to measure even if flawed... and can certainly be improved over time. The methodology they use is the best they have been able to come up with.

I don't think adding meters like... how many patches per capita/employees... and is going to offer useful information... but if the consensus is to add such things, be my guest. It would probably be just as useful as adding patches as a percentage of userbase size.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 6:42 UTC (Thu) by dlang (guest, #313) [Link] (8 responses)

given that userbase is at best an educated guess, making charts based on employees or developers would be far better (you do have the problem of how to count debian employees or developers)

I also find it odd (as in, showing an agenda) that Canonical with 100 patches is considered the ultimate evil, but Debian with <300 isn't mentioned at all.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:33 UTC (Thu) by filipjoelsson (guest, #2622) [Link]

There are flaws in your reasoning. Patches / employees will not deliver more flattering numbers for Canonical, it will generate an error. Since neither Debian nor Gentoo has any employees at all (and that's why Debian isn't mentioned), their contribution in this respect is infinitely higher than that of Canonical. In the same way, counting spare time developers from Gentoo and Debian on par with salaried developers of Red Hat or Novell is irrelevant.

Perhaps comparing users / patches is a better statistic? Well, different distros attract different types of users. I'd guess that the communities of Debian and Gentoo are more technically inclined than the community of Ubuntu. Besides, Ubuntu has a pretty large user base - so that'd be neither better nor more flattering for Ubunto, I think.

So, what is there to compare? I suppose what's left is patches from companies generating revenue selling Linux. And if purely community driven distros show up on the charts, let them stay as a reminder to those who actually pay salaries to developers. Wind River, rPath and Mandriva may find themselves similarly challenged.

This does not imply that I agree with Greg. I think it was shortsighted to not include Gnome/KDE/XFCE in the comparison - not that I know if it would have made a difference.

It does imply that I think ubuntuers are whining, however. A more suitable reaction would have been: "Shit! Are we at the bottom? Well, we'll overtake rPath before the next time he gives a talk. And, Mandriva's next after that!"

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:36 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

Yes, counting users tends to be fuzzy but Fedora has published some statistics which provides a good estimate. If others do too (along with smolt profiles), we would get a better understanding of our user base and use that to convince vendors and ourselves to support Linux better.

https://fedoraproject.org/wiki/Statistics

It is possible to be more accurate but at the cost of privacy which is not a good balance but more details at

https://fedoraproject.org/wiki/Statistics/Ideas

I am sure there is a agenda involved in getting commercial vendors involved to participate more which I think is a good thing. If there are accusations of anything else. being explicit is (somewhat) courteous. I don't see anybody mentioning "evil". Debian contributing as many patches as it has as a volunteer based organization seems pretty remarkable to me neverthless.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:36 UTC (Thu) by alex (subscriber, #1355) [Link] (4 responses)

I also find it odd (as in, showing an agenda) that Canonical with 100 patches is considered the ultimate evil, but Debian with <300 isn't mentioned at all.

I don't. All it says is a purely volunteer based effort like Debian still submits more patches than the more widely used and commercially funded Ubuntu.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 16:43 UTC (Thu) by mdz@debian.org (guest, #14112) [Link] (2 responses)

...which is fantastic, and shows that Debian is a large, mature and relatively healthy community of developers. The most significant group of contributors to Linux, Greg tells us, is still individuals who don't acknowledge corporate sponsorship of their work.

Debian has very successfully enabled thousands of developers to contribute, and has grown into a very successful and independent project. I hope that Ubuntu is in such good shape when it's 15 years old.

For now, however, Ubuntu has a relatively small developer community which is highly dependent on Canonical. The Debian kernel team is larger than the Ubuntu kernel team and has been active for a much longer time.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 17:16 UTC (Thu) by alex (subscriber, #1355) [Link] (1 responses)

I agree it's good and I'm not really into giving Canonical/Ubuntu a shoeing over their perceived lack of contribution upstream. The numbers are what they are and if people don't want to feel picked on because of Canonicals poor showing compared to the other major distros then they know what they need to do.

IIRC Google got similarly picked on at last years OLS and I think their stats have improved this year. I'm not sure if it's cause and effect though...

LPC: Fitting into the kernel ecosystem

Posted Sep 23, 2008 19:56 UTC (Tue) by chema (subscriber, #32636) [Link]

"IIRC Google got similarly picked on at last years OLS and I think their stats have improved this year. I'm not sure if it's cause and effect though..."

I think Google stats enhancement is called Andrew Morton ;)

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 14:20 UTC (Fri) by hmh (subscriber, #3838) [Link]

There are some key differences between Debian and Ubuntu as far as the kernel and kernel contributions go.

Debian kernels are really, REALLY close to mainline kernels. They sometimes contain backports of patches that are not in mainline yet, but have been already accepted for eventual inclusion. And the rare Debian-originated patch is almost always sent upstream nowadays (I am not sure it was like that a few years ago, and sometimes the patches are not ACCEPTED upstream but still remain in the Debian kernel).

Ubuntu kernels have a lot more custom changes inside than Debian's as far as I know (they certainly used to), and you WILL get a major black eye when you do that and don't send your changes upstream. I am not sure if Ubuntu still deserves it, but they DO have a reputation of not bothering to send any of their kernel changes upstream in LKML (and that reputation obviously falls on Canonical shoulders).

Also, most Debian people I know that happen to also work on the kernel (and that includes myself!), do it directly *upstream*. Our work almost always gets into Linux mainline well before it ever makes it to any Debian kernel. Since Debian simply doesn't pay anyone to do any work (it is 100% volunteer-based), that work is NOT credited to Debian by most (any?) of us kernel developers that are also Debian developers.

For reference, you could credit something like 200 commits from me alone to Debian's name (I didn't check the date of the earliest commit, so some of those might be too old for the time-frame GregHK is using)... and I am not even one of the most prolific Debian developers that do upstream kernel work, AFAIK.

In the end, we Debian developers [that don't do kernel work in paid time by someone else] work upstream in the kernel without crediting Debian, and Debian nowadays doesn't have much to send upstream that is not in mainline. The obvious result is that Debian has a low patch-count.

THAT is why nobody is on Debian's case for a low patch-count. It isn't low at all, if you consider the "Debian developers doing work in the kernel the same way they do work for Debian", and that it is all unpaid work.

On the other hand, I *expect* Canonical to have a rule that anyone doing upstream work of any sort during paid time by Canonical, must do so using an email that credits it to Canonical. I am certain that RedHat, SuSE, Novell, IBM, Intel, and all others HAVE such rules in place.

So one really can't compare the Debian patch-count to the Canonical patch-count in any way that is even remotely favorable to Canonical...

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:37 UTC (Thu) by harinath (subscriber, #47697) [Link]

Since Debian doesn't pay its developers, there's no value to contributing upstream under the aegis of Debian: you might as well be a upstream developer directly, after all.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 16:34 UTC (Thu) by joey (guest, #328) [Link] (1 responses)

Debian has several developers who do work on the kernel. Ted T'so comes to mind. Problem for this kind of study is how to categorize such a person. Does Ted work for IBM, or is he a Debian developer, or is he a core kernel contributor? Well, all three. So who gets credited for his work in the statistics? Personally, I think that Ted should. We don't say that IBM, or VA Linux, or Debian is a major force behind ext3/4. Ted is. If the stats don't reflect this, there's something a bit fishy about them.

LPC: Fitting into the kernel ecosystem

Posted Sep 22, 2008 21:03 UTC (Mon) by niv (guest, #8656) [Link]

"Debian has several developers who do work on the kernel. Ted T'so comes to mind. Problem for this kind of study is how to categorize such a person. Does Ted work for IBM, or is he a Debian developer..."

Yes, Ted works for IBM.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:34 UTC (Thu) by tzafrir (subscriber, #11501) [Link] (1 responses)

How many of the current patches in the Debian kernel come from Ubuntu?

Debian and Ubuntu maintain to separate kernel versions with different sets of configuration.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 11:47 UTC (Thu) by maks (guest, #32426) [Link]

> How many of the current patches in the Debian kernel come from Ubuntu?

one: debian/patches/features/all/export-gfs2-locking-symbols.patch
needed to build gfs1 module needed for redhat-cluster irc.

very easy question as the Ubuntu kernel team has never even tried to cooperate. Packaging is different and diverged a lot. There is near *zero* communication.

Also they don't care to contribute their patches upstream! For 2.6.26 I wanted to make sure to have the interesting ones upstream merged. Some of them were found irrelevant by now, but other merged (df0bcab2c66ac876d5e80864fca5cce944a44540, 292d73551d0aa19526c3417e791c529b49ebadf3,
3b9408870757bd9e07fd03ac6318258f22b8dfa3,
a4fa7ef037b17f2a3b9b393cb924e571fc04e784, e1fefea9cc4bc231b5c23fe19e3682fe061dc097, a4fa7ef037b17f2a3b9b393cb924e571fc04e784).

Also it is true as all good people left Ubuntu (mjg59, kylem,..) their patchset is no longer that huge compared too earlier acpi hacks and so on.. You still find atrocities that would never find their way in a davej managed fedora kernel.

LPC: Fitting into the kernel ecosystem

Posted Sep 25, 2008 14:21 UTC (Thu) by SEMW (guest, #52697) [Link]

> How may employees to Red Hat, Novell, Wind River and Canonical each have? How many developers do Debian, Gentoo, Mandriva have?

Just sticking to the companies (I agree with filipjoelsson in that comparing companies with volunteer organisations like Debian in this is meaningless):

Canonical has 130 employees; Red Hat, 2200; Novell, 4100; Wind River, 1507; Mandriva, 80.

Patches per employee, then, are: Canonical, 0.77; Red Hat, 5.4; Novell, 1.77; Wind River, 0.14; Mandriva, 3.0.

This is still lowish on Ubuntu's part, but when you additionally take in account that Red Hat and Suse Linux have both been around 3.5 times as long as Ubuntu, and Mandriva 2.5 times, the numbers start to look somewhat less drastic than Greg presents them.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 4:15 UTC (Thu) by drosser (guest, #29597) [Link] (13 responses)

If you focused on what Greg was saying, and didn't get emotional about the way he said it, his points were correct and for the most part Canonical (via Mark Shuttleworth) agrees. For the community of developers who do "Linux Plumbing", the feedback loop from Ubuntu is currently disconnected. From Greg's position and point of view, this in inarguably a Bad Thing.

Speaking for myself, I'm a bit more ambivalent. The developer community Ubuntu must keep happy over all others is the Debian developer community, just as Red Hat and Novell must keep the ecosystems around Fedora and OpenSUSE alive and vibrant. If Red Hat Enterprise Linux and Suse Linux Enterprise Server feed off of their open communities, one could argue that Ubuntu has a stronger base from which to start and, given their desktop focus, not as far to go when building their distribution.

We should also keep in mind that Canonical gets press in far greater proportion to the size of their company. I would guesstimate that Red Hat has between one or two orders of magnitude more engineers on salary than Canonical.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:45 UTC (Thu) by Lovechild (guest, #3592) [Link] (12 responses)

I also think the emotional backlash is covering up what Greg is trying to say. If you consume upstream projects, you should value for your own sake contributing to them. Otherwise you have no influence on where they go, it might be in directions unsuitable to your goals forcing you to keep up with an avalance of changes (especially in the case of the kernel - having a feature out of the tree is a major amount of work to keep solid and tested).

That being said, seeing as the first line of defence against Greg's stats were that they didn't include contributions before they consolidated on the @canonical emails for contributions to the kernel. He corrected that, now mdz still complains about the statistics. The best way to prove that they are indeed providing patches would be to point them out. This time the complain didn't even have a point of correction attached, it was just that the stats might be wrong. Sure if they are, let's fix them. If you want glory and fame for your contributions as well as once and for all dispell the "Canonical doesn't give back" accusations then just prove that you do give back in measure.

One good thing that has come out of this whole debate seems to have been Canonical pledging to provide more work upstream. Just the other day they announced the hiring of a team of designers to help with interaction design. I can't wait to see who is on there and what they can do to help us. Aside that their Apport service is pure awesome, I wish though that it could be made more of an upstream project. As a feature it is vastly better than any other bug reporting tool and it helps make Ubuntu very pleasurable to use during their development cycle.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 9:43 UTC (Thu) by hppnq (guest, #14462) [Link] (4 responses)

According to the article this is what Greg was trying to say:

if you are a developer, if you want to be a part of the ecosystem, and if you work for a non-contributing company: quit.

This view is not uncommon in the business ecosystem.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 11:53 UTC (Thu) by Lovechild (guest, #3592) [Link] (3 responses)

You confuse the point as it relates to businesses consuming Free Software and developers employed by said companies.

If your business depends on Free Software, then as a business it is in your best interest to help shape it. If you work for one of these companies and they prevent you from participating then quitting is a sensible option - for you and for the company. Eventually brain drain will force these companies to reevaluate their position and take a more active role, till then good fortune has it that the sector is booming, jobs are not far between so quitting is not neccesarily a bad choice. That is if you, the developer, enjoy being part of the Free Software community and participating actively, if not then by all means enjoy your job.

Not once has the Canonical developers been blamed for Canonical' lack of contributions, that is a business decision, as such the business is the one with which the problem is debated.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 13:31 UTC (Thu) by hppnq (guest, #14462) [Link] (2 responses)

Indeed, why didn't Greg make his point in the Novell canteen instead of at this relatively high profile public conference?

The way I understood it, is that not a single distributor of (free) software is required to do anything, except adhere to the license(s) under which the software is distributed. Confusing this kind of clarity with more or less meaningless statistics and emotional pleas is business as usual.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 14:50 UTC (Thu) by mmcgrath (guest, #44906) [Link] (1 responses)

"The way I understood it, is that not a single distributor of (free) software is required to do anything"

This is the difference between the spirit and the letter of open source software. The letter must be followed, and is. The spirit should be, otherwise high profile developers will crap all over you for not being part of the ecosystem.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 20:08 UTC (Thu) by hppnq (guest, #14462) [Link]

Has it started already?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 18:04 UTC (Thu) by mdz@debian.org (guest, #14112) [Link] (6 responses)

If you consume upstream projects, you should value for your own sake contributing to them.

This is a popular meme in this discussion, but I think it misses the point. About half of the people in the room raised their hands during Greg's talk to indicate that they were running Ubuntu on their laptops. Does that mean that they should be expected to provide packaging and integration patches for Ubuntu? Of course not. It's free!

Many of them contribute to projects which are upstream of Ubuntu, and thus benefit it indirectly. Ubuntu, by creating a system that they want to use as their development platform, provides an indirect benefit as well. A lot of people use Ubuntu and don't contribute anything, directly or indirectly. This doesn't make them unethical or shortsighted.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 19:35 UTC (Thu) by dowdle (subscriber, #659) [Link] (4 responses)

I doubt those developers develop on their laptops... and perhaps do not use Ubuntu as a development platform.

I don't know one way or another... so I'm not going to assume they don't... as long as you don't assume they do.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 5:50 UTC (Fri) by Cato (guest, #7643) [Link]

Why on earth would a developer have a laptop and NOT develop on it? While laptops aren't ideal development platforms perhaps, they let you get a lot of work done while on planes, trains, etc.

Ubuntu works just fine as a development system - it's basically Debian with a nice UI and more polish, so lots of development tools are just an apt-get away.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 14:46 UTC (Fri) by bboissin (subscriber, #29506) [Link] (2 responses)

Actually I for example use Ubuntu on my laptop, develop for upstream projects.
Ubuntu is quite a nice distro but I'm not really happy with their relations wrt upstream. In fact for the packages in universe/multiverse, I prefer reporting the bug directly to debian otherwise I'm not sure if it will land in debian or upstream.

I would really prefer if ubuntu did less hacks to "fix" stuff, but instead worked with upstream to find the proper fix. I know it's harder and it takes more time but it benefits more people (and there are less chances for the fix to create a regression).

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 21:18 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

If you are not happy with your distribution's relationship to upstream, you should really express it to them especially if you are a upstream developer yourself. It is really important that distributions get the message that upstream contributions are the key to a healthy ecosystem and play a important part in it. Otherwise switch to a distribution which understands this.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 23:22 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Here's a question I have for you. How much of your discomfort over how "fixes" are handled in Ubuntu is there because of previous actions taken by Canonical employees specifically versus Ubuntu community volunteers or for that matter Ubuntu contributors who are employeed to work on Ubuntu packages by other companies.

Are the problems you see systemic in to the entire Ubuntu contributor base. Or are the problems you see associated with only the manhours that Canonical as a corporate entity has direct influence over how they are spent?

If you think the problem is systemic you should see if you can start a discussion inside Ubuntu to look at reforming at the package maintainership model that is being used. A systemic problem could be addressed by adjusting the team concept Ubuntu is using to add more individual accountability. For example teams could grow a specific tasking to just deal with upstream patch submission and sheparding and make a specific individual accountable for that in some way...if that sort of thing isn't there already. I'm thinking some sort of public flogging for failure to push patches upstream... or maybe a form of gladiatorial combat. Lot's of options really.

-jef

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 23:22 UTC (Thu) by salimma (subscriber, #34460) [Link]

Does that mean that they should be expected to provide packaging and integration patches for Ubuntu? Of course not.
If you're a user, you can contribute back by testing and reporting bugs (and perhaps provide patches for packaging bugs).

In that vein, I don't see why developers working for a Linux company should not be expected to have the ability to work with upstream.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 4:19 UTC (Thu) by jbailey (guest, #16890) [Link] (6 responses)

(obDisclosure: I'm a former employee of Canonical and still an Ubuntu Core
Dev)

I'm curious about the Debian/Ubuntu numbers in relation to one another.
How many of the Ubuntu contributors were contributing upstream before, and
are now simply associated with Ubuntu instead; How many of the Debian
contributors or contributions are paid for by Canonical.

My experience in doing stuff for Ubuntu was that the people contributing to
upstream tended to be the people who were doing it anyway, so made it a
part of the work that they were already doing. Those who weren't already
contributing upstream rarely started to do so.

Ubuntu hasn't managed to grow a culture of contributing upstream, even when
they tried directly to do so. It just somehow never worked out. Having
been there, I don't know what the right way to do that is.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 12:06 UTC (Thu) by maks (guest, #32426) [Link] (2 responses)

hello jeff,

> Ubuntu hasn't managed to grow a culture of contributing upstream, even when
> they tried directly to do so. It just somehow never worked out. Having
> been there, I don't know what the right way to do that is.

easy they prefer to employ people who waffle all day long like the guy whose title is Community Manager, but can't write a single line of code.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 17:03 UTC (Thu) by crimsun (guest, #13750) [Link]

I fully respect your work with initramfs-tools and many more, but to lambast Jono Bacon as someone incapable of writing a single line of code is a bit much. He may not be a kernel plumber, but he has contributed to KDE, Jokosher, and others.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 5:53 UTC (Fri) by Cato (guest, #7643) [Link]

The Ubuntu community is one of its greatest advantages: new users find it is friendly and helpful, and it's often quicker and easier to get problems solved through the Ubuntu forums than to pay for Windows support from Microsoft. It's also very hard for Microsoft or Apple to replicate this.

So the Community Manager is actually one of the most important people at Canonical, on a par with the gnarliest kernel hackers...

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 12:07 UTC (Thu) by mbanck (subscriber, #9035) [Link]

How many of the Debian contributors or contributions are paid for by Canonical.

That is hard to say without looking exactly at how they did come up with the numbers. However, I think in general one can say Canonical employees who are also Debian Developers tend to be careful not to mix up their email accounts when doing stuff with their respective hat on; so I would be surprised if a lot of patches had been submitted to the kernel by Canonical staff on company time with a @debian.org address.

Michael

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 14:51 UTC (Thu) by jmm (subscriber, #34596) [Link] (1 responses)

> How many of the Debian contributors or contributions are paid for by
> Canonical

Beyond what people post with their debian.org address, there´s quite a number of Debian developers working on the kernel in their regular day job.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 21:57 UTC (Thu) by jeffm (subscriber, #29341) [Link]

If they're doing the kernel development work as part of their day job, why should a distribution they also perform work with receive the credit for the work? Their day job has funded the work.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 7:07 UTC (Thu) by fcrozat (subscriber, #175) [Link]

There seems to be an error in the table order, between Mandriva and Gentoo : eitheir the numbers for both distros where switched or sorting is wrong ;)

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:25 UTC (Thu) by pkern (subscriber, #32883) [Link] (4 responses)

One important thing to note (IMHO) is that Ubuntu didn't reuse Debian's kernel packages, at least not in a way they could contribute fixes back to Debian. (Different kernel versions, probably also quite different packaging.)

So if at all they'd need to contribute back to Linux upstream...

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 13:08 UTC (Thu) by cortana (subscriber, #24596) [Link] (3 responses)

It's always seemed rather strange to me that the Ubuntu and Debian kernel teams don't work on a common kernel package.

Ubuntu-Debian shared kernel

Posted Sep 18, 2008 15:38 UTC (Thu) by rfunk (subscriber, #4054) [Link] (2 responses)

The kernel team makes a major release every three months.
Ubuntu makes a major release every six months.
Debian makes a major release about every 18 months or so.

Now do you see why Ubuntu and Debian can't share kernel packages?

Ubuntu-Debian shared kernel

Posted Sep 18, 2008 23:25 UTC (Thu) by salimma (subscriber, #34460) [Link] (1 responses)

Now do you see why Ubuntu and Debian can't share kernel packages?
No. Why can't the kernel package be stabilized the same way other packages are stabilized? Base it off Debian's unstable distribution rather than the stable release.

Ubuntu-Debian shared kernel

Posted Sep 19, 2008 13:05 UTC (Fri) by rfunk (subscriber, #4054) [Link]

Other packages have longer release cycles than the kernel has.

Also, Ubuntu and Debian have different policies about what is allowed to
go into the kernel. Ubuntu allows "binary blobs" that Debian does not.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 8:25 UTC (Thu) by pointwood (guest, #2814) [Link] (2 responses)

I'm not going to argue whether his numbers are wrong or right, but I think his definition of the "Linux ecosystem" is wrong. I would say that all the free software that's included in the various distributions repositories are part of the Linux ecosystem.

Canonical has clearly focused a lot on the desktop (where Redhat and Novell have been more focused on the server side), particularly a desktop based on Gnome and they have no doubt been pretty succesful in that regard. A lot of what they have done is polish and making things "just work", stuff that I think can be hard to measure.

To get a better picture of how much or how little Canonical have contributed to the ecosystem, you need to include all the software that's part of Ubuntu.

Canonical isn't exactly a big company so there are limits as to how much they can contribute. I think part of the reason they are in focus is because they have been quite succesful on the desktop and gotten a lot of users. Would the critic be different if it was for example Slackware?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 12:14 UTC (Thu) by mbanck (subscriber, #9035) [Link] (1 responses)

Canonical has clearly focused a lot on the desktop (where Redhat and Novell have been more focused on the server side), particularly a desktop based on Gnome and they have no doubt been pretty succesful in that regard. A lot of what they have done is polish and making things "just work", stuff that I think can be hard to measure.

Sure, however if you would compare patches/lines of code contributed to GNOME for Redhat, Novell and Canonical, I am pretty sure there would be again a difference by several orders of magnitude between the first two and Canonical.

Michael

LPC: Fitting into the kernel ecosystem

Posted Sep 21, 2008 10:36 UTC (Sun) by kripkenstein (guest, #43281) [Link]

Well, first it would be good to see some actual figures. However, it seems they aren't available at present.

But even if it is a difference of, say, two orders of magnitude, that is probably about the difference in size between Canonical, Canonical's relevant dev teams, and Canonical's revenues in comparison to Red Hat and Novell's corresponding figures.

Once Canonical is profitable (which Red Hat and Novell already are), I will be very critical if it doesn't step up collaboration. Until then, I would encourage it but I won't be critical regarding that matter.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 9:03 UTC (Thu) by herodiade (guest, #52755) [Link] (4 responses)

  • The raw patchs numbers should be pondered by the size (number of employees) of companies, if one want to evaluate how good citizen a workplace is. Big corporations doing huge revenues from Linux mostly by selling proprietary components on top, having less than a few percent employees contributing f/oss and tons of marketing droids and devs on proprietary stuff, patenting everything in sight will produce more patchs than a 4-persons company of wich 3 are developers but that produce only free software. Which indeed don't mean the big corp is a better place to work for devs liking opensource.
    Think Adobe vs. Tresys, or Oracle vs. Hwaci, or Google vs. Linuxtronix.
    If Canonical would be found contributing an average 10 patches per employee while Novell would contribute, say, 5 patches/employee, then the GKH conclusion should be reversed: it would mean Canonical would be a better place than Novell for a developer that want to work with the community; it would mean they do a better job given their size/ressources.
  • GKH has been pretty clear about what he was accounting. He said Canonical is not big Linux plumbing (kernel, xorg, gcc, ...) contributor. Since their distro sells itself (or is mostly perceived) as a usable and polished desktop, maybe their works would show up in higher level, "non plumbing" components (like gnome and so) ? It would be interesting to have contributions numbers on the high level, desktop components.
  • The best effect this talk may have would be to reach Mark Shuttleworth ears. Then maybe he would think about those problems right at the design phase of his projects. That could prevent futures proprietaries launchpad like stuff, or their badly designed (wrt upstreaming contribution) online translation environment. Those are place where, imho, Canonical actual work with the community leave much to be desired. Mark had been deaf to that sort of comments on how Canonical work with upstream up to now. Thanks to such public bashing, he may eventually stop denegating problems.
  • There's a few commercial debian derived distributions that seems to make huge deals and money from linux nowadays (with the rise of umpc), but are so scarce contributors that we don't even think about accounting hem here. I think Xandros is competing for the Great Leecher title.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 9:39 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

> The raw patchs numbers should be pondered by the size (number of
> employees) of companies, if one want to evaluate how good citizen a
> workplace is.

Ubuntu/Canonical is the first to insist it should be given precedence because of its number of users. It can not ask to be considered as a small entity when there is work to do, and as a big entity when there are objectives to set. If Canonical wants to be taken as a big player, it must assume big player responsibilities.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 18:24 UTC (Thu) by mdz@debian.org (guest, #14112) [Link] (2 responses)

A great majority of Canonical's work shows up in Ubuntu itself, not in upstream projects. Most of what we do is packaging and integration, making the whole mess actually work for end users, not writing it in the first place.

Sometimes this involves writing patches. Some of those patches are appropriate to contribute back, either to Debian (packaging patches) or further upstream (code patches). We try to do that, and we do a better job with some projects than with others. Some of the patches actually belong in Ubuntu and don't make sense anywhere else.

We've actually gone out and asked upstream projects for feedback, via a survey, and are working to improve collaboration with them based on that. This all comes down to having a personal relationship where both sides understand what to expect from each other.

Greg, unfortunately, prefers a different approach, which involves telling people with no influence over the situation how bad he thinks it is.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 8:16 UTC (Fri) by jospoortvliet (guest, #33164) [Link] (1 responses)

You should notice that this is about appearances more than reality. Ubuntu appears to many users as one of the major distributions - the often-quoted top-three is Red Had, Novell and Cannonical. That is (and should be) surely something you can be proud off.

But this leads to people also expecting Cannonical to be in the top-three in terms of upstream contributions. And no matter how you put it, it is not.

This discrepancy tends to be very annoying to other organizations who are less visible, yet do more in terms of code contributions.

You could say Cannonical/Ubuntu simply does a better job at advertising its work. This is not in a small part due to the way it communicates and works - a rather revolutionary way (which has been copied by the Fedora and OpenSuse concepts). That's a good thing, don't get me wrong, but it creates a sense of unfairness in some minds, and rightly so.

A good solution would be to communicate more honestly about the amount of contributions Cannonical does. The general public will hardly notice it, and it won't hurt Cannonical in terms of marketing, but it will alleviate the percieved unfairness.

At the same time, Cannonical should obviously do whatever it can to increase upstream contributions. Putting the upstream work in numbers somewhere on the Cannonical/Ubuntu site would help in this regard, as it would make it one of the priorities in the Ubuntu community (and make the whole discussion more transparent at the same time).

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 16:16 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Correction... Fedora came first. In fact Fedora inspired Ubuntu's own effort to build "A better Fedora than Fedora."

If you have a hard time understanding where that quote came from, you should probably read this blog post, it is an excellent read:
http://gregdek.livejournal.com/32787.html

And this video is really nice too:
http://www.redhatmagazine.com/2008/09/16/video-the-histor...

-jef

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 10:59 UTC (Thu) by dag- (guest, #30207) [Link] (2 responses)

I accept most of what Greg had to say (at least through LWN's eyes) but there is one thing I would contest:

----
Then, there is the matter of redistributors who base their products on another distributor's work; these are distributors like Ubuntu or CentOS. There are no contributions back to the community from that kind of distributor at all. They are not functioning as a part of the Linux ecosystem.
----

CentOS is different than a normal distribution (at least for the core OS) in the sense that our aim is to be 100% compatible with RHEL. That of course means we cannot change anything (except the things we legally have to). Bug-reports and patches go up to Red Hat and we wait for them to be accepted and pour down again.

Now, I don't think CentOS in itself is a big contributor to patches, much like the article already explains why RHEL is not in itself a big contributor. But saying CentOS functions outside the Linux ecosystem is probably one bridge too far and depending how you interpret "ecosystem" it is wrong or intentional :-)

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 11:25 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

The presentation itself has more details at

http://www.kroah.com/log/linux/lpc_2008_keynote.html

"The "enterprise" release lasts a long time. These packages that were taken is a snapshot in time, that is slowly maintained and updated with bug and security fixes by the developers working for the distros. Those changes and fixes flow back into the original projects for inclusion in their main releases when possible, but for the most part, the majority of changes that go into these releases come from newer releases from the projects, so changes flow predominatly one way into the distro."

Given this, I don't think it was meant as a slight against CentOS. Just a general statement about the nature of enterprise distributions one of which CentOS is based on.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 13:20 UTC (Thu) by smoogen (subscriber, #97) [Link]

Dag,

For the most part, patches and fixes that CentOS devs make goes up one step into Red Hat bugzilla. Others go all the way upstream.

The proper counter to all this is the size of the team of people. I think CentOS has 5-8 people who are considered core devs.

Having such numbers/size might balance more of the fixes that Canonical, etc make upstream.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 15:27 UTC (Thu) by tle@holymonkey.com (guest, #47821) [Link] (4 responses)

An objective listing of the numbers would have been fine, but the negativity expressed in Greg's presentation is disturbing. The editor (who usually isn't afraid to inject his own opinion) is being unfair by reporting on the "picking on Canonical" without pointing out that it is plain wrong.

All Ubuntu components are released under an open source license. Patches and bug reports are actively pushed upstream. They should be praised for the contributions they make to the kernel and other projects, and encouraged to contribute more.

Instead of putting down organizations who contribute only a little, please focus your energies on praising those that contribute more. Any contribution should be gracefully accepted.

Cheers,
Tim

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 16:12 UTC (Thu) by tzafrir (subscriber, #11501) [Link] (2 responses)

If everything works so well, how is it, then, that the bugs and troubles Ubuntu's user-base generate so few upstream patches? At least in those plumbing packages. Is it because they don't fix problems or because they don't forward fixes upstream?

A I pointed out elsewhere in this thread: I don't think Ubuntu is required by some ethical standards to contribute. It should make sense for Ubuntu for economical reasons to do so. While I appreciate RedHat, Novell and others, I don't suppose they employ so many developers just for the PR value.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 17:37 UTC (Thu) by tle@holymonkey.com (guest, #47821) [Link]

> At least in those plumbing packages. Is it because they don't fix problems or because they don't forward fixes upstream?

I agree that the discussion should focus on a distribution's _process_ of bug reporting and submission of patches to upstream.

Ubuntu does all of their bug reporting, patch submission, and upstream reporting in the open. Just like any open source project, don't just complain about something not working. At the VERY least, look at the process, and point out where it is failing.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 18:55 UTC (Thu) by mdz@debian.org (guest, #14112) [Link]

Users' bugs and troubles don't generate patches: developers do. Ubuntu has millions of users, and dozens of developers (only some of whom work for Canonical). We receive several thousand bug reports per month, for the entire free software stack.

We focus primarily on fixing integration and packaging problems (which are our unique responsibility), followed by the most critical problems we inherit from upstream. This is similar to how Debian works, though the scales involved are dramatically different.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 20:44 UTC (Thu) by hppnq (guest, #14462) [Link]

The editor (who usually isn't afraid to inject his own opinion) is being unfair by reporting on the "picking on Canonical" without pointing out that it is plain wrong.

See this Linux foundation paper and you will notice that Greg and your editor cooperated on the research that (very probably, no specific source is mentioned) produced these statistics.

It surprised me to not see a reference to this paper, I must admit. Especially since it raises a specific point that many here seem to miss in their eagerness to shoot from the hip: that Linux kernel development does not really need the contributions of even more companies.

So why the commotion? We could be having a nice discussion about whether we had not better call this ecosystem "GNU/Linux". ;-)

debian-specific numbers

Posted Sep 18, 2008 17:07 UTC (Thu) by joey (guest, #328) [Link] (3 responses)

My guess is that Greg K-H is mostly counting patches from @debian.org addresses to get his debian numbers. Since I get similar, though not identical, numbers from doing that. But there's another way to do it: Since it's easy to get a list of all current Debian developers. I generated such a list, and then grepped for commits by each name to the kernel in the past 5 years.

In total, there were 131 DDs who had committed a change to the kernel, and the total number of changes committed by all DDs was 3127.

Here are the top DD kernel committers, limited to the ones who have more commits than Greg's numbers say Ubuntu has. ;-) Just for kicks, I've included the organization they have in their email address too.

502 Roland Dreier (cisco.com)
339 Matthew Wilcox (wil.cx)
310 Roland McGrath (redhat.com)
236 Nathan Scott (aconex.com)
209 Henrique de Moraes Holschuh (eng.br)
178 Kyle McMartin (mcmartin.ca)
113 Brice Goglin (myri.com)
112 Mark Brown (sirena.org.uk)

Note 1: I suspect that the stuff Jon developed and Dave is using is much better than my stupid shell script. This is more of a back of the envelope calculation to see how a different approach can yield significantly different results. These numbers should not be trusted, used in presentations at confereneces, or used to tell people to quit their jobs. (If you do, don't put my name on the slides.)

Note 2: No attempts were made to deal with the issue that different DD and kernel committers could easily have a name collision.

Note 3: Most of these large committers are not on Debian's kernel team, which seems like Debian is missing an obvious opportunity to have a better maintained kernel.

debian-specific numbers

Posted Sep 18, 2008 17:17 UTC (Thu) by joey (guest, #328) [Link]

Update: According to Greg's slides, to figure out who to count for Debian he "guessed and looked at who they list as their kernel team, combined with a few email addresses that do use a debian.org address".

debian-specific numbers

Posted Sep 18, 2008 23:07 UTC (Thu) by dilinger (subscriber, #2867) [Link]

> Note 3: Most of these large committers are not on Debian's kernel team, which seems like Debian is missing an obvious opportunity to have a better maintained kernel.

...which won't happen until the politics and headaches involved in being a kernel team member are gone. Having a benevolent dictator who is able to communicate with others would be a good start.

debian-specific numbers

Posted Sep 19, 2008 0:20 UTC (Fri) by nix (subscriber, #2304) [Link]

Well, having Roland in there is going to have an impressive effect on
commit counts for other projects too (glibc, GNU make...)

There aren't very many active free software developers who started as many
core projects as Roland (although part of that was that he started doing
GNU stuff back in the eighties...)

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 17:27 UTC (Thu) by crimsun (guest, #13750) [Link] (6 responses)

Disclaimer: I am a former Ubuntu core dev (albeit nonpaid).

I feel Greg's keynote was a bit heavy-handed, but certainly a "better job" could be done to contribute more visibly. That said, Ubuntu, as a distribution, is doing a fairly decent job of getting novice users interested in tinkering with the plumbing. While, as a distribution, it is certainly not unique in that regard, I'd like to emphasise one thing:

We're missing the point, which is that it matters not which company is contributing "the most", because Ubuntu is also "about" enabling its users/consumers to become familiar with the plumbing so that they can contribute.

In the end, we're all consumers.

I've mentored several people (and continue to do so). We start with smaller things, like adding quirks to various HDA codec patches in ALSA, and progress to debugging interoperability concerns in the entire audio stack (e.g., why does GNOME's mixer applet throw an error? Oh, because it lacks an HDA codec patch. Let's fix that then figure out why this legacy audio app is abusing the ALSA API, etc.).

Enough already with assigning blame. It's time to demonstrate how "ordinary people" make a difference. In fact, I'm giving a talk at this year's Ohio Linux Fest on just how to empower users to fix their audio stacks.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 18:59 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (5 responses)

It's not about which companies is contributing..the most.

It's about which companies are contributing "enough." Is Canonical contributing enough? When Shuttleworth talks about Canonical's business needs, and how Canonical needs to see upstream projects change how they are doing things to better serve those business needs... is Canonical contributing "enough" to those upstream projects to have their business needs seriously considered? Shuttleworth has been very public about challenging core upstream projects like the kernel and X.org to do things differently for Canonical's benefit. It's only fair that Canonical be challenged to pony up the manpower to make that happen.

A significant amount of the heat being generated now is in direct response to public statements Shuttleworth has made. Which other company in that list of companies has a CEO which likes to publicly blog about how he feels upstream projects should be doing things?

If you just stick with actual corporate entities who are paying developers to do open development... is Canonical doing their fair share to re-invest in the open source infrastructure that everyone is using?

Right now, at this very moment in time...
If you are a skilled open source developer, with intimate knowledge of one of the critical subsystems that make up a linux desktop, looking to get a pay check for your work, is Canonical's commitment to supporting the ecosystem strong enough for you compared to other potential employers looking to hire you?

I'll let you in on a little secret. The pacing item in the continued advancement of the open ecosystem are upstream developer manhours.... not downstream integrator manhours...nor even users. The real work and challenges are in front of the upstream projects themselves. As downstream integrators, we should work on increasing the available upstream development manhours, or else we we'll run out of new things to integrate.

-jef

What is enough?

Posted Sep 19, 2008 11:33 UTC (Fri) by grantingram (guest, #18390) [Link] (1 responses)

It's about which companies are contributing "enough." Is Canonical contributing enough?

That is a good point but Canonical is currently a loss making Linux start up. A well funded and high profile start - but it still is not turning a profit. How much contribution do you expect from them? What level of contribution and where would satisfy you?

Shuttleworth has been very public about challenging core upstream projects like the kernel and X.org to do things differently for Canonical's benefit.

Well the idea of coordinated releases is not just being sold on the basis that it is good for Canonical but also good for everyone....

The real work and challenges are in front of the upstream projects themselves. As downstream integrators, we should work on increasing the available upstream development manhours, or else we we'll run out of new things to integrate.

Here I disagree! The remarkable success of Ubuntu demonstrates that applying large amounts of "polish" to existing software can have an enormous effect on the success of the free software ecosystem as a whole. Clearly we need upstream development efforts but "polish" is also very valuable!

What is enough?

Posted Sep 19, 2008 16:37 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Here is what I expect from Canonical. I expect Canonical's CEO to spend far less time publicly speaking about what he thinks upstream projects and contributors should be doing with their time. I think the vast, vast majority of the criticism that Canonical is getting, and why Canonical is being singled out, is because Shuttleworth has made it a point to go to the press and to conferences and talk about interactions with upstream projects as the CEO of Canonical, to better meet Canonical's business interests.

By doing these sorts of things, Shuttleworth has himself raised expectations on Canonical's own participation in the upstream process.

You don't see him singling out other companies by name, no Tivo, or Linspire or Nokia or YellowDog or whomever. There are probably a ton of companies out there trying to build a business around some sort of linux "distribution" in some shape or form and making small but targeted contributions to upstream projects. That's not the issue.

The issue here is is that the very outspoken CEO of Canonical is out making public statements, challenging upstream projects to change how they are doing their work to benefit Canonical...without providing additional resources to those projects.

If Tivo's CEO decided to be as public as Shuttleworth about seeing upstream processes changes, I would fully expect Tivo to be challenged to provide manpower as well. It's really as simple as that.

Canonical is in the spotlight, because Shuttleworth put Canonical in the spotlight.

-jef

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 16:36 UTC (Fri) by AJWM (guest, #15888) [Link]

As downstream integrators, we should work on increasing the available upstream development manhours, or else we we'll run out of new things to integrate.

It seems to me, from crimsun's message above, that he and Ubuntu are doing that by providing a training ground for developers new to Linux. Sure, not everyone who tweaks an audio stack or whatever will go on to be an upstream developer, but some will. Not everyone who goes to school goes into research to discover new knowledge, either -- that doesn't mean we should berate schools for not doing enough.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 16:54 UTC (Fri) by markshuttle (guest, #22379) [Link] (1 responses)

I think you must be referring to my call for syncronization and coordination in the stabilisation and release of diverse components in the free software stack. That's based on my belief that it will make all of free software more effective in the battle with proprietary software, not based on any specific advantage to Canonical.

Of course, Ubuntu is best known for our commitment to a firm cadence of releases - we do them every six months, and it works well for us and our users. We took that up from GNOME, who pioneered the approach in large aggregated projects. Other projects, like the kernel, are already converging on a regular cadence of releases. There are lots of examples in nature and in economics that demonstrate the gains in efficiency that come from such syncronization. And every participant, include Canonical and its competitors, would benefit.

I'm not trying to tell anybody what they should do, to benefit Canonical. I'm pointing out that we will all be more successful if we think systemically, and do what nature does.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 17:29 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

See here's the thing. There's all sorts of ways to synchronize a set of systems. For a vast array of highly non-linear, weakly couple systems, you can get what is called stochastic synchronization. its a pretty fascinating process...very cool...easily demonstrated with a set of weakly coupled pendulums. One of the coolest demos is with a couple of metrones riding on top of a wooden plank which is rolling on top of a couple of full cola cans. Really its a really cool thing to watch happen.

You start out with components, with random phasing and even frequencies somtimes. But through their own natural weak coupling, these components spontaneously reach a state of synchronization for long periods of time,until such time that the weak non-linear coupling interactions takes the system back to an unsyncronized state.

Here's the coolest part, if you try to force these sorts of systems from the outside, you end up breaking their natural cyclic nature by enhancing one of the weak non-linear coupling processes. Your driving force finds some sort of destructive resonance.. and you break the system.

So here's my suggestion to you. Be a weak non-linear coupler. Don't be a strong driving force. Help the naturally complex non-linear system do its thing and find its own stochastic syncronization. Don't break it.

-jef

a bad precedent?

Posted Sep 18, 2008 19:55 UTC (Thu) by sbishop (guest, #33061) [Link] (9 responses)

I wish Greg had simply posted his comments to his blog. Using an engineering-focused conference to attack a distribution sets a bad precedent. I'm not passing judgment on what Greg said, but I think that it is important for the politics and engineering of Linux to be kept as separate as possible.

a bad precedent?

Posted Sep 18, 2008 20:02 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (7 responses)

Correction... he was attacking a company...not a distribution.

Canonical!=Ubuntu

-jef

a bad precedent?

Posted Sep 19, 2008 2:28 UTC (Fri) by sbishop (guest, #33061) [Link] (6 responses)

"Correction... he was attacking a company...not a distribution."

If flat out telling a distribution's core developers to quit isn't an attack on that distribution, then I can't imagine what _would_ qualify...

a bad precedent?

Posted Sep 19, 2008 5:31 UTC (Fri) by bronson (subscriber, #4806) [Link] (5 responses)

Did GregKH "flat out tell" Canonical employees to quit?

He did say that if your current employer discourages you from contributing your work back upstream that you should quit, yes. Is there anything wrong with this sentiment? It applies to Via, TI, and all these other pseudo-open-source sweatshops that dump an unlicensed almost-working ball of code on some FTP server and disappear. Is this an accurate description of Canonical? Probably not.

a bad precedent?

Posted Sep 19, 2008 11:05 UTC (Fri) by grantingram (guest, #18390) [Link] (4 responses)

Did GregKH "flat out tell" Canonical employees to quit?

Well almost... I was quite astonished when I read his slides! The whole thing strikes me as really rather unpleasant. I'm really not sure what the aim of the talk was other than to generate heat. Perhaps it was cold....

a bad precedent?

Posted Sep 19, 2008 11:48 UTC (Fri) by AlexHudson (guest, #41828) [Link] (3 responses)

Not exactly unheard of in the other direction, though ;)

The Lightbulb Moment!

Posted Sep 19, 2008 15:19 UTC (Fri) by grantingram (guest, #18390) [Link]

I'd forgotten about that - suddenly it all starts to make a bit more sense... a sort of flame war conducted over twenty three months.

a bad precedent?

Posted Sep 19, 2008 15:46 UTC (Fri) by jspaleta (subscriber, #50639) [Link] (1 responses)

More evidence that the strong reaction to Canonical, is a reaction to Mark Shuttleworth's previous public statements as the CEO of Canonical.
He's very outspoken. That can have a downside. Trust me, I know intimately all about the downside of being too outspoken.

And in this specific case, it could be chalked up to a "reap what you sow" moment.

That was a pretty aggressive move on Shuttleworth's part. Taking negative public sentiment, and using it to bolster your own company's interests by calling out another company for taking advantage of thousands of programmers and contributors. Who would have thought that other entities would replicate Shuttleworth's aggressive tactics at community manipulation. Mark should have patented that idea, then he wouldn't have to worry about competitors doing the exact same thing to him on down the line.

I think if Shuttleworth decided to take a step back from the public eye, and let the technical leads at Canonical who are actually doing the work do the bulk of the communication about any Canonical activity...that could help reorient relations. He would be better of letting the technical people inside Canonical lead with action, and following up on that technical leadership by talking about it publicly. Instead he spends a lot of time talking about what Canonical will do or what Canonical is thinking about. He primes the pump, he stokes the flames...he just doesn't realize whose pumps and flames he's priming and stroking.

-jef

a bad precedent?

Posted Sep 19, 2008 21:45 UTC (Fri) by nix (subscriber, #2304) [Link]

Sounds like he's being a classic undiplomatic geek to me. :)

a bad precedent?

Posted Sep 19, 2008 6:06 UTC (Fri) by Cato (guest, #7643) [Link]

I agree - it really was an attack on Canonical, and even if justified should have made it very clear that he works for Novell, who have a direct interest in not seeing Canonical succeed, even if Greg is only focused on how the community works.

Overall, I think Canonical does need to contribute more, particularly around the kernel where I've encountered some annoying bugs. However, they first need to get more revenue - unlike Red Hat and Novell, they don't have a significant revenue stream from enterprise support agreements. In fact the ratio of paying customers to non-paying users is far, far higher for Red Hat / Novell than Canonical/Ubuntu.

Personally I find the Ubuntu stance on free software and particularly patents infinitely preferable to Novell getting into bed with Microsoft. If Novell's upstream contributions really depend on the Microsoft deal, it would be a lot better in the long-term if they scaled back upstream work and tore up the Microsoft agreement.

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 20:53 UTC (Thu) by leoc (guest, #39773) [Link] (1 responses)

So is there some online equivalent to that white board mentioned at the end?

LPC: Fitting into the kernel ecosystem

Posted Sep 18, 2008 21:23 UTC (Thu) by mdz@debian.org (guest, #14112) [Link]

There wasn't much to it, but I took a photo which I can post when I get home (I forgot the cable for my camera).

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 0:18 UTC (Fri) by tpo (subscriber, #25713) [Link] (2 responses)

The discussion, the heat and the hate here seem to be centered around the problem of freeloading, which is a moral argument. Yet most devs, including many participants here, are being paid for their Linux work. Making money is fundamentally not a moral motivation [1], but never the less, many feel the need to lambast others for "freeloading". Quite weird, isn't it?

So Ubuntu doesn't contribute as much as Redhat and Novel. How come? RH and Novel have lots of developers that contribute "upstream", Canonical doesn't. How come? RH has a *lot* of industry clients. A RH license sells in the range of a $1000 per seat. Is such an amount for a free OS frivolous or amoral? Now how much does an Ubuntu license cost? Does that fact give RH moral superiority over Canonical? I can't argue about Novel, since I don't know about them.

After many years I went to a LUG meeting recently and I was absolutely overwhelmed by the fact that half of the Linux enthusiasts (!) there had absolutely not _any_ command of the shell. Never ever touched a shell in their life. They were _not_ running RH of course and this feat was _not_ the achievement nor of Novel nor of RH.

The command of a shell and contributing back what "upstream" seems to percive as worthy could be correlated.

A last thing. Let's assume Mark Shuttleworth would get sick and tired of the flames, finally realize, that the other companies are in it for the money (are they?) and not for trying to serve humanity, shut his company down, fire all the Debian devs getting paid by him and thus /dev/null Ubuntu from this world. Would that be a desirable evolution? Weren't Canonical only freeloading anyway?

*t, from the shoulders of ancient giants

[1] It seems making money off Linux is a pill that has been swallowed a long time ago.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 8:05 UTC (Fri) by seyman (subscriber, #1172) [Link]

> this feat was _not_ the achievement nor of Novel nor of RH.

Please read the studies that Novell and Red Hat have done of the usability of the Linux desktop, the patches that both have contributed to GNOME and KDE and go look at the hosting they've provided for the two projects.

LPC: Fitting into the kernel ecosystem

Posted Sep 23, 2008 18:05 UTC (Tue) by dag- (guest, #30207) [Link]

For a normal business hours support contract, you pay $349/year for RHEL or SLES, and $750/year for Ubuntu LTS for similar class of hardware.

For a 24/7 support contract, you pay $1500/year for SLES, $2500 for RHEL and $2750/year Ubuntu LTS.

Even though Ubuntu LTS is also available for free, and RHEL has a free CentOS spinoff, prices are not that much different and Ubuntu LTS is not cheaper than RHEL or SLES.

Source: https://www.redhat.com/security/updates/errata/
http://support.novell.com/lifecycle/
http://www.ubuntu.com/support/paid

Date: 23/09/2008

Other contributions than code, too

Posted Sep 19, 2008 7:54 UTC (Fri) by tajyrink (subscriber, #2750) [Link]

Canonical has via Ubuntu contributed more of helpful, friendly attitude in the Linux user world than all other major distribution vendors combined.

I'm thinking that Ubuntu is something that shakes us techie nerds to realize there is more than code in the world. There is helping, community, friendship. You may think 'ubuntu' is marketing bs, I think it's the thing that makes Ubuntu what it is. Also, it's marketing, but maybe in addition to accepting "money" in Linux world we should not hate "marketing". They are all part of this world, anyway, and not evil by default but means to achieve something. I love Fedora's infinity etc. thing, too, it has a touch of also some non-technical in it.

This comment is as much to Greg KH (who has partially a point and partially is just offensive) as some of the other commenters. I'd hope for Greg and others also to read through Ubuntu's Code of Conduct and maybe even agree to it.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 15:26 UTC (Fri) by MattPerry (guest, #46341) [Link] (2 responses)

Greg needs to talk to a lawyer and have them explain the GPL to him. Canonical isn't required to contribute any patches to the kernel. Greg should be thankful for what he gets. As long as Canonical abides by the license then they are fine. If Greg doesn't like that then he's free to use a different license for his code.

miss the point

Posted Sep 19, 2008 16:24 UTC (Fri) by alex (subscriber, #1355) [Link]

No one is saying Canonical *have* to contribute. However as "members" of the "community"* they shouldn't be immune to people questioning their actual record of contributions which is all I think GregKH did.

* intentional quotes, the community is vague amorphous blob of anyone with an opinion of FLOSS and there is no real membership.

LPC: Fitting into the kernel ecosystem

Posted Sep 19, 2008 18:17 UTC (Fri) by jikos (subscriber, #43140) [Link]

Greg is not saying that what Ubuntu developers do (or rather do not) is illegal at all.

He just says (as far as my undersntading goes) that they don't contribute to the "ecosystems" as much as would be appropriate with respect to how large userbase they have. Nothing more, nothing less, just a simple feeling/opinion statement.

LPC: Fitting into the kernel ecosystem

Posted Sep 21, 2008 22:49 UTC (Sun) by boog (subscriber, #30882) [Link]

Why so much hate? Ubuntu (which I don't use, but would recommend to new users) has been stunningly successful with a relatively small investment in a very short time, but surely Canonical has many fewer developers and much less money than Red Hat/Novell/IBM, and they have had less time to sort out their way of doing things. So mistakes in contributing upstream seem more like errors of youth. And the Ubuntu community educates and cuddles all those new users - something that lkml is not very good for.

Dear Greg ...

Posted Sep 21, 2008 23:14 UTC (Sun) by Felix_the_Mac (guest, #32242) [Link]

"Greg finished up with what appears to be the message he came to the Linux Plumbers Conference to deliver: if you are a developer, if you want to be a part of the ecosystem, and if you work for a non-contributing company: quit."

My message would be if: If you care about free software and you work for Novell: Quit.

LPC: Fitting into the kernel ecosystem

Posted Sep 22, 2008 16:47 UTC (Mon) by drag (guest, #31333) [Link] (1 responses)

It's pretty rude stuff coming from Greg.

I'll try to be brief:

* Yes it is true that Ubuntu takes the work of others and uses it.

* This is how it is _suppose_ to work. Debian fights tooth and nail for good licensing for the expressed purpose of letting other people use their software and redistribute it without worries. Ubuntu is really just doing what they are suppose to be doing.. reusing, redistributing. Kernel developers do the same thing for 'software freedom'.

(Greg is one of those hugely outspoken pro-Freedom sorts, but yet when somebody else takes advantage of this freedom he takes a shit on them?)

* Making userspace better and improving graphics and UI is MORE important then hacking on the kernel nowadays for a very very large number of users. The sort of users that Ubuntu caters to.

* Making software usable is significantly harder then making software.

---------------------------------

Ultimately it's all just petty jealousy. People don't like that Ubuntu isn't doing most of the work, but is getting all the credit. So they are overly critical of Ubuntu. Instead they should just be asking Ubuntu to help users understand the software and positive (and negative) experiences come from far more then Ubuntu.

Look at this way....

Would Dell be supporting Linux on the desktop and improving support for their hardware without Ubuntu? Would Intel be interested in supporting Linux heavily on their new embedded systems if the only user experience their customers would receive would be the generic Redhat-style shit desktop experience?

Ubuntu, for very good reasons, is able to push the popularity of Linux and this has had very significant and positive effects on the whole Linux 'ecosystem'

Ignoring this fact is just expressing a form of willful ignorance.

LPC: Fitting into the kernel ecosystem

Posted Sep 22, 2008 19:53 UTC (Mon) by polar (guest, #51861) [Link]

Fair arguments, but he did make that talk at the Linux /plumbers/ conference.

An alternative view

Posted Sep 23, 2008 13:41 UTC (Tue) by cyperpunks (subscriber, #39406) [Link] (1 responses)

If I wanted a Linux distro with support, which would I a prefer?

A company full of people writing kernel code or some guys doing desktop integration?

An alternative view

Posted Sep 23, 2008 18:49 UTC (Tue) by dlang (guest, #313) [Link]

that depends who you want to have using it.

if you are talking about a unix greybeard, you want to pay the company doing the software development

if you are talking your parents/grandparents/etc you may want to pay the company doing the desktop integration, since that's what makes the system usable for them (and keeps your phone from ringing off the hook with their questions)

LPC: Fitting into the kernel ecosystem

Posted Sep 23, 2008 22:42 UTC (Tue) by frankie (subscriber, #13593) [Link] (2 responses)

In a past google talk, Greg expressed the same observations about Google contributes. The only true Google contributor is Andrew Morton, which is quite peculiar. So I think it is not a specific issue with Canonical: if you look at figures, indeed the main part of contributors are not sponsored by the big actors. This is a fact, not a personal attack.

LPC: Fitting into the kernel ecosystem

Posted Sep 23, 2008 22:49 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

There are plenty of free software contributors at Google (Ian Lance Taylor
and Guido van Rossum to name just two: a random grep of the changelogs of
any major plumbing component will find more although I suppose it's hard
to find many people who have the same sort of influence on anything that
Guido does on Python).

Of course if one considers GCC, the binutils and Python to be
insignificant or not plumbing enough, then maybe one can avoid this: but
any definition of 'plumbing' that excludes GCC and GNU ld in particular is
a silly definition.

LPC: Fitting into the kernel ecosystem

Posted Sep 23, 2008 22:54 UTC (Tue) by frankie (subscriber, #13593) [Link]

Of course, Greg was pointing the kernel at the time of that talk.

LPC: Fitting into the kernel ecosystem

Posted Sep 25, 2008 6:44 UTC (Thu) by dkite (guest, #4577) [Link] (2 responses)

If I'm going to spend money on a lts subscription from someone, and want it
to go towards supporting the ecosystem, this is an important data point.

As for it being a bit impolite, come on. Kernel people don't do polite. It
could be worse (or better). Al Viro could have said something.

I think it does the universe some good to knock Canonical and Ubuntu down a
peg or two. Restores some equilibrium.

Derek

LPC: Fitting into the kernel ecosystem

Posted Sep 25, 2008 12:20 UTC (Thu) by mdz@debian.org (guest, #14112) [Link] (1 responses)

Canonical does not, and will never, change for Ubuntu. There is no "subscription" version. It's free.

LPC: Fitting into the kernel ecosystem

Posted Oct 2, 2008 12:43 UTC (Thu) by mdz@debian.org (guest, #14112) [Link]

I meant "charge", of course, not "change"

LPC: Fitting into the kernel ecosystem

Posted Sep 25, 2008 15:12 UTC (Thu) by obi (guest, #5784) [Link]

I don't see why every distribution out there should invest its resources into the same areas.

- Redhat seems to invest a lot of effort into "Linux Plumbing" and as a non-Redhat user I'm very grateful for it.
- Ubuntu seems to invest a lot of effort into integrating, launchpad, translations, "community", evangelising (like with their "shipit" - cds sent for free), etc - and as a non-Ubuntu user I'm also grateful for all that.

There's clearly a market of users out there that only started considering Linux when Ubuntu appeared. Increasing userbase is very important. New Linux developers are a subset of the total Linux users after all; mindshare matters.

Linux Plumbing is important but not the only thing that needs work, just take the MacOS X example for instance.

Apple's iPhone is another example of a different focus - the hardware is more or less the same as all those Palm, WinCE, Pocket PC, Symbian devices. The plumbing didn't really make a difference. But the UI, integration, infrastructure (appstore) mattered a lot. Now everybody else learned their lesson, and we'll all get phones with better user interfaces (modulo some bad rip-offs too). What Apple did could in theory have been done by us Linux people or other proprietary companies years ago - but they went out and JFDI.

Ubuntu has a similar role IMHO - it tries to focus on something else; it could be an evolutionary dead-end, or it could pay off, in which case the other distros will also have learned something in the process. All good.

LPC: Fitting into the kernel ecosystem

Posted Oct 1, 2008 13:56 UTC (Wed) by pyellman (guest, #4997) [Link]

I wonder where Novell would rank if someone were to do an analysis of
organizations/companies contributions to rancor, discord and ill will
within the community?

Peter Yellman

LPC: Fitting into the kernel ecosystem

Posted Oct 8, 2008 23:23 UTC (Wed) by calc (guest, #22286) [Link] (3 responses)

I keep seeing the number of employees for Canonical being touted as 130 people. However, Canonical definitely has more than 130 employees in total but the number of people actually working on Ubuntu is only about 50.

Its fairly easy to find this information via launchpad.

LPC: Fitting into the kernel ecosystem

Posted Oct 8, 2008 23:26 UTC (Wed) by calc (guest, #22286) [Link] (1 responses)

If you want to know roughly how many employees Canonical has overall see:

https://launchpad.net/~canonical

LPC: Fitting into the kernel ecosystem

Posted Oct 9, 2008 4:30 UTC (Thu) by njs (subscriber, #40338) [Link]

"Membership: You are not allowed to view this team's membership."

So maybe not so easy as all that :-)

LPC: Fitting into the kernel ecosystem

Posted Oct 9, 2008 4:19 UTC (Thu) by calc (guest, #22286) [Link]

I see now where the 130 number came from it was on Dustin Kirkland's blog about Canonical employee numbers from 2007. So yes the company is growing very fast at least percentage-wise but is still quite small.


Copyright © 2008, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds