LCE: Linux, hardware vendors, and enterprise distributors

Enterprise distributions are an important part of the economic success story of Linux. The creation of highly stable, highly supported distributions has brought significant revenue streams to some distributors and enabled the deployment of Linux into many "mission critical" situations. Enterprise distributions encourage the commercial world to take Linux seriously. At LinuxConf Europe, however, your editor has stumbled into a few conversations which characterized enterprise distributions as one of the bigger problems the development community has now. Then a talk by Dirk Hohndel made that point again in a different context.

Dirk's talk was on how to get hardware vendors to support Linux. He knows what he is talking about: as the Linux CTO at Intel, Dirk is charged with, among other things, implementing Intel's commitment to provide free drivers for all of its hardware. His core point is that hardware vendors understand money better than anything else; getting them to support Linux will require showing them that it is in their economic interest to do so. To that end, he praised how Dell has taken care to put together hardware which is entirely supportable with free drivers to ship with Ubuntu pre-loaded. That sort of decision will quickly get the attention of the relevant vendors.

There were some suggestions on what to tell hardware vendors who are thinking about adding open source support for their products. Development in the open is crucial; drivers should be released early and made available for the community to work with. Intel did this with some of its early network drivers; the resulting level of interest and community participation exceeded all expectations. Vendors need to understand that they cannot design software just for their device, that they need to think bigger. This is a hard message for vendors to hear, but, in the long run, they benefit from a better kernel which will be better suited to their needs in the future.

It is important that software support be available immediately when the hardware is made available. If there is no driver for several months after the hardware release, competitors will have had time to get their answering products to market before Linux users can use the original product. That sort of time lag is forever in the hardware world. Vendors also need to continue to maintain their code after it gets into the mainline; there is nobody else who can ensure that it continues to work on all versions of the hardware.

One thing that the community could do to help would be to improve the tone of the discussion on our mailing lists. That tone is often quite hostile; it does not create a friendly environment for engineers working for hardware vendors who want to engage with the community.

There is another place where life gets difficult for hardware vendors, though; this is where the enterprise distributors come in. When Intel releases a driver for a new product, that driver goes into the mainline kernel. But the release cycle implemented by the enterprise distributors will not pick up that driver for as much as two years after it gets into the mainline. So enterprise customers are not able to make use of that hardware for a long time after its release, even though the driver is available.

Intel has competitors which will never release free drivers for their hardware. But they do put out closed-source modules for the enterprise distributions. So their customers are able to use that hardware from the outset.

In other words, Intel is being punished for playing by the rules and releasing their drivers to the community. This is exactly the wrong sort of incentive to create for hardware vendors. If they conclude that they will do better by just shipping binary-only modules, that is the course they will take.

Dirk's complaint echoes other conversations your editor has heard in the last few days. The development community has been very insistent in its message that code should be merged upstream, and that this merge should happen early. In the kernel area, the development cycle has been shortened to the point that changes find their way into a stable release after a maximum of a few months. But the enterprise distributions, by freezing kernels for years at a time, are pushing us back to the old, multi-year development cycle and sending a very different message to vendors.

The discussion of enterprise distributor policies is not new; see this article from last June for a previous installment. But this discussion appears to be reaching a new level of urgency, with some developers calling enterprise distributions one of the biggest problems the community is facing today. There is a fundamental conflict between the fast-moving development community and the sort of stasis that the enterprise distributions try to create. This conflict becomes especially acute when customers want the best of both worlds: no changes combined with fast-moving development and support for current hardware.

There are no easy solutions in sight. The enterprise distributions may be forcing a model from the proprietary software world on Linux, but there are reasons for the creation of that model in the first place. The kernel development community has gotten quite good at integrating vast numbers of changes while still producing a stable result, but any software which has recently seen significant changes will occasionally produce unwelcome surprises when dropped into a production environment. Slowing the rate of development is not an option, and it should be noted that the enterprise distributors are at the top of the list of companies which are setting the pace. Getting around this problem is going to be a challenge - but this community is good at facing challenges.