The plumbing layer as the new kernel

Last week's Unix wars article discussed the long-feared possibility of Linux fragmenting into a number of incompatible—and weaker—systems. Before leaving that subject behind entirely, it is worthwhile to take a look at an effort that is intended to be a unifying force in the Linux community: the growth of a well-defined and actively developed "plumbing layer." By wrapping the kernel in layers of higher-level software, the plumbers hope to create a new core that will function similarly across all distributions. Thus far, we have not seen clear evidence that all distributions want to play along, though.

The kernel has long been the unifying force across Linux distributions. Especially in recent years, as distributors have worked harder to stay close to the mainline, the kernel is one thing that could be counted on to be nearly the same on any system. The layers on top of the kernel can vary widely between distributions, though, with some distributors, such as Android, having replaced almost everything above the kernel with their own code. That creates differences between distributions that can make life harder for users, developers, and for the maintainers of the distributions themselves.

There have been attempts in the past to create a common distribution core at a higher level than the kernel. The Linux Standard Base is one such endeavor, but, despite years of effort and lots of resources poured into it, the LSB has never been hugely helpful. Its lowest-common-denominator focus left much of the system uncovered, and its stated intent of making life easier for distributors of binary software has failed to create excitement in the community. Another effort was the ill-fated United Linux project; it may well have failed in any case, but it was certainly doomed by having the SCO Group as one of its founding members. Since then, there have been few overt efforts to create a common distribution core with the arguable exception of Debian, which has always, to some extent, seen itself as being that core.

What we have seen in recent years has been an effort to bring together and organize developers working in the "plumbing layer" that wraps the kernel. This layer is not precisely defined; it includes the bootstrap and initialization systems, udev, communications mechanisms like D-Bus, and related utilities. Interestingly, the C library—the layer that intermediates most access to the kernel—has typically not been a part of that group; perhaps that situation will change as a result of the more community-oriented focus in the glibc project.

The Linux Plumbers effort was never explicitly envisioned as the creation of a new common distribution core; instead, it was just a way to help developers work together and make the system better. Recently, though, things have started to change; in particular, much of the work on systemd has been justified with the claim that it would work to reduce fragmentation between distributions. Systemd is tightly tied to the Linux kernel—it will not run anywhere else—and is meant to integrate many of the kernel's features into a tighter, well-functioning, and quickly evolving system. That system then becomes the core of a modern, fast-moving distribution.

A number of people have commented on this trend; see this recent remark from Martin Langhoff or this Google+ discussion, for example. There are some clear benefits from moving in this direction, but some challenges as well.

One challenge is ABI stability. Kernel developers try hard to ensure that they do not break applications as the kernel evolves. That policy is unlikely to change anytime soon, but the truth is that there are some in our community who would like to move the stability commitment further out, allowing the ABI between the kernel and the plumbing layer to change in incompatible ways. Such changes should not bother users as long as the kernel and the plumbing layer change at the same time, but they would be bothersome indeed if only one of those components changes, or for anybody who is not using the "standard" plumbing code. The ABI stability requirements have proved frustrating to plumbing-layer developers in the past; one can only expect that to happen again in the future.

The other problem is that there will be disagreements on what the larger core should look like; one need only read the many systemd discussions to get a sense for how deep those disagreements are. Assuming that we have achieved all we can through the reproduction of classic Unix systems, there are no precedents for what the Linux system of the future should look like. There are no POSIX standards to guide development. So developers are breaking new ground, and that will always lead to disagreement and conflict.

Add to this disagreement a certain degree of discomfort among developers who feel that they are losing influence over the direction things are going. Debian developer Marco d'Itri expressed it clearly:

There is no special effort to exclude anybody from the development of the plumbing layer. But the usual rule of free software development holds: those who are doing the work have the biggest say in the directions it takes. For better or for worse, many of the developers doing work at the plumbing level have concentrated into a relatively small number of companies. That leaves a number of distributions out in the cold.

It also leaves them with a choice: sign on to this new, unofficial core distribution, or continue to go it alone without. There are plenty of examples of distributions that have followed their own path for many years; see Slackware as the classic example. Gentoo is another distribution that has built much of its own core infrastructure, including its own init system that few people have heard of. It can be done, but there is a real cost in the form of duplicated effort and an inability to benefit from the work done by others.

Predicting how this situation will turn out is not easy. This is a time of rapid change in both the community and the wider industry; it's not clear what is going to work in the long run. But the vision of a more tightly coupled system that can enable Linux distributions to evolve more quickly has some appeal. If these developers can pull it off, they may help to ensure that a unified Linux plays a major role for years to come.