Kernel development

Kernel release status

The current development kernel is 3.18-rc3, released on November 2. Linus complained that things aren't slowing down as he would like, but doesn't seem too worried: "That said, I don't think there is anything particularly horrible in here. Lots and lots of small stuff, with drivers accounting for the bulk of it (both in commits and in lines), but networking and core kernel showing up too. Nothing particularly stands out." With this prepatch, the codename for the release has changed to "Diseased Newt."

Stable updates: 3.17.2, 3.16.7, 3.14.23, and 3.10.59 were released on October 30.

Comments (none posted)

Vetter: Atomic Modeset Support for KMS Drivers

For those who are interested in the grungy details of getting the new atomic modesetting operations working with existing graphics drivers, Daniel Vetter has the scoop: "So I've just reposted my atomic modeset helper series, and since the main goal of all that work was to ensure a smooth and simple transition for existing drivers to the promised atomic land it's time to elaborate a bit. The big problem is that the existing helper libraries and callbacks to driver backends don't really fit the new semantics, so some shuffling was required to avoid long-term pain. So if you are a driver writer and just interested in the details then read for what needs to be done to support atomic modeset updates using these new helper libraries."

Comments (3 posted)

Linux 3.16.y.z extended stable support

The Ubuntu kernel team has announced that they will be providing extended support for the 3.16 kernel series. The team will pick up where Greg Kroah-Hartman left off, with 3.16.7, and will provide support until April 2016.

Full Story (comments: 12)

Supporting solid-state hybrid drives

In recent years we have seen the addition of a number of subsystems to the kernel that provide high-speed caching for data on (relatively) slow drives; examples include bcache and dm-cache. But there is nothing preventing drive manufacturers from building this kind of caching into their products directly. The result of such bundling is "solid-state hybrid drives" — rotating drives that have some flash storage built in as well. Properly used, that flash storage can speed accesses to frequently used data. But it turns out that getting to "properly used" is not quite as straightforward as one might think.

Of course, one can simply leave everything up to the drive itself. Left to its own devices (so to speak), the drive will observe which blocks are frequently accessed and work to keep those blocks in fast storage. But the operating system — or the programs running on that system — will often have a better idea of which data will be most useful in the future. If that information is communicated to the drives, the result should be better use of fast storage, and, thus, better performance.

Enabling that communication is the goal of this patch set posted by Jason Akers. The response to that patch set from the kernel community makes it clear, though, that there is still some work to be done to figure out the best way to get the best possible performance from such drives.

This patch set uses the per-process I/O priority value as a way of signaling information about cache usage. That priority can be set by way of the ionice command. Using a few bits of the priority field, the user can specify one of a number of policies (listed here in symbolic form):

• IOPRIO_ADV_EVICT says that the data involved in I/O operations should be actively removed from the cache, should it be found there. It's a way of saying that the data will, with certainty, not be used again in the near future.

• IOPRIO_ADV_DONTNEED says that the data should not be cached, but that there is no need to actively evict it from the cache if it's already there.

• IOPRIO_ADV_NORMAL leaves caching policy up to the drive, as if no advice had been provided at all.

• IOPRIO_ADV_WILLNEED indicates that the data will be needed again in the near future and, thus, should be stored in the cache.

This patch set is unlikely to be merged in anything close to its current form for a few reasons. One of those is that, as a few developers pointed out, associating I/O caching policy with a process is a bit strange. Any given process may want different caching policies for different files it works with; indeed, it may want different policies for different parts of the same file. Creating a single, per-process policy makes this kind of use nearly impossible.

Beyond that, as Dave Chinner pointed out, the process that generates an I/O operation in user space may not be the process that submits the I/O to the block subsystem. Many filesystems use worker threads to perform actual submission; that breaks the link with the process that originally created the I/O operation. Filesystems, too, may wish to adjust caching policy; giving metadata a higher priority for the cache than data is one obvious possibility. As it happens, there is a way for filesystems to adjust the I/O priority value on individual requests, but it is not the most elegant of APIs.

For these reasons, some developers have suggested that the caching policy should be set on a per-file basis with a system call like fadvise() rather than on a per-process basis. Even better, as Jens Axboe noted, would be to add a mechanism by which processes could provide hints on a per-operation basis. The approach used in the non-blocking buffered read proposal might be applicable for that type of use.

There is another problem with this patch set, though: the types of "advice" that can be provided is tied tightly to the specifics of how the current generation of hybrid drives operates. It offers low-level control over a single level of cache and not much else. Future drives may operate in different ways that do not correspond well to the above-described operations. Beyond that, hybrid drives are not the only place where this kind of advice can be provided; it can also be useful over NFS 4.2, with persistent memory devices, and with the upcoming T10/T13 "logical block markup descriptors." There is a strong desire to avoid merging a solution that works with one type of current technology, but that will lack relevance with other technologies.

Martin Petersen has put some time into trying to find an optimal way to provide advice to storage devices in general. His approach is to avoid specific instructions ("evict this data from the cache") in favor of a description why the I/O is being performed. He described his results as "a huge twisted mess of a table with ponies of various sizes", but it's not all that complicated in the end.

That table consists of a set of I/O classes, along with the performance implications of each class. There is a "transaction" class with stringent completion-time and latency requirements and a high likelihood that the data will be accessed again in the near future. The "streaming" class also wants fast command completion, but the chances of needing the data again soon are quite low. Other classes include "metadata" (which is like transactions but with a lower likelihood of needing the data again), "paging," "data," and "background" (which has low urgency and no need for caching).

Given an (unspecified) API that uses these I/O classes, the low-level driver code can map the class of any specific I/O operation onto the proper advice for the hardware. That mapping might be a bit trickier than one might imagine, though, as the hardware gets more complex. There is also the problem of consistency across devices; if drivers interpret the classes differently, the result could be visible performance differences that create unhappy users.

These issues will need to be worked out, though, if Linux systems are to drive hybrid devices in anything other than the default, device-managed mode. Given a suitable kernel and user-space API, the class-based approach looks like it should be flexible enough to get the most out of near-future hardware. Getting there, though, means a trip back to the drawing board for the authors of the current hybrid-drive patches.

Comments (4 posted)

open() flags: O_TMPFILE and O_BENEATH

Not much happens on a Linux system without one or more calls to open() or one of its variants. There is no other way to create a file or access a file that already exists. It thus follows that the various flags that control the behavior of open() have a significant effect on the functionality of the system as a whole. Here, we'll look at two specific open() flags; one of them is a relatively recent addition to the kernel, while the other is still in the proposal stage.

O_TMPFILE

The O_TMPFILE flag has been discussed a few times in these pages; the abrupt nature of its addition meant that it had little review and a fair number of post-merge problems. The concept behind this flag is simple enough: it requests the creation of a file with no associated directory entry. It is thus meant for temporary files that will not be opened by any other process.

Eric Rannaud recently asked a question: what should happen when a process makes a call like the following?

    int fd = open("/tmp", O_TMPFILE | O_RDWR, 0);

The flags request the creation of a writable temporary file, but the third argument (the file mode) says that there should be no access (read or write) allowed. As it happens, POSIX is clear enough about this situation when a file is created with ordinary O_CREAT: the provided mode only applies after the creation of the file. So, while a process can create a file that it cannot itself access in general, it can still get a working file descriptor in the act of creation itself.

As it happens, though, file creation with O_TMPFILE does not work that way; the file mode is applied from the beginning, so the open() call listed above will fail. This behavior was widely recognized to be a bug, and Eric's fix was merged for the 3.18-rc3 release. But there are a couple of interesting side notes that are worth looking at.

One is that this call:

    int fd = open("/tmp", O_TMPFILE | O_RDONLY, 0666);

will still fail. When the O_TMPFILE feature was implemented, it seemed that there was no use case for a temporary file that could not be written to, so this case (O_TMPFILE with O_RDONLY) was explicitly forbidden. But it turns out that there is a use case for this type of file: creating an empty file with a specific set of extended attributes atomically. The open() call would be followed by one or more fsetxattr() calls; once everything is in place, linkat() can be used to make the file visible in the filesystem. Linus initially agreed that this use case should be supported, but later changed his mind. So read-only O_TMPFILE files will remain unsupported.

Amusingly, the original bug was discovered while digging into a related glibc bug. It seems that, when O_TMPFILE is used, the mode argument isn't passed into the kernel at all. In the case of open() on x86-64 machines, things work out of sheer luck: the mode argument just happens to be sitting in the right register when glibc makes the call into the kernel. Things do not work as well with openat(), though, with the result that, in current glibc installations, O_TMPFILE cannot be used with openat() at all. The bug is well understood and should be fixed soon.

O_BENEATH

When a developer makes a call to openat(), they will normally expect that the file being opened or created will be located in the specified directory. As is often the case, though, surprises lurk for the unwary. Trouble can come from a surprising symbolic link or deliberately malicious input; either way, it can lead to files being created or opened where they should not be.

David Drysdale has a solution in the form of the O_BENEATH flag for openat(). If this flag is included in the call, the file being accessed must exist in or below the directory provided. The enforcement of this rule is simple enough: the provided path is constrained to not start with "/" or contain "../". Any symbolic links traversed while resolving the path must meet the same conditions.

This feature was implemented as part of the filesystem access restrictions found in the Capsicum patch set. It turns out that there are other potential users as well, though. In particular, when combined with a secure computing ("seccomp") filter, O_BENEATH can be used to safely give a sandboxed process a directory to create files in.

The initial review concerns raised against this patch have been addressed in the current version. It is a relatively simple and non-invasive patch, so there is a reasonable chance that we'll see it enter the mainline during a near-future merge window.

Comments (6 posted)

Kdbus meets linux-kernel

There has been a long history of attempts to put interprocess messaging systems into the Linux kernel; in general, these attempts have not gotten very far. From the beginning, though, the expectations around "kdbus," an in-kernel implementation of the widely used D-Bus mechanism, have been higher than the usual. Kdbus has been under development for more than two years, and was unveiled at linux.conf.au in January. But it had never been posted to the linux-kernel mailing list for review and, with luck, eventual inclusion — until October 29, when Greg Kroah-Hartman posted a twelve-part series for consideration.

In short, kdbus is a mechanism by which processes can find each other and exchange messages. It is meant to facilitate certain kinds of interprocess communications in a way that is both secure and reasonably fast. For those wanting details, this document covers kdbus functionality in a fairly thorough way.

A whirlwind overview

For those not wanting to read an 1800-line file, here's a brief summary. When kdbus starts up, it creates a set of device nodes under /dev/kdbus; any actions involving kdbus require opening one or more of those nodes. A "bus" is essentially a namespace within which processes can communicate with each other. A fairly normal default configuration involves a single "system" bus for communicating with privileged services, and one "user" bus for each logged-in user. The user bus would be used, for example, to allow the processes implementing the user's desktop environment to talk to each other.

While there is a single bus namespace at boot time, things need not remain that way. A set of buses exists within a kdbus "domain"; domains are organized into a hierarchy. So, for example, a container-management system would create a new domain for each container, then use a bind mount to make the appropriate subtree of /dev/kdbus available within the container. Thereafter, processes within the container can communicate with each other without having any access to communications outside of the container. There is currently no provision for using kdbus to communicate between containers.

Messages are, at their simplest, a set of bytes with no interpretation by the kernel at all. Messages can pass file descriptors between processes; the passing of sealed files and memfds is also supported. Message recipients can specify a set of sender credentials that must be supplied with a message for policy checking; those credentials are attached to the message by the kernel. There is also a built-in policy mechanism describing which processes can adopt "well-known names" and which processes can communicate with which others.

Kdbus is intended to be fast with both large and small messages. For the largest of messages, zero-copy transfer between processes is supported. Experience has shown, though, that a message must be about 512KB or larger before page-mapping tricks become cheaper than just copying the data. There is support for broadcast messages, along with a mechanism based on bloom filters for filtering out unwanted broadcasts without waking up the (uninterested) recipients.

In general, kdbus is meant to be a replacement for D-Bus that addresses the various issues that have come up with the latter over time. The goal is not to be the ultimate messaging system for all possible applications. While the kdbus developers are open to the idea of adding more functionality in the future, they are trying to keep a lid on the complexity at this stage.

Reviews

Given the (systemd-ish) origins of the kdbus code, one might well have expected the discussion to be somewhat hostile at times. In truth, while there have been concerns expressed, the discussion has remained mostly friendly and entirely technical. Developers are taking a deep look at the code and discussing how it can be improved; one cannot say that kdbus is not getting a fair hearing.

One of the initial questions was, inevitably, why does this functionality need to be in the kernel in the first place? The kernel already provides a number of interprocess communication primitives, and tools like D-Bus have successfully used them for many years. See this message from Greg for a detailed answer. In short, it comes down to performance (fewer context switches to send a message), security (the kernel can ensure that credentials passed with messages are correct), race-free operation, the ability to use buses in early boot, and more. There do seem to be legitimate reasons to want this kind of functionality built into the kernel.

Credentials

The handling of credentials drew a couple of different criticisms; the first was that credentials are checked when a message is sent — not when the connection to the bus is first created. Eric Biederman raised concerns that failure to capture credentials at open() time could lead to exploitable vulnerabilities. He did not actually point out any such vulnerabilities, though, and, in the past, such vulnerabilities have tended to be associated with later read() and write() calls. Since kdbus does not support either call on any of its file descriptors, that kind of vulnerability should not be an issue here. Still, there is some discomfort among the more security-oriented reviewers that the late capture of credentials is asking for trouble.

Another problem, raised by Andy Lutomirski, is that checking credentials at message-sending time makes privilege-separation architectures impossible:

If that privilege is checked every time a message is sent, the ability to drop privileges in this way is lost. Kdbus developer Daniel Mack responded that, in the D-Bus world (which carries over into the kdbus design), there is no concept of "opening a connection" to a service like journald. Instead, one connects to a bus and sends messages to services; each message has to stand on its own. As Daniel put it:

This particular disagreement reflects a fundamental difference in how developers see kdbus being used. It does not look like an easy one to resolve without some significant design changes on the kdbus side; any such changes would move it away from the D-Bus model and are likely to encounter resistance from the kdbus developers.

A related issue, also raised by Andy, is that the recipient of a message specifies which credential information should accompany that message. This information can include user and group IDs, process command line, control group information, capabilities, security labels, the audit session information, and more. The sender of a message has no control over whether this information is sent. Andy thinks that sending this information will lead to information leaks and security problems.

Instead, Andy said, the sending process should explicitly specify which credential information should accompany a message and that security-related requests should explicitly document what credentials are required. "Otherwise it becomes unclear what things convey privilege when, and that will lead immediately to incomprehensible security models, and that will lead to exploits." The response from kdbus developer Tom Gundersen is that "by simply connecting to the bus and sending a message to some service, you implicitly agree to passing some metadata along to the service". It allows the recipient to be sure that the necessary information will be supplied, even if the recipient's security model changes (requiring different information) in the future. Again, Andy disagrees, insisting that the provision of credentials should be a matter of negotiation between both sides.

Namespace and device issues

Both Eric and Andy also raised an entirely different set of concerns having to do with the way the domain namespace works. The decision to attach globally visible names to domains leads to some unfortunate consequences in their view. The first (and smaller) of these is that the existence of a namespace forces kdbus domains into a hierarchical structure, even though there is nothing that is actually hierarchical about them. Each domain is an independent entity with no particular relation to its parent domain outside of the naming scheme.

The real problem, though, is that a global namespace implies the need for some sort of control to keep malicious processes from polluting that namespace. That, in turn, means that creating a kdbus domain is a privileged operation. Quite a bit of work has gone into allowing unprivileged users to create containers. But if a new container cannot be given a kdbus domain without privilege, that model breaks down. Lennart Poettering acknowledged this concern in an apparently private email publicly responded to by Andy; he said that allowing unprivileged domain creation should be possible, as long as the checks for namespace collisions remain in place.

Andy's reply there was that none of the other container-oriented primitives have global names, and that there is a reason for that: avoidance of just this type of namespace collision possibility. Kdbus domains, he asserts, would be better off without the globally visible names. There would appear to be a couple of reasons why these names exist. One would be to make it easy for a privileged process to tap into any domain and watch traffic for debugging purposes. That particular need could probably be met by way of a domain pointer in each process's /proc area.

The bigger problem relates to another fundamental kdbus design decision: to base the whole thing around device nodes found in /dev. If there are kdbus devices for multiple domains in /dev, they must be organized into that directory's hierarchical namespace. Such a namespace is essentially unavoidable if the device nodes are to be available to (and, importantly, locatable by) processes in the system. For this reason, a couple of reviewers have said that the device abstraction is a mistake. Rather than implementing kdbus operations as a set of ioctl() calls on a device, perhaps kdbus should have a set of dedicated system calls that would eliminate the need for the device nodes altogether. That would also eliminate the need for a global kdbus domain namespace.

Eric expressed a related concern: the use of device nodes implies the existence of dynamically allocated device numbers. That will interfere with the checkpointing and restoring of containers, since there is no way to guarantee that the same device numbers will be available when the container is restored. That breaks a use case that works with D-Bus today, so Eric has described it as a regression.

Going forward

From one perspective, the response on the mailing list should be encouraging for the kdbus developers. While the obligatory "why do this in the kernel?" questions were asked, there does not appear to be much fundamental opposition to putting this kind of functionality into the kernel. That suggests that, sooner or later, the kernel will have an answer for users who have asked for a native messaging solution.

The form of that solution remains up in the air, though. Kdbus will clearly have to change to address the review comments that have been posted (and those yet to come); how radical that change needs to be remains to be seen. It could be that, as Alan Cox put it, "it would be far more constructive to treat the current kdbus as a proof of concept/prototype or even a draft requirements specification". Or perhaps the concerns that have been raised can be addressed with a simpler set of changes.

Either way, it does not look like the long-playing kdbus story will come to a close anytime soon. That may be frustrating for those who are waiting for this functionality to become available in a mainline kernel. But this process can only be hurried so much if the end result is to be a solution that will stand the test of time. Once kdbus goes into the kernel it will become much harder to change, so it is worth taking the time to get the interface (and its semantics) right first.

Comments (52 posted)