Kernel development

Kernel release status

The 3.13 merge window remains open; just over 9,900 non-merge changesets have been pulled into the mainline so far. See the article below for an update on what has been pulled for the 3.13 release.

Stable updates: 3.12.1, 3.11.9, 3.10.20, and 3.4.70 were released on November 20; each contains the usual set of important fixes.

Comments (none posted)

Quotes of the week

— Jon Masters

— Paul McKenney

— Thomas Gleixner

— Shobhit Kumar (thanks to Greg Kroah-Hartman)

Comments (3 posted)

The "Jailhouse" hypervisor

The Jailhouse project has announced its existence. Jailhouse is a Linux-native hypervisor like KVM, but with a focus on minimalism and isolation of virtual machines on dedicated CPUs. "Jailhouse is a partitioning hypervisor that can create asymmetric multiprocessing (AMP) setups on Linux-based systems. That means it runs bare-metal applications or non-Linux OSes aside a standard Linux kernel on one multicore hardware platform. Jailhouse ensures isolation between these 'cells', as we call them, via hardware-assisted virtualization. The typical workloads we expect to see in non-Linux cells are applications with highly demanding real-time, safety or security requirements." The project is in an early stage and looking for interested developers.

Comments (4 posted)

3.13 Merge window, part 2

The 3.13 merge window appears to be winding down, despite the fact that, as of this writing, it should have the better part of a week yet to run. There are now just over 9,900 non-merge changesets that have been pulled for 3.13; that is about 3,300 since last week's summary. Given the patch count and its slowing rate of increase, there is a good chance that Linus will close the merge window short of the full three weeks that had been expected this time around. It turns out that even diving trips on remote islands with bad Internet service can't slow the kernel process that much.

Some of the interesting user-visible changes pulled since last week's summary are:

• The multiqueue block layer patch set has been merged at last. This code will pave the way toward cleaner, higher-performing block drivers over time, though the conversion of drivers has not really begun in 3.13.

• The ARM big.LITTLE switcher code has been merged, providing basic support for heterogeneous ARM-based multiprocessor systems.

• The ARM "BE8" big-endian subarchitecture is now supported.

• The kernel has a new "power capping framework" allowing administrator control of peripherals which can implement maximum power consumption limits. Initially, support is limited to devices implementing Intel's "Running Average Power Limit" mechanism. See Documentation/power/powercap/powercap.txt for an overview of this subsystem and Documentation/ABI/testing/sysfs-class-powercap for details on the sysfs control interface.

• The new "tmon" tool can be used to monitor and tweak the kernel's thermal management subsystem.

• The split PMD locks patch set has been merged into the memory management subsystem. This code should result in significantly better performance in settings with a lot of transparent huge page use.

• The ability to wait when attempting to remove a module whose reference count has not yet dropped to zero has been disabled. This feature, accessible via rmmod --wait, has been deprecated for the last year.

• The size of huge pages on the SPARC64 architecture has changed from 4MB to 8MB. This change was necessary to enable this architecture to support up to 47 bits of physical address space. SPARC64 also supports the full tickless mode in 3.13.

• New hardware support includes:
  • Block: STEC, Inc. S1120 PCIe solid-state storage devices. Also note that the Compaq Smart Array driver has been disabled in this release; it will be removed altogether unless somebody complains.

  • Graphics: Marvell Armada 510 LCD controllers. Also: the radeon driver now supports dynamic power management by default on a range of newer chipsets.

  • I2C: Samsung Exynos5 high-speed I2C controllers, STMicroelectronics SSC I2C controllers, and Broadcom Kona I2C adapters.

  • Input: Microsoft Hyper-V synthetic keyboards, Neonode zForce infrared touchscreens, and LEETGION Hellion gaming mice.

  • Miscellaneous: ARM Versatile Express serial power controllers, Freescale i.MX6 PCIe controllers, Renesas R-Car Gen2 internal PCI controllers, TPO TD028TTEC1 LCD panels, ST Microelectronics STw481x power management chips, AMS AS3722 power management chips, and TI BQ24735 battery chargers.

  • Video4Linux: Conexant CX24117 dual DVB-S/S2 tuner modules, TI video processing engines, TI LM3560 dual-LED flash controllers, and ST Micro remote controls.

  • Watchdog: MOXA ART watchdog timers, Ralink SoC watchdog timers, and CSR SiRFprimaII and SiRFatlasVI watchdog timers.

Changes visible to kernel developers include:

• The new helper function:

      int dma_set_mask_and_coherent(struct device *dev, u64 mask);
  

  Will attempt to set both the streaming (non-coherent) and coherent DMA masks for the given device. Many drivers have been converted to this function, often with bugs fixed along the way.

• Most locking-related code has been moved into the new kernel/locking subdirectory.

• printk() and friends no longer implement the "%n" directive, which was seen as an invitation to security problems.

• The confusingly-named INIT_COMPLETION() macro, part of the completion mechanism, has been renamed to reinit_completion(). Its purpose has always been to reinitialize a completion that has already been used at least once; the new name should make that clearer.

• The new set_graph_notrace tracing filter allows the selective pruning of subtrees from graph trace output. See the commit changelog for an example of how this feature works.

Next week's LWN Kernel Page will contain an update with the final changes merged for the 3.13 kernel, which, most likely, will be released around the end of the year.

Comments (2 posted)

The past, present, and future of control groups

Much has been said about the problems surrounding control groups and the changes that will need to be made with this kernel subsystem. At the 2013 Korea Linux Forum, control group co-maintainer Tejun Heo provided a comprehensive overview of how we got into the current situation, what the problems are, and what is being done to fix them.

The idea behind control groups is relatively simple: divide processes into a hierarchy of groups and use those groups to provision resources in the system. The reality has turned out to be rather messier. So, Tejun asked: how did we get to this point? To begin with, he said, much of what is being done with control groups is new; all of it is new to Linux in particular, and some is new in general. So the community did not have any sort of model to follow when designing this new feature.

Beyond that, though, it is worth looking at who did the work. Control groups started as a new interface to the "cpuset" mechanism, which is used to partition the CPUs in a system among groups of processes. Few people, Tejun said, cared much about this feature, which is used mostly by the high-performance computing crowd. So few kernel developers paid much attention to what was being done.

Then control groups gained the memory controller, a mechanism for restricting the amount of memory used by each group. The core memory management developers did not really care about this work, so they did not participate in it and did not want to hear about it. The block controller came about the same way; Tejun does work in the block subsystem, but he had no real interest in the block controller and mostly just wanted it to stay out of his way. This environment led to a situation where controllers were written by developers without extensive experience in the subsystems they were working with; those controllers had to work on a non-interference basis so that the core developers could ignore them. As a result, controllers have been "bolted onto the side" of the existing kernel subsystems.

The result, Tejun said, is "not pretty." Even worse, though, is that the barriers between the controllers and the subsystems they work with inevitably broke down over time. So control groups are, as a whole, isolated and poorly integrated with the rest of the kernel, but they still manage to complicate the development of the rest of the kernel.

The developers who did all this work were good programmers, Tejun said, but they were not all that experienced with kernel development. So the code they produced was "kind of alien," not conforming to the usual coding style and practices. They repeated a lot of mistakes that the community has made and fixed in the past — repetition that could have been avoided with more review, but, he said, few people were paying active attention to the work being done in this area.

Mistakes were made

What kinds of mistakes were made? Start with hierarchy support — or the lack thereof — in a number of controllers. Control groups allow the organization of processes into a true hierarchy, with policies applied at various levels in the tree. But making a truly hierarchical controller is hard, so a number of controller developers simply didn't bother; instead, they ignored the tree structure and treated every group as if it were placed directly under the root. This was not a good decision, Tejun said; if a controller could not be made hierarchical, it should have at least refused to work with nested control groups. That would have indicated to users that things wouldn't work as they might expect and avoided the creation of a non-hierarchical interface that must now be supported.

The ".use_hierarchy" flag added by the memory controller to enable hierarchical behavior in subtrees was an especially confusing touch, he said.

Another clear mistake was the "release_agent" mechanism. The idea was to notify some process when a control group becomes empty; it was a good idea, he said, in that it allows that process to clean up groups that are no longer in use. But it was implemented as a user-mode helper — every time a control group becomes empty, the kernel creates a new process to run the release agent program. This is an expensive and complex operation for the simple task of sending a notification to user space. The rest of the kernel had moved away from this kind of process-based notification years ago, but the control group developers reimplemented it. We have much better notification mechanisms that should have been used instead, but nobody who could have pointed that out was paying attention when this code was merged.

Yet another problem is the heavy entanglement with the virtual filesystem (VFS) layer. Many years ago, the original sysfs implementation was also deeply tied to the VFS with the idea that it would simplify things. But that didn't work; the results were, instead, lots of memory used and locking-related problems. So sysfs was reworked to look a lot like a distributed filesystem, and things have worked better ever since. When the control group developers set out to create their administrative filesystem, though, they repeated the sysfs mistake. So now control groups have a number of related problems, such as locking hassles whenever an operation needs to work across multiple groups. Tejun is now working on separating things properly; some of that work was merged for the 3.13 kernel.

In engineering, Tejun said, nothing is free; everything comes down to a tradeoff between objectives. Or, in other words, "extremes suck," but control groups went to an extreme with regard to flexibility. Allowing multiple, independent hierarchies is the biggest example; this feature results in a situation where the kernel cannot tell which control group a given process belongs to. Instead, that membership is expressed by a list of arbitrary length. Controllers are all entirely separate from each other, with no coordination between them; they also behave in inconsistent ways. All this flexibility makes it difficult to make changes to the code, since there is no way to know what might break.

Flexibility also led to a range of implementation issues beyond the lack of hierarchical support in some controllers. The core code is complex and fragile. Developers took a lot of shortcuts in areas like security, leading to problems like denial-of-service issues. But, perhaps worst of all, the kernel community committed to a new ABI for control groups without really thinking about it; as a result, we ended up with a lot of accidental features. The ability to assign a process's threads to different control groups is one of those — most controllers only make sense at the process level. The control interface is filesystem-based, but no thought went into permissions, so it is possible to change the ownership of subdirectories, essentially delegating ownership of a subtree of groups to another user. The control group developers have, for all practical purposes, created a new set of system calls without the kind of review that system calls must normally go through.

What now?

The first step has been to fix the controllers that do not support the full control group hierarchy. Unfortunately, they cannot simply be fixed in place without breaking existing users. So there will have to be a "version 2" of the control interface that users can opt into. Controllers must be fully hierarchical or they will simply be unavailable in the new interface. The interface change will also allow the developers to enforce a certain degree of consistency between controllers. It will be possible, Tejun said, to mix use of the old and new interfaces without breaking things.

The multiple control group hierarchies will be going away. Most users will not really notice the change, but some were using multiple hierarchies to avoid enabling expensive controllers for processes that don't need them. In the new scheme, that need will be met by making it possible to enable or disable specific controllers at any level of the hierarchy. But all controllers will see the same process hierarchy; among other things, that will make it possible for them to cooperate more effectively. The resulting system will not be as flexible as multiple hierarchies are, but there seems to be an emerging consensus that it will suffice for the known use cases out there.

A lot of controllers will need updates to work in the new scheme, he said. There are a number of people working in the problem and the work is "70-80% there" at this point.

There will be, Tejun said, "no more faking things that we can't do properly." That is especially true when it comes to security which, he said, is a matter of noting and dealing with all of the relevant details — something that has not been done in the control group subsystem. In particular, the whole concept of delegating subtrees of the control group hierarchy to untrusted users is "broken"; there is no way to prevent denial-of-service attacks (or worse) under that scenario. To allow users to move to the new API without breaking things, it will still be possible to do this kind of delegation by changing the ownership of control group directories, but, he said, it will not be secure, just like it is not secure now.

A more secure approach might be the use of a trusted user-space agent process — something that is likely to be necessary in the future anyway. A number of these agents already exist: systemd is one, Google has its own, Ubuntu has one based on Google's code, and Android has an agent as well. In the Android case, Google actually has to "break the kernel" to make it work the way it wants. There is a need for some kind of common scheme so that processes can interoperate with any agent without having to know which one it is.

Tejun had hoped to have a prototype implementation of a reworked control group subsystem available by about now, but that has not happened. It may be ready by the end of the year, with, hopefully, the work being complete around the middle of 2014.

In summary, he repeated that control groups embody a lot of functionality that has not existed in Linux before. When he looks at the current code, he often gets angry at the mistakes that were made, but he is quite confident that he is making plenty of horrible mistakes of his own. So he fully expects future developers to be just as angry with him. That just goes with the territory. The important thing, he said, is to minimize the commitment that is made to user space; in that way, he hopes, we will not get locked into too many mistakes in the future.

[Your editor thanks the Linux Foundation for travel assistance to attend the Korea Linux Forum.]

Comments (10 posted)

Device trees II: The harder parts

A devicetree describes the hardware in a system using a tree of nodes, with each node describing a single device. As we observed last week, there are often relationships between devices which do not fit with the model of a strict hierarchical tree. Devicetree can address these needs through a range of techniques best described as cross-tree linkages.

Cross-tree linkages

Two of the more messy things to deal with in board files are interrupts and GPIOs (General Purpose Input/Output pins). This is because there are several different interrupt controllers and several different GPIO controllers. Both interrupts and GPIOs are identified by simple numbers; keeping track of the allocation of those numbers can become clumsy.

In the GTA04, the OMAP3 SoC contains 16 banks of 32 GPIOs, which can reasonably be treated as a single block and will probably be numbered 0-191. The twl4030 has a further 18 GPIO lines (unused) which will presumably be 192-209. The tca6507 LED driver can be configured to treat any of the seven output lines as a GPIO and one of them is. So it is GPIO 210.

There are two approachs to tracking these numbers. One is to hard-code the numbers, or at least to use lots of #defines like:

    #define GPIO_WIFI_RESET (OMAP_MAX_GPIO_LINES + TWL4030_GPIO_MAX)

This approach is simple but can be fragile in the face of change. The other is to use callbacks.

When the "gpio-twl4030" driver registers its 18 GPIOs it will be assigned a range of numbers; it would be best to not assume what numbers they will be until they are assigned. To this end, the platform_data provided to gpio-twl4030 can include a function to be called when initialization is complete, as is used by board-omap3beagle.c in the beagle_twl_gpio_setup(). This function can then store the numbers where appropriate and register the platform devices which depend on those GPIOs.

This hand-coded, delayed initialization can get very messy and is consequently error prone. Devicetree (that bringer of joy) makes this much easier. When one device depends on the service of another, such as a GPIO, an interrupt, a regulator, a timer, etc., the target device is identified by a reference to the relevant node in the devicetree. Unfortunately there is not as much uniformity here as we might like.

To reference an interrupt, the controller node and the interrupt number within the set controlled by that node are given separately, so:

    interrupt-parent = <&intc>;
    interrupts = <76>;

means that the interrupt to attach to is number 76 of those controlled by the node called "intc".

If no "interrupt-parent" is present, the ancestors of the current node are searched until either "interrupt-parent" or "interrupt-controller" is found. In the latter case the node containing "interrupt-controller" is the target node.

If a node responds to interrupts from different controllers, that situation cannot be represented with this approach. For that reason there is work to provide a syntax like:

    interrupts-extended = <&intc 76>

so the parent and the offset are both specified for each interrupt.

Depending on the interrupt controller, it might be necessary to specify more than one number to identify an interrupt. The number of numbers needed is specified with the attribute "#interrupt-cells" in the node for the interrupt controller. The exact meaning of the numbers can only be discovered by examining the devicetree bindings documentation (or the code); often one number will contain flag bits describing whether the interrupt should be edge- or level-triggered and whether high or low (or both) levels are interesting.

To reference a GPIO a syntax similar to the proposed "interrupts-extended" is the standard, so:

    gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;

(where GPIO_ACTIVE_HIGH is defined somewhere in include/dt-bindings/) will sort out the required GPIO number.

Naturally, in each case the device which provides the interrupt or GPIO will need to be initialized before it can be found and used. It wasn't very many kernel versions ago that this was a real problem. However in the 3.4 kernel, drivers gained the ability for their initialization (or "probe") routine to return the error "EPROBE_DEFER" which would cause the initialization to be tried again later. So if a driver finds that a GPIO line is listed in the devicetree, but no driver has registered GPIOs for the target node yet, it can fail with EPROBE_DEFER and know it can try again later. This can even be used to remove the need for callbacks and delayed registration in board files, but it is really essential for devicetree, and happily it works quite well.

It is worth highlighting that the standard attribute name to identify the GPIO for a device is "gpios" in the plural because, of course, a device might require multiple GPIOs and the descriptors can simply be listed on the one line. Hunting through the sample devicetree files in arch/arm/boot/dts, one finds extremely few cases where multiple GPIOs are specified in one attribute. What seems to happen more often is that there are multiple different "xx-gpios" attributes. For example, an MMC card driver might expect a "cd-gpios" to identify the "Card Detect" line, and a "wp-gpios" to identify the "Write Protect" line. This approach has the benefit of being more explicit (and so less confusing) and of making it easy to indicate that a particular line is simply not present on some board.

While interrupts and GPIOs allow a list of targets with some implicit meaning, regulators don't. Every request for a regulator must include a supply name, so the battery charger declares its dependence on a regulator with:

    bci3v1-supply = <&vusb3v1>;

Every regulator request is of the form "xxx-supply".

As described above, interrupts can sometimes specify the type of trigger, and GPIOs can sometimes specify the active level. Regulators, instead, have no extra parameters that can be passed, even though it would sometimes be useful to specify the required voltage — many regulators are programmable and the GTA04 WiFi chip requires 3.15 volts, which isn't the default (and which cannot yet be set at all using devicetree).

One final cross-tree linkage is implicit in the "reg" attribute mentioned in part 1. As with the "interrupts" attribute, the device that provides the registers is implicit, though, unlike "interrupts", it cannot even be made explicit with something like "reg-parent". Rather the device that provides the registers is always exactly the parent of the device which uses the registers.

We've already observed that a hierarchical tree often cannot accurately reflect reality. Were we to create a "reg-extended" attribute following the pattern of "interrupts-extended" we may well be able to discard the hierarchy altogether and replace the "device tree" with a "device list" where each device contains references to the other devices that it depends on to provide registers, interrupts, GPIOs, etc. This is already happening to some extent. Many so-called "platform devices" are described by devicetree nodes which appear at the top level of the tree rather than where they fit in a device hierarchy.

A simple example is the "aux-keys" device node for the GTA04.

    aux-keys {
        compatible = "gpio-keys";
    
        aux-button {
            label = "aux";
            linux,code = <169>;
            gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;
            gpio-key,wakeup;
            pinctrl-names = "default";
            pinctrl-0 = <&aux_pins>;
        };
    };

The GTA04 has two physical buttons, one of which is referred to as the "AUX" button and is connected to a GPIO input on the OMAP3. This node describes that part of the hardware. As you can see it identifies a particular GPIO, notes the key-code that it should generate in Linux, asserts that the key could wake the device from suspend, and gives some "pinctrl" information which assures that the particular pad on the OMAP3 is configured as a GPIO input.

Given what we have already learned about the tree structure of devicetree you might expect this node to appear as a child of a node describing the particular GPIO, which in turn would be a child of the GPIO controller within the OMAP3. However that is not the case. Instead, this "aux-keys" node appears at the top level, immediately under "/". While this seems a little odd in the case of a single button, it would make more sense if you imagined a device with multiple related buttons, such as volume-up and volume-down. If they were wired to two separate GPIOs, then placing the node as a child of both GPIOs is impossible and as a child of either would be untidy. Having two separate nodes (one per key) would obscure the fact that there is a single conceptual device: the "volume control".

So we find that some devices, such as the accelerometer and other sensors on the I2C bus, appear in devicetree at the end of a path reflecting how the CPU would address the device, some devices such as a GPIO-based keypad exist at the top level and refer to the components that they combine, and still other devices, such as the GSM modem in the GTA04, cannot be represented as a single device at all.

Not all fricasseed frogs and eel pie.

While exploring and enabling devicetree for the GTA04 has been a lot of fun, there have been some less exciting discoveries.

Firstly, the fact that devicetree support is still quite incomplete is a mixed blessing. On the one hand it is very easy to add devicetree support to many devices and this results in a positive feeling of achievement. On the other, there are fairly significant elements of functionality that are far from trivial. These, such as the omap-dss display driver and the cpu-freq support for OMAP, can largely be worked-around by hacking in some old-style "board-file" style initialization, but that isn't nearly so rewarding.

Secondly the devicetree compiler "dtc" which converts .dts source files to .dtb binaries is fairly primitive. If you do something wrong you'll mostly get either:

    Error: /home/git/gta04-mainline/arch/arm/boot/dts/omap3-gta04.dts:407.12-13 syntax error
    FATAL ERROR: Unable to parse input tree

or silent success as ARM maintainer Russell King recently observed (there are a couple of other error messages, but not many).

The compiler will often succeed for files which will make no sense to the kernel because there is no checking for the validity of attribute names or value ranges. The kernel does have fairly good schema documentation in "Documentation/devicetree/bindings", but this is not machine readable and dtc couldn't read it even if it were. Fortunately there is hope on the horizon. Tomasz Figa recently posted a proposed mechanism for writing machine-readable devicetree schemata which would allow more checking to be added to dtc. A proposal is certainly a long way from working code, but this is still an encouraging step.

What does the future hold?

Devicetree has already had clear benefits, such as the fun this author had in learning something new and the various cleanups that it has motivated in the driver support code in Linux. However it has also required a substantial amount of effort and that effort is ongoing. Such effort needs more justification than some fun and cleanup.

The significant benefit that devicetree promises is for operating system vendors and their clients. Currently, a Linux distribution can create a release for the x86_64 architecture and it can be expected to run on every x86_64 machine in existence. This is because there is just one platform to target. This is not the case for ARM. For ARM, there are many platforms. However if we can expect every ARM device to come with a devicetree description, then the Linux distributors could target "ARM + devicetree", and that is a credible platform concept.

If we could get to the point where a device plus a devicetree file could be reasonably expected to run with every subsequent release of Linux, then that would be a happy place to be. I would be able to upgrade Debian or openSUSE on my device with confidence, even though no-one else in the world has tested the combination.

My own personal experience suggests that might be overly optimistic. I've been regularly updating my GTA04 to the latest kernel, updating the board file as necessary, and I have always had regressions. Sometimes minor (the display has a green tinge), sometimes major (the battery won't charge). Every patch that broke my device was tested by its developer, often on several devices. But none had quite the same set of components as mine and so nobody noticed until I did.

A consequence of the lack of a standard platform is that there are lots of different components available which different designers interconnect in lots of different ways resulting in an impossibly large test matrix. Based on my (limited) experience, I have very little confidence that a kernel that nobody has tested on my device will actually work on my device. And so the promise that devicetree offers seems particularly hollow to me. Of course it is entirely possible that my recent experience is not the norm for others nor for the future. We can but hope.

Comments (15 posted)