Kernel development

Kernel release status

The 4.5 kernel was released on March 13 after a bit of indecision on Linus's part: "So this is later on a Sunday than my usual schedule, because I just couldn't make up my mind whether I should do another rc8 or not, and kept just waffling about it. In the end, I obviously decided not to, but it could have gone either way."

The 4.6 merge window is open; see the separate article below for a summary of what has found its way into the mainline so far.

Stable updates: 4.4.5, 3.14.64, and 3.10.100 were released on March 9; 4.4.6, 3.14.65, and 3.10.101 followed on March 16.

It's also worth noting that 2.6.32.71 was released on March 12. This kernel has been maintained ever since its release in December 2009, but Willy Tarreau has indicated that this is the last release. "In theory there is no reason to deploy this release unless you're still in the process of validating a new kernel. Kernel 3.2 is an excellent upgrade which still has more than two extra years to live"

Comments (none posted)

Quotes of the week

— Ingo Molnar

— also Ingo Molnar

— Dave Airlie

— Linus Torvalds

Comments (7 posted)

A (nearly) mainline kernel running on the Nexus 7

As was discussed at the 2015 Kernel Summit, there are essentially no commercial Android devices running mainline kernels. At the recently concluded Linaro Connect event, though, John Stultz demonstrated a Nexus 7 tablet running mainline with just a few patches. It even has accelerated graphics via the freedreno driver. "This is really great, because we now have a very-close to mainline test bed on a actual consumer device. So we can make sure upstream doesn't introduce any regressions (just recently, two ABI breaks that affected android were recently caught) and allows us to make sure when we push Android functionality upstream, that any interface changes required by maintainers can be properly tested to make sure what lands upstream really works."

Comments (11 posted)

The 4.6 merge window opens

As of this writing, the 4.6 merge window has gotten off to a relatively slow start, with a mere 0x3ff (1023) non-merge changesets pulled into the mainline so far. Linus may be proceeding slowly because the changes that have been merged are concentrated on tricky, low-level code, such as the x86 system-call machinery, CPU hotplugging, and the floating-point unit management code. It may well make sense to let any problems in this area make themselves known before further muddying the water.

That said, we have already seen some interesting new features merged, including:

• The code implementing shared futexes in anonymous memory has seen some serious optimization, eliminating a bottleneck that made these futexes significantly slower than the private variety.

• The perf subsystem has seen the usual set of improvements; see the pull request for details.

• Address-space layout randomization, which aims to thwart exploits by making things harder to find in a process's address space, has been fully extended to 32-bit programs running on 64-bit systems. In particular, the locations for libraries, the vDSO virtual system call area, and areas allocated with mmap() are now randomized.

• The post-init read-only memory patches have been merged, hopefully hardening the kernel against certain data-overwrite exploits.

• The new irqaffinity= command-line option allows an administrator to limit the CPUs to which interrupts will be directed; this feature is expected to be useful for CPU-isolation users who want to ensure that no interrupts will be delivered to the isolated processors.

• New hardware support includes:
  • Miscellaneous: Broadcom BCM6345 interrupt controllers, Alpine MSIX interrupt controllers, Analog Devices AXI SPI Engine controllers, Active-Semi ACT8945A voltage regulators, Hisilicon HI655X PMIC regulators, Maxim 77620/MAX20024 voltage regulators, Linear Technology LTC2990 I2C system monitors, ZyXEL NSA320 fan-speed and temperature sensors, and ISSI IS31FL32XX I2C LED controllers.

  • Pin control: ST Microelectronics STM32F429 pin controllers, Microchip PIC32 pin controllers, Mediatek MT2701 pin controllers, Qualcomm IPQ4019 pin controllers, and Mediatek MT7623 pin controllers.

  • Realtime clocks: Epson RX-6110 realtime clocks, Alphascale asm9260 realtime clocks, and Microchip PIC32 realtime clocks.

Changes visible to kernel developers include:

• The project to rework the CPU hotplug mechanism stalled for a couple of years, but it is now back, partly as a result of the Linux Foundation's funding of the realtime project. This work replaces a tangled set of notifiers with a more straightforward state machine that, one hopes, is easier to follow and make work correctly. The initial state machine has been merged for 4.6; see this pull request text for a description of what has been merged so far.

• The low-level resource-management code understands a new resource type: IORESOURCE_SYSRAM, meant to indicate system RAM. This makes it easier for kernel code to distinguish system memory from other resource types without having to use strcmp() on the resource name.

• The new function memcpy_mcsafe() will copy a block of memory; unlike memcpy(), it will not crash the system if the copy causes a machine-check error. It is intended for use with persistent memory, where the possibility of memory errors is higher and the system should recover when they happen.

• Simple wait queues have been merged, making life easier for code that just needs to wait without the fancy features that ordinary wait queues have accumulated over the years.

• The new function get_device_system_crosststamp() allows the simultaneous acquisition of timestamps from the core system clock and a peripheral device. Its purpose is to enable tight synchronization of events across a system; see this commit changelog for more information. In 4.6, the e1000e network driver will use this capability.

This merge window is likely to remain open until March 27, with the most likely date for the 4.6 release being May 15.

Comments (none posted)

Thread-level control with resource groups

The kernel's control-group mechanism allows processes to be divided into groups for the purposes of tracking and resource control. Both the API and underlying implementation of this mechanism have been going through considerable change in recent years. As part of that change, the newer control-group API has lost the ability to separately manage threads within a process, a loss that is not welcome in some quarters. Current work to replace that functionality is not finding an entirely warm reception either, though.

No threads need apply

Within the kernel, the distinction between a "thread" and a "process" is not entirely clear; for the most part, everything that can run is just a thread. At the user level, though, the two are seen as separate, with processes having their own address space while threads share a single address space (to oversimplify things slightly). Within the kernel, the "everything is a thread" notion led to the first version of the control-group interface (the "v1 API") managing everything at the thread level.

When control-group maintainer Tejun Heo designed the new (v2) control-group interface, one of the key changes he made is that the individual threads that make up a process cannot be assigned to different groups; the entire process is assigned (or moved) as a unit. This restriction comes about for a couple of reasons:

• Many of the resources managed by control groups belong to the process as a whole; think about memory usage in a setting where all threads share the same address space, for example. Putting different threads into different control groups makes it impossible to say what the memory-usage policy actually is. Bringing in multiple controllers with overlapping responsibilities — memory and I/O bandwidth, for example — muddies the picture even further. By keeping threads together (and in a single group hierarchy as well, another change made by the v2 API) the new mechanism makes a coherent resource-management policy possible.

• Control groups are seen to be a mechanism by which the system administrator can manage resources, and the API is designed around that use case. Resource management for threads, though, is more of a concern for an application itself rather than the administrator. Using the same interface for both is seen as conflating two use cases, creating possible security issues, and leading to possible performance problems — an API designed for occasional use by administrators may not perform well if an application uses it to make frequent changes.

  Additionally, in the v1 API, giving a process access to a portion of the virtual filesystem tree used to manage control groups requires the intervention of an outside process, possibly leading to a situation where applications have to be written for any management scheme that distributions may adopt. Tejun sees that as a cost imposed by the kernel's failure to provide a proper interface for this type of control.

Limiting control groups to processes steps around all of these problems without, it was thought, creating any new ones. But there are indeed use cases for managing resources at the thread level. Most of those use cases seem to be oriented around scheduling; an application may well want to use control groups to manage the division of its available CPU time among its component threads. In the v2 API, that ability has been lost.

Enter resource groups

Tejun has been pondering the thread-level control problem for a while now. In early January he posted a lengthy writeup of what he intended to do, but got no responses. That is often the case in kernel development; developers would rather see the code than a lot of words about what somebody intends to do. Besides, many were probably still recovering from their new-year hangovers on January 5. So the real discussion had to wait for the posting of Tejun's "resource group" mechanism patch set in March.

A "resource group" can be thought of as a special kind of control group designed for use within a process to control its own threads. They are thus distinct from "system groups," which are control groups as implemented in current kernels. Tejun uses the terms "rgroup" and "sgroup" for those two types of groups, but that leads to language like "a top-level rgroup of a process is a rgroup whose parent cgroup is a sgroup," so it might be better to spell things out.

Unlike control groups, resource groups are essentially invisible; they cannot be managed with the regular control-group API. That is done, in part, to separate the management interfaces, but it is also done because there is quite a bit less management to be done with resource groups. A process may be moved from one control group (system group) to another at any time, but resource-group membership for threads is forever. Among other things, that addresses the performance issues that can come from frequent group changes: if such changes cannot be done, the performance problems go away.

Resource-group membership is managed, instead, at process-creation time via a new flag (CLONE_NEWRGRP) to the clone() system call. A thread thus can never change its own resource-group membership, but it can create new threads in a different group. Thus resource groups, unlike system groups, can only cover specific subtrees of the process tree.

Controllers ordinarily operate on system groups only, but they can be made available at the resource-group level as well. Tejun's patch set does that for only one controller: the CPU (scheduling) controller. This controller is enabled on a specific resource group by calling setpriority() with the new PRIO_RGRP flag. For the top-level resource groups within a process, the priority can only be set as high as the priority of the process itself. Lower-level resource groups can have any priority, since they only affect the relative scheduling within the group itself.

Scheduling of resource groups works the same way it does with system groups. If a given system group contains two processes and a resource group with a few processes of its own, the CPU time given to the system group will, by default, be divided in three and split equally among those two processes and that one resource group. All of the threads within the resource group will then contend for that one-third of the available CPU time allotted to the group.

Another area of interest for thread-level control is the "cpuset" mechanism that allows threads to be restricted to specific CPUs. Tejun has chosen not to address this problem quite yet; as he said in January: "cpuset can also benefit from thread granularity; however, the situation around cpuset is murkier, so let's stay away from it for now." A dive into the murk will eventually become necessary, but there are enough thorny issues to deal with even without addressing the cpuset problem.

On the existence of processes

Like many other aspects of the control-group refactoring, the resource-group patches are going to have to overcome some resistance before getting into the mainline kernel. Mike Galbraith led the resistance in this case by complaining that the scheduler has no notion of processes. Everything there is just a sched_entity structure that can represent either a thread or a (system) control group. According to Mike, pushing the "process" concept into the scheduler is not a good idea. He is particularly concerned about the inability to move threads between resource groups, citing the thread-pool use case where threads do work for different users (and thus should run in different groups) over their lifetime. Ingo Molnar took the criticism further, accusing Tejun of ignoring concerns that have been expressed in the past and pushing forward with a problematic design.

Tejun responded that the resource-group patch set is his attempt to address the concerns that have been raised, most of which had to do with the loss of thread-level management. The scheduler may not recognize processes as such, but users do. So, he said, a process is a good place to separate system-level administration from user-level control:

Beyond that, he said, for some resources it makes no sense to go below the process level.

For the thread-group use case, he said, the best solution remains to put the CPU controller on a separate system-group hierarchy. That remains possible with the v1 API, and that capability is not going away anytime soon. But, he noted, doing so sacrifices the ability to manage all resources in a unified manner. In another response he described this organization as "completely alien to how the rest of the system is organized"; such modes can be useful, and they will remain supported, but there will be costs to using them.

Part of Mike's disagreement appears to be a desire to not allow processes to control resource management on his systems at all. As he put it: "That's what happens when control freak meets control freak, one of them ends up in pieces. There can be only one, and that one is me, the administrator." That view clearly runs contrary to what other users would like, though; there are constituencies out there for process-level resource-management control.

That is where the conversation stands as of this writing, but it seems certain that the discussion is not yet done. There is a fundamental disagreement here over how control groups should work, especially with regard to how they interact with the CPU scheduler. Tejun remains confident, though, that his design meets the requirements of those who are making real use of control groups now, and who need thread-level scheduling. Thus far, there have not been any well-developed alternatives proposed.

Comments (1 posted)