Kernel development

Brief items

Kernel release status

The current development kernel is 3.11-rc1, released on July 14. "Ignoring the lustre merge, I think this really was a somewhat calmer merge window. We had a few trees with problems, and we have an on-going debate about stable patches that was triggered largely thanks to this merge window, so now we'll have something to discuss for the kernel summit. But on the whole, I suspect we might be starting to see the traditional summer slump (Australia notwithstanding)." This release, alas, also has a new codename: "Linux for Workgroups."

Stable updates: 3.10.1, 3.9.10, 3.4.53, and 3.0.86 were all released on July 13. Greg warns that 3.9.10 may be the final release in the 3.9.x series.

Comments (none posted)

Quotes of the week

— Ingo Molnar

— James Bottomley

Comments (none posted)

Gettys: Traditional AQM is not enough

Here's a lengthy posting from Jim Gettys on the current state of the fight against bufferbloat and what needs to be done now. "Many have understood bufferbloat to be a problem that primarily occurs when a saturating 'elephant flow' is present on a link; it is easiest to test for bufferbloat this way, but this is not the only problem we face. The dominant application, the World Wide Web, is anti-social to any other application on the Internet, and it’s collateral damage is severe. Solving the latency problem, therefore, requires a two prong attack."

Comments (28 posted)

Kernel development news

The 3.11 merge window closes

Linus announced the release of 3.11-rc1 — and the closing of the 3.11 merge window — on July 14. While the merge window was open, 9,494 non-merge changesets were pulled into the mainline kernel repository. The last of those changes changed the kernel's codename to "Linux for Workgroups" and modified the boot-time logo; the new version appears to the right. Clearly, Linux development has moved into a new era.

Of those 9,494 changes, 1,219 were pulled since last week's summary. User-visible changes in that final batch of patches include:

• The new O_TMPFILE ABI has changed slightly in response to concerns expressed by Linus. In short, open() ignores unknown flags, so software using O_TMPFILE on older kernels has no way of knowing that it is not, in fact, getting the expected temporary file semantics. Following a suggestion from Rasmus Villemoes, Al Viro changed the user-space view of O_TMPFILE to include the O_DIRECTORY and O_RDWR bits — a combination that always results in an error on previous kernels. So applications should always get an error if they attempt to use O_TMPFILE on a kernel that does not support that option.

• The zswap compressed swap cache has been merged into the mainline. The changes to make the memory allocation layer modular, called for at this year's Storage, Filesystem, and Memory Management Summit, appear not to have been made, though.

• The "blk-throttle" I/O bandwidth controller now properly supports control group hierarchies — but only if the non-default "sane_behavior" flag is set.

• The "dm-switch" device mapper target maps I/O requests to a set of underlying devices. It is intended for situations where the mapping is more complicated than can be expressed with a simple target like "stripe"; see Documentation/device-mapper/switch.txt for more information.

• New hardware support includes:
  • Systems and processors: ARM System I/O memory management units (hopefully pointing to an era where ARM processors ship with a standard IOMMU) and Broadcom BCM3368 Cable Modem SoCs.

  • InfiniBand: Mellanox Connect-IB PCI Express host channel adapters.

  • Miscellaneous: Intel's "Rapid Start Technology" suspend-to-disk mechanism and Intel x86 package thermal sensors (see Documentation/thermal/x86_pkg_temperature_thermal for more information).

  • Video4Linux: OKI Semiconductor ML86V7667 video decoders, Texas Instruments THS8200 video encoders, and Fushicai USBTV007-based video capture devices.

  • Watchdog: Broadcom BCM2835 hardware watchdogs and MEN A21 VME CPU carrier board watchdog timers.

  • Staging graduations: TI OMAP thermal management subsystems.

Changes visible to kernel developers include:

• Module loading behavior has been changed slightly in that the load will no longer fail in the presence of unknown module parameters. Instead, such parameters will be ignored after the issuing of a log message. This change allows system configurations to continue working after a module parameter is removed or when an older kernel is booted.

• The MIPS architecture now supports building with -fstack-protector buffer overflow detection.

Recent development cycles have lasted for about 70 days (though 3.10, at 63 days, was significantly shorter). If that pattern holds for this cycle, the 3.11 kernel can be expected around September 9.

Comments (44 posted)

Some stable tree grumbles

In the dim and distant past (March 2005), the kernel developers were having a wide-ranging discussion about various perceived problems with the kernel development process, one of which was the inability to get fixes for stable kernel releases out to users. Linus suggested that a separate tree for fixes could be maintained if a suitable "sucker" could be found to manage it, but, he predicted, said sucker would "go crazy in a couple of weeks" and quit. As it turns out, Linus had not counted on just how stubborn Greg Kroah-Hartman can be; Greg (along with Chris Wright at the time) stepped forward and volunteered to maintain this tree, starting with the release of 2.6.11.1. Greg has continued to maintain the stable trees ever since. Recently, though, he has expressed some frustrations about how the process is working.

In particular, the announcement of the review stage for the 3.10.1 release included a strongly-worded complaint about how subsystem maintainers are managing patches for the stable tree. He called out two behaviors that he would like to see changed:

• Some patches are being marked for stable releases that clearly do not belong there. Cosmetic changes to debug messages were called out as an example of this type of problem.

• More importantly: a lot of the patches marked as being for the stable tree go into the mainline during the merge window. In many cases, that means that the subsystem maintainer held onto the patches for some time — months, perhaps — rather than pushing them to Linus for a later -rc release. If the patches are important enough to go into the stable tree, Greg asked, why are they not going to Linus immediately?

Starting with the second complaint above, the explanation appears to be relatively straightforward: getting Greg to accept changes for the stable tree is rather easier than getting Linus to accept them outside of the merge window. In theory, the rules for inclusion into the stable tree are the same as for getting patches into the mainline late in the cycle: the patches in question must fix some sort of "critical" problem. In practice, Linus and Greg are at least perceived to interpret the rules differently. So developers, perhaps unwilling to risk provoking an outburst from Linus, will simply hold fixes until the next merge window comes around. As James Bottomley put it:

Greg's plan for improving things involves watching linux-next starting around the -rc4 mainline release. If patches marked for the stable series start appearing in linux-next, he'll ask the maintainers why those patches have not yet found their way to Linus. Some of those patches may well find themselves refused entry into the stable tree if they only show up in the mainline during the merge window.

The topic of fully inappropriate patches, while the lesser part of Greg's complaint, became the larger part of the discussion. There are, it seems, any number of reasons for patches to be directed at the stable tree even if they are not stable material. At one extreme, Ben Herrenschmidt's description of how the need to get code into enterprise kernels drives the development process is well worth reading. For most other cases, though, the causes are probably more straightforward.

For years, people worried that important fixes were being overlooked and not getting into the stable updates; that led to pressure on developers to mark appropriate patches for the stable tree. This campaign has been quite successful, to the point that now, often, developers add a stable tag to a patch that fixes a bug as a matter of reflex. Subsystem maintainers are supposed to review such tags as part of their review of the patch as a whole, but that review may not always happen — or those maintainers may agree that a patch should go into the stable tree, even if it doesn't adhere to the rules. And sometimes subsystem maintainers can't remove the tag even if they want to. All this led James to propose doing away with the stable tag altogether:

James (along with others) proposes that putting a patch into the stable tree should require an explicit action on the subsystem maintainer's part. But Greg dislikes this idea, noting that maintainers are already far too busy. The whole point of the stable tree process is to make things as easy for everybody else as possible; adding work for maintainers would endanger the success of the whole exercise. That is especially true, he said, because some developers might encounter resistance from their employers:

Another proponent of explicit maintainer involvement is Jiri Kosina, who, in his work with SUSE's kernels, has encountered a few problems with stable kernels. While the stable tree is highly valuable to him, some of the patches in it cause regressions, some are just useless, and, for some, there is no real indication of why the patches are in the stable tree in the first place. Forcing maintainers to explicitly nominate and justify patches for the stable tree would, he said, address all three types of problem.

The first type — patches that introduce bugs of their own — will probably never be eliminated entirely; that is just how software development works. Everybody in the discussion has acknowledged that, once a buggy fix is identified, Greg quickly makes a stable release with that patch removed, so regressions tend not to stay around for long. Useless patches include those that are backported to kernels that predate the original bug; this problem could be addressed by placing more information in the changelog describing when the bug was introduced. The final type of problem raised by Jiri — mysterious patches — turned out to be security fixes. Jiri (and others) would like security fixes marked as such in the changelog, but that is unlikely to happen; instead, more effort is being made to notify distributors of security fixes via private channels.

In other words, while changes are likely to be made, they will not be fundamental in nature. Greg is likely to become fussier about the patches he accepts for the stable tree. Chances are, though, that he will never be as hard to please as Linus in this regard. In the end, the consumers of the stable tree — distributors and users both — want fixes to be included there. The stable kernel series is one of the biggest successes of the kernel development process; any changes to how they are created are likely to be relatively small and subtle. For most of us, the fixes will continue to flow as usual.

Comments (5 posted)

On kernel mailing list behavior

As has been widely reported, the topic of conduct on kernel-related mailing lists has, itself, been the topic of a heated discussion on the linux-kernel mailing list. While numerous development communities have established codes of conduct over the years, the kernel has never followed suit. Might that situation be about to change? Your editor will attempt a factual description of the discussion, followed by some analysis.

What was said

The setting was an extensive discussion on policies for the management of the stable kernel series and, in particular, the selection of patches for stable updates. It was an interesting discussion in its own right (which will be covered here separately), and it was generally polite. Even so, there came a point where Sarah Sharp couldn't take it anymore:

For the record, she was responding to this note from Linus:

Ingo's contribution was:

Whether these messages constitute "advocating for physical intimidation and violence" or even "advocating for verbal abuse" will be left for the reader to decide. But Sarah's point was clearly not that these specific messages were out of line; she is concerned with the environment on the linux-kernel mailing list in general. She has since taken the discussion to other forums (with more examples) and, in general, seems intent on changing the nature of the community's discourse.

Needless to say, responses on the list were mixed, though they were generally polite and restrained. A number of people, Linus included, pointed out that the number of personal attacks on the list is actually quite small, and that Linus tends to reserve his strongest language for high-level maintainers who (1) are able to take it, and (2) "should know better" than to do whatever it was that set Linus off. Opinions differ on whether that is a good thing. Jens Axboe said:

On the other hand, Neil Brown echoed the feelings of a number of participants who worry that the tone of the discussion tends to discourage people from joining the community: "He is scolding people senior developers in front of newcomers. That is not likely to encourage people to want to become senior developers." Being flamed can be hard on the recipient, but it can also affect the community by deterring other developers from participating.

For his part, Linus has made it clear that he feels little need to change his tone on the list:

Sarah responded that one can be clear without being abusive; she also suggested that Linus use his power directly (by threatening not to pull patches from the offending maintainer) rather than using strong words. For what it's worth, Linus did acknowledge, later in the discussion, that one of his more famous rants was "Not my proudest moment."

Unsurprisingly, there were few concrete outcomes from the discussion (which is still in progress as of this writing). Sarah has called for the creation of a document (written by "a trusted third party") describing acceptable conduct in the kernel community. There will almost certainly be a Kernel Summit discussion on this topic; as Linus pointed out, this kind of process-oriented discussion is the reason why the Kernel Summit exists in the first place.

Some analysis

There are, it seems, some simple statements that should not be overly controversial in the context of a discussion like this. Most people prefer an environment where people are pleasant to one another to an environment where people are harsh or abusive. An abusive community can certainly deter some potential contributors from joining; consider, for example, whether OpenBSD might have more developers if its communications were more congenial. Various development communities have set out to improve the quality of their communications, sometimes with clear success.

How do these thoughts apply in the kernel context?

It is worth pointing out that this is not the first time people have expressed concerns about how the kernel community works; it was, for example, a topic of discussion at the 2007 Kernel Summit. Numerous developers have pushed for improvements in how kernel people communicate; these efforts have happened both publicly and in private. Even Linus has said, at times, that he wished the discussion on linux-kernel were more constructive.

Your editor will assert that, in fact, the situation has improved considerably over the years. Much of that improvement is certainly due to the above-mentioned efforts. Abusive personalities have been confronted, managers have occasionally been contacted, trolls have been ignored, and more. The improvement is also certainly a result of changes in the kernel development community. We are as a whole older (and thus more restrained); the community is also much more widely paid to do its work, with the result that image-conscious companies have an incentive to step in when their developers go overboard. The tone is far more "professional," and true personal attacks are rare (though examples can certainly be found if one looks).

Over the years, the kernel development community has continued to grow. One might argue that it would have grown much more rapidly with a different culture in its mailing lists, but that is hard to verify. It is true, though, that much of that growth has come from parts of the world where people are said to be especially sensitive to direct criticism. For all its troubles, the kernel community is still sufficiently approachable that over 3,000 people per year are able to get their work reviewed and merged.

That said, the kernel is still viewed as one of the harshest communities in the free software world. It seems fairly clear that the tone of the discussion could bear some improvement, and that the current state of affairs repels some people who could otherwise be valuable contributors. So efforts like Sarah's to make things better should be welcomed; they deserve full consideration on the part of the community's leaders. But this kind of effort will be working against some constraints that make this kind of social engineering harder.

One of them is that the kernel absolutely depends on the community's unwillingness to accept substandard code. The kernel has to work in a huge variety of settings for an unbelievable number of use cases. It must integrate the work of thousands of developers and grow rapidly while staying maintainable over the long term. It is a rare software project indeed that has attained the size of the kernel and sustained its rate of change without collapsing under its own weight. If we want to still have a viable kernel a decade from now, we must pay close attention to the code that we merge now.

So it must be possible for developers to speak out against code that they see as being unsuitable for merging into the kernel. And the sad fact is that, sometimes, this message must be conveyed forcefully. Some developers are either unwilling to listen or they fail to receive the full message; as Rusty Russell put it:

The size of the community, the fact that some developers are unwilling to toss aside code they have put a lot of time into, and pressure from employers can all lead to a refusal to hear the message and, as a consequence, the need to be explicit. Any attempt to make it harder for developers to express their thoughts on the code could damage the community and, more to the point, is almost certain to fail.

That said, Rusty concluded the above message with this advice: "But be gentle with people. You've already called their baby ugly." There are certainly times when the community could be gentler with people without compromising on their code. That, of course, is exactly what people like Sarah are asking for.

Whether a documented code of conduct would push things in that direction is hard to say, though. Simply obtaining a consensus on the contents of such a document is likely to be a difficult process, though the discussion itself could be helpful in its ability to produce counterexamples. But, even if such a document were to be created, it would run a real risk of languishing under Documentation/ unheeded. Communities that have tried to establish codes of conduct have also typically included enforcement mechanisms in the mix. Groups like Fedora's "hall monitors" or Gentoo's "proctors" typically have the ability to ban users from lists and IRC channels when abuses are seen. Mozilla's community participation guidelines describe a number of escalation mechanisms. It is not at all clear that the kernel is amenable to any such enforcement mechanism, and, indeed, Sarah does not call for one; instead, she suggests:

It is far from clear, though, that a document calling for any sort of substantive change would acquire signatures from a critical mass of kernel developers, or that developers who are unwilling to sign the document would be willing (or able) to avoid dealings with those who have.

So proponents of more polite discourse on linux-kernel are almost certainly left with tools like calling out undesirable behavior and leading by example — precisely the methods that have been applied thus far. Those methods have proved to be frustratingly slow at best, but, helped by the overall changes in the development community, they have proved effective. It was probably about time for another campaign for more civility to push the community subtly in the right direction. Previous efforts have managed to make things better without wrecking the community's ability to function efficiently; indeed, we have only gotten better at kernel development over time. With luck and some support from the community, we should see similar results this time.

Comments (244 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.11-rc1 . (July 15, 2013)

• Greg KH: Linux 3.10.1 . (July 15, 2013)

• Greg KH: Linux 3.9.10 . (July 15, 2013)

• Sebastian Andrzej Siewior: 3.8.13-rt14 . (July 11, 2013)

• Greg KH: Linux 3.4.53 . (July 15, 2013)

• Greg KH: Linux 3.0.86 . (July 15, 2013)

Core kernel code

• Mel Gorman: Basic scheduler support for automatic NUMA balancing V4 . (July 11, 2013)

• Waiman Long: qrwlock: Introducing a queue read/write lock implementation . (July 15, 2013)

• David Howells: [RFC][PATCH 00/10] Associative array & Massive expansion of keyring capacity . (July 17, 2013)

Development tools

• David Ahern: perf: kvm live mode . (July 11, 2013)

• Dan Carpenter: smatch v1.59 released . (July 15, 2013)

• Jiri Olsa: perf tool: Add support for multiple data file storage . (July 17, 2013)

Device drivers

• Richard Genoud: Sound support for at91sam9x5-wm8731 based boards . (July 11, 2013)

• Rafael J. Wysocki: ACPI / hotplug / PCI: Major rework + Thunderbolt workarounds . (July 12, 2013)

• Sergei Shtylyov: misc: add driver for Renesas R-Car Gyro-ADC/speed-pulse interfaces . (July 15, 2013)

• Darren Hart: MinnowBoard Support V2 (Serial and Ethernet) . (July 15, 2013)

• Jonas Jensen: rtc: Add MOXA ART RTC driver . (July 17, 2013)

• Laurent Pinchart: Renesas VSP1 driver . (July 17, 2013)

Documentation

• Michael Kerrisk (man-pages): man-pages-3.52 is released . (July 15, 2013)

• Paul Gortmaker: Any value in having a netdev FAQ? . (July 17, 2013)

Filesystems and block I/O

• Li Zhong: partial revoke implementation for procfs . (July 11, 2013)

• David Herrmann: Anonymous Inode Allocations . (July 11, 2013)

• Edward Shishkin: Reiser4 for Linux-3.10 . (July 17, 2013)

Memory management

• Robin Holt: Transparent on-demand struct page initialization embedded in the buddy allocator . (July 12, 2013)

• Kirill A. Shutemov: Transparent huge page cache: phase 0, prep work . (July 15, 2013)

• Davidlohr Bueso: mm/hugetlb: per-vma instantiation mutexes . (July 15, 2013)

• Andrea Arcangeli: adding compaction to zone_reclaim_mode > 0 #2 . (July 17, 2013)

Architecture-specific

• Mark Rutland: arm64: initial CPU_HOTPLUG support . (July 15, 2013)

Miscellaneous

• Luis R. Rodriguez: backports-v3.10 released - first release! . (July 15, 2013)

• Stephen Hemminger: iproute2 3.10 . (July 16, 2013)

• Rob Landley: initmpfs v2: use tmpfs instead of ramfs for rootfs . (July 17, 2013)

Page editor: Jonathan Corbet
Next page: Distributions>>