Kernel development

Kernel release status

The current development kernel is 4.9-rc8, released on December 4. Quoth Linus: "So if anybody has been following the git tree, it should come as no surprise that I ended up doing an rc8 after all: things haven't been bad, but it also hasn't been the complete quiet that would have made me go 'no point in doing another week'."

The December 4 regressions report shows eleven known problems in the 4.9 kernel.

Stable updates: 4.8.12 and 4.4.36 were released on December 2. The 4.8.13 and 4.4.37 updates are in the review process as of this writing; they can be expected on or after December 9.

Those looking for a fix for the easily exploitable race condition known as CVE-2016-8655 will have to wait a bit longer. The fix is in 4.9, but has not yet found its way into the stable update stream.

Comments (none posted)

Quotes of the week

— Thomas Graf

— Linus Torvalds predicts the future

Comments (1 posted)

LSF/MM 2017: Call for Proposals

The 2017 Linux Storage, Filesystem, and Memory-Management Summit will be held March 20-21 in Cambridge, MA, colocated with the Linux Foundation's "Vault" event. The call for agenda items has gone out; interested attendees are encouraged to post their ideas for discussion. The deadline for submissions is January 15.

Full Story (comments: none)

Debating the value of XDP

All parts of the kernel are shaped by the need for performance and scalability, but some subsystems feel those pressures more than others. The networking subsystem, which can be asked to handle steady-state workloads involving millions of packets per second, has had to look at numerous technologies in its search for improved performance. One result of that work has been the "express data path" (or XDP) mechanism. Now, however, XDP is seeing some pushback from developers who see it as "pointless," a possible long-term maintenance problem, and a distraction from the work that networking developers need to be doing.

The core idea behind XDP is optimizing cases where relatively simple decisions can be made about incoming packets. It allows the loading of a BPF program into the kernel; that program gets an opportunity to inspect packets before they enter the networking stack itself. The initial use case for XDP was to enable the quick dropping of unwanted packets, but it has since expanded to cover simple routing decisions and packet modifications; see this in-progress documentation for some more information on how it works.

The core benefit of XDP is that the system can make quick decisions about packets without the need to involve the rest of the networking code. Performance could possibly be further improved in some settings by loading XDP programs directly into the network interface, perhaps after a translation step.

Thus far, most of the public discussion about XDP has been focused on the details of its implementation rather than on whether XDP is a good idea in the first place. That came to an end at the beginning of December, though, when Florian Westphal, in a posting written with help from Hannes Frederic Sowa, let it be known that he disagrees: "Lots of XDP related patches started to appear on netdev. I'd prefer if it would stop..." He would rather that developers turned away from a "well meaning but pointless" approach toward something that, in his view, is better suited to the problems faced by the networking subsystem.

That something, in short, is any of the mechanisms out there (such as the data plane development kit) that allow the networking stack to be bypassed by user-space code. These mechanisms can indeed yield improved performance in settings where a strictly defined set of functionality is needed and the benefits that come from a general-purpose network stack can be done without. Additionally, he said, some problems are best solved by utilizing the packet-filtering features implemented in the hardware.

XDP, Westphal said, is an inferior solution because it provides a poorer programming environment. Networking code done in user space can be written in any of a range of languages, has full debugging support available, and so on. BPF programs, instead, are harder to develop and much more limited in their potential functionality. Looking at a number of use cases for XDP, including routing, load balancing, and early packet filtering, he claims that there are better solutions for each.

Thomas Graf responded that he has a fundamental problem with user-space networking: as soon as a packet leaves the kernel, anything can happen to it and it is no longer possible to make security decisions about it. User-space code could be compromised and there is no way for the kernel to know. BPF code in the kernel, instead, should be more difficult to compromise since its freedom of action is much more restricted. He also said that load balancing often needs to be done within applications as well as across machines, and he would not want to see that done in user space.

Sowa, instead, questioned the early-drop use case, asking whether the focus was on additional types of protection or improved performance. Like Westphal, he suggested that this problem could be solved primarily with hardware-based packet dropping. The answer from Tom Herbert made it clear that he sees both flexibility and performance as being needed:

Networking maintainer David Miller also said that he sees XDP as being a good solution for packet dropping use cases. Hardware-based filters, he said, are not up to the job, and XDP looks to him to be the right approach.

Sowa's other concern was not so easily addressed, though. As XDP programs gain functionality, they will need access to increasingly sophisticated information from the rest of the networking stack. That information can be provided by way of functions callable from BPF, but those functions will likely become part of the kernel's user-space ABI. That, in turn, will limit the changes that the networking developers can make in the future. These concerns mirror the worries about tracepoints that have limited their use in parts of the kernel. Nobody in the discussion addressed the ABI problem; in the end, it will have to be handled like any other user-space interface, where new features are, hopefully, added with care and a lot of review.

In the end, the discussion probably changed few minds about the value of XDP — or the lack thereof. Stephen Hemminger probably summarized things best when he said that there is room for a number of different approaches. XDP is better for "high speed packet mangling", while user-space approaches are going to be better for the implementation of large chunks of networking infrastructure. The networking world is complex, and getting more so; a variety of approaches will be needed to solve all the problems that the kernel is facing in this area.

Comments (3 posted)

Development statistics for 4.9

The 4.9 development cycle, expected to come to a close on December 11, will be by far the busiest cycle in the history of the kernel project. A look at the repository activity for this release, though, shows that it is, in many ways, just another typical cycle for the kernel community with one large exceptional factor.

As of this writing, 16,150 non-merge changesets have been pulled into the mainline repository for 4.9. That far exceeds the 13,722 changesets pulled for 3.15 which was, until now, the busiest cycle ever. There is no doubt that the merging of the Greybus driver had a lot to do with that but, even if one subtracts the 2,388 changesets associated with Greybus, 4.9 would still (just barely) set a new record. Clearly there has been a lot going on in the last few months.

The changes in 4.9 were contributed by a total of 1,719 developers — also a new record. The most active of those developers were:

Most active 4.9 developers

By changesets Johan Hovold 599 3.7% Viresh Kumar 421 2.6% Greg Kroah-Hartman 376 2.3% Alex Elder 345 2.1% Chris Wilson 335 2.1% Mauro Carvalho Chehab 238 1.5% Kuninori Morimoto 204 1.3% Wolfram Sang 183 1.1% Wei Yongjun 175 1.1% David Howells 149 0.9% Arnd Bergmann 135 0.8% Bryan O'Donoghue 134 0.8% Ivan Safonov 134 0.8% Sergio Paracuellos 120 0.7% Bjorn Helgaas 117 0.7% Javier Martinez Canillas 108 0.7% Alex Deucher 107 0.7% Markus Elfring 106 0.7% Colin Ian King 105 0.7% Linus Walleij 95 0.6%

By changed lines Jes Sorensen 66930 7.0% Rex Zhu 58288 6.1% Laurent Pinchart 33602 3.5% Greg Kroah-Hartman 32002 3.3% Mauro Carvalho Chehab 14747 1.5% Chris Wilson 14661 1.5% Ken Wang 14556 1.5% Viresh Kumar 13389 1.4% Dom Cobley 12816 1.3% Christoph Hellwig 12459 1.3% Darrick J. Wong 11646 1.2% Hans Verkuil 11480 1.2% Johan Hovold 11465 1.2% Ram Amrani 11318 1.2% Alex Elder 11060 1.2% David Howells 11045 1.2% Netanel Belgazal 10858 1.1% Lijun Ou 10849 1.1% Huang Rui 10157 1.1% Maruthi Srinivas Bayyavarapu 10092 1.1%

The top four developers of the Greybus code, Johan Hovold, Viresh Kumar, Greg Kroah-Hartman, and Alex Elder, ended up as the top four changeset contributors overall. Between the four of them, they contributed 1,741 changesets in 4.9. The fifth developer in the by-changesets column, Chris Wilson, worked on the i915 graphics driver.

In the "lines changed" column, Jes Sorensen removed the rtl8723au network driver, which has been superseded by the rtl8xxxu driver. Rex Zhu worked on the AMD "PowerPlay" graphics driver, Laurent Pinchart worked on various media drivers (including the Greybus camera driver), Greg Kroah-Hartman added a lot of Greybus code, and Mauro Carvalho Chehab did a lot of work as the media subsystem maintainer and in support of the ongoing documentation transition to Sphinx.

228 companies (that we know about) supported work on the 4.9 kernel. The most active of those employers were:

Most active 4.9 employers

By changesets Linaro 1876 11.6% Intel 1869 11.6% (Unknown) 1293 8.0% (Consultant) 1055 6.5% (None) 924 5.7% Red Hat 916 5.7% Google 541 3.3% Samsung 535 3.3% AMD 476 2.9% IBM 401 2.5% Renesas Electronics 331 2.0% Huawei Technologies 274 1.7% Linux Foundation 262 1.6% Mellanox 237 1.5% SUSE 233 1.4% ARM 228 1.4% Oracle 207 1.3% Texas Instruments 167 1.0% BayLibre 146 0.9% Broadcom 136 0.8%

By lines changed AMD 105820 11.1% Red Hat 104492 10.9% Intel 89456 9.4% Linaro 75310 7.9% (Consultant) 61718 6.5% (Unknown) 45676 4.8% (None) 34464 3.6% Google 31054 3.2% Samsung 25438 2.7% Oracle 14425 1.5% Huawei Technologies 14220 1.5% IBM 13432 1.4% Raspberry PI 12816 1.3% QLogic 12439 1.3% Mellanox 12284 1.3% Cavium Networks 11741 1.2% NXP Semiconductors 11662 1.2% Cisco 11507 1.2% Linux Foundation 11182 1.2% Amazon 10858 1.1%

Linaro made it to the top of the "by changesets" column this time around by virtue of work on — wait for it — Greybus. Otherwise there is not a lot that is surprising in this table. As usual, quite a few companies work to support Linux, but the list of the most active of those tends not to change much from one cycle to the next.

Finally, it is worth noting that 301 developers made their first contribution to the kernel during the 4.9 development cycle. That, too, is arguably a record ("arguably" because 2.6.25 had 332 new developers but, since the tracking only begins with the Git era at 2.6.12, some of those "new" developers had certainly made contributions before). Let's look at first-time contributors since 3.0:

The plot shows a slight but observable upward slope, especially since the recent-period low of 183 during the 3.5 development cycle; the number of new contributors coming into the kernel community is growing over time. For all its faults, the kernel community is still able to attract new developers and, thus, continues to grow. That suggests that the pace of kernel development is unlikely to slow much in the near future.

Comments (1 posted)