LWN.net Weekly Edition for November 14, 2019

Welcome to the LWN.net Weekly Edition for November 14, 2019

This edition contains the following feature content:

• Debian reconsiders init-system diversity: a general resolution (GR) may be in the offing to determine how much support the distribution should provide to init systems other than systemd.
• Analyzing kernel email: some researchers report on kernel email analysis at ELCE.
• Statistics from the 5.4 development cycle: our usual look at who developed the code and which companies paid for it.
• Emulated iopl(): an effort to remove some ancient cruft from the kernel.
• The 2019 Automated Testing Summit: a report from the gathering of kernel test system developers in Lyon, France at the end of October.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Debian reconsiders init-system diversity

Many community-based Linux distributions have made the decision to switch to systemd, and most of those decisions were accompanied by lengthy, sometimes acrimonious mailing-list discussions. No distribution had a harder time of it than Debian, though, where arguments raged through much of 2013 before the Debian Technical Committee decided on systemd in early 2014. Thereafter, it is fair to say, appetite for renewing the init-system discussion has been low. Now, though, the topic has returned to the fore and it would appear that the project is heading toward a new general resolution (GR) to decide at what level init systems other than systemd should be supported.

The new systemd discussion had been simmering for some time before flaring up in late 2018 when it became clear that the project had been unable to fully support systemd alternatives due to a lack of developer interest. The current discussion, instead, is a direct result of Debian project leader Sam Hartman's Bits from the DPL posting on October 29. Therein, he stated his belief that a new general resolution on init-system support is necessary, and that he would be proposing one in the near future.

The immediate motivation for a reconsideration would appear to be the proposed addition of elogind, a standalone fork of the systemd-logind daemon, to Debian. Elogind would provide support for systemd's D-Bus-based login mechanism — needed to support small projects like the GNOME desktop — without the need for systemd itself. The addition of elogind has been controversial; it is a difficult package to integrate for a number of reasons. Much of the discussion has evidently been carried out away from the mailing lists, but some context on the problem can be found in this bug report. In short: merging elogind appears to be complex enough that it would be hard to justify in the absence of a strong commitment to the support of non-systemd init systems. It seems possible that this commitment no longer exists across the distribution as a whole; the purpose of a general resolution would be to determine whether that is the case or not.

Feature pressure

Unsurprisingly, Debian developers have a variety of opinions on this issue. This response from Russ Allbery is worth reading in its entirety. He argues that the 2014 decision (of which he was a part) never really nailed down the project's position toward other init systems. That was a necessary compromise at the time, he said, but it is causing stress now: "while I feel somewhat vindicated by the fact that this didn't immediately fall apart and has sort of worked, I think it's becoming increasingly untenable". The time has come, he says, to make the project's position clear so that developers can and will do the right thing:

Josh Triplett zeroed in on one of the issues that is testing the init-system peace now. There is, he said, an increasingly long list of features that are only available with systemd, and application developers want to use those features:

The responses to this argument took a couple of different approaches. Ted Ts'o described those features as "the 'embrace, extend, and extinguish' phenomenon of systemd which caused so much fear and loathing". Debian has avoided much of this stress until now, he said, because it ships a relatively old version of systemd, but that won't last forever. Upstream developers are going to use newer systemd features, and Debian will have to accept that if it wants to package and ship their code. "I don't think we have a choice but acknowledge reality and accept that some packages may simply be incompatible with alternative init systems". The situation doesn't make him happy, he said, but there is little to be done about it.

Allbery, instead, suggested that the project could decide on a set of interesting systemd features, then create non-systemd implementations of those features where necessary. That is essentially what is being done with elogind, for example. All init systems could be made to parse systemd unit files, and there would be a minimal set of directives they would be expected to support. There would, in effect, be a definition of the ABI an init system must support, rather than any rules about specific systems. Done properly this approach could ease the feature pressure, but there is one little problem:

Some, like Ian Jackson, claimed that much of this functionality already exists and works. Svante Signell also argued that the systemd alternatives are well maintained and that Debian should remain open to contributions from developers who are working on those alternatives.

Others, including Triplett, said that not only do the developers of other init systems lack the desire to implement systemd features, but they argue that those features should not exist in the first place. That, Allbery added, is the question that the project has to answer: what will be Debian's policy toward software that requires features that will be specific to systemd? Should there be a specific subset of systemd features that Debian software will be allowed to depend on, with the idea that alternative init systems will eventually gain implementations of those features?

The first GR draft

Hartman posted a draft general resolution on November 7 that does not directly address Allbery's question. It focused strongly on init systems, rather than support for advanced systemd features; there are three alternatives:

• "Affirm init diversity": running init systems other than systemd is an important project goal, and provision of init scripts would remain mandatory.
• "Systemd but we support exploring alternatives": systemd would be the preferred init system, but alternatives would remain important; elogind is given as an example of the sort of project that should be supported. Hartman posted another message with a detailed description of what this option might imply.
• "Systemd without diversity as a priority": there would be no requirement to support anything but systemd in Debian.

Hartman thus appears to have left out Allbery's suggestion of defining an init-system ABI entirely. It should be noted, though, that this is explicitly a draft; it is likely to evolve considerably before it reaches the point where the project will vote on it.

There is one final bit of context to consider as part of this vote: according to Marco d'Itri, it has already happened. "We have already voted with apt: less than 1% of new installs use sysvinit". That suggests a relatively low level of interest in systemd-free Debian. Perhaps users with that interest have moved to Devuan but, to all outside appearances, progress there has slowed considerably since its developers fell out with each other in April. Even if Debian chooses to continue to support multiple init systems, it is not clear that the development effort to make that support work will be there.

The free-software community values diversity and choice in its systems. But we also value standardization and the progress that can be made when we all focus our efforts on a single system. Many parts of the Linux system are essentially monocultures: examples include the kernel, the C library, and, until relatively recently, the compiler. There are strong arguments in favor of both standardization and diversity. In the case of GCC, the lack of competition is often cited as a cause for a long period of relative stagnation. If one looks at desktop environments, instead, it is often said that the scattering of developer effort has prevented the success of the Linux desktop as a whole.

It would thus seem that there is no obvious answer to the question of init-system diversity, regardless of how one feels about systemd in particular. Most community distribution projects have long since decided that they lack the resources (and maybe the desire) to support multiple init systems. Most of those, especially the larger ones, have settled on systemd, even if they feel that a viable alternative to systemd would be good to have. Debian has, so far, been an exception to this trend. It will be interesting to see whether the project has the will and the development resources to continue on that course.

Comments (182 posted)

Analyzing kernel email

Digging into the email that provides the cornerstone of Linux kernel development is an endeavor that has become more popular over the last few years. There are some practical reasons for analyzing the kernel mailing lists and for correlating that information with the patches that actually reach the mainline, including tracking the path that patches take—or don't take. Three researchers reported on some efforts they have made on kernel email analysis at the 2019 Embedded Linux Conference Europe (ELCE), held in late October in Lyon, France.

The presentation (slides [PDF]) actually listed four speakers, though one could not make it to ELCE. The three present were Ralf Ramsauer, from the Technical University of Applied Sciences Regensburg, Sebastian Duda, of Friedrich–Alexander University Erlangen–Nürnberg, and Wolfgang Mauerer, of Siemens AG in Munich. Lukas Bulwahn, who is a hobbyist active in the Linux Foundation ELISA Project and employed at BMW AG, was unable to attend. In the introduction, Mauerer jokingly suggested that the goal of the research was to understand more "than the NSA already knows" about the behavior of kernel developers. Really, though, the presentation was meant partly as a request for comments; the researchers have been observing the kernel community for some time and have been pulling out pieces they find interesting, but they would be happy to hear other ideas on the kinds of analysis that would be useful to the community.

Development process

Ramsauer said that the goal of the research is "formalizing and assessing the Linux kernel development process". There are a number of motivations for doing so, both from inside and outside the community; his personal motivation is to write more papers to finish his PhD work, he said with a grin. There is interest from the safety-critical development world, automotive and industrial equipment makers, for example. The safety-certification bodies require documented development practices; since the development process of Linux is not something under the control of the equipment makers, the best they can do is to document the existing practice to those bodies. Beyond that, the techniques they are working on can be used to monitor the health of Linux and other projects, as is done by the CHAOSS project.

Tracing the development process has come up recently in the kernel community as well. The interest in tracking patches through various stages in their development was the underlying motivation for the proposal to add change IDs to patches. There were alternatives proposed, such as adding the Message-ID of the previous patch posting to an update, but that is not consistently done throughout the kernel. The community would like to find a solution to this problem that avoids the, perhaps only minor, inconvenience of maintaining those IDs.

The problem stems from the disconnect between activity on the mailing list and commits in the Git repositories. He went through the typical lifecycle of a patch set, starting with its development, which is done in private typically on some Git branch. Eventually the patch set surfaces on one or more kernel mailing lists, where it is reviewed. The review comments are reflected in the patches and they are posted again. That process is followed, iteratively, until the patches are acceptable and merged (or not).

Each posting of the patch set, along with all of the comments and discussion associated with it, can be identified by the Message-ID in the posting, which is new for each iteration. But the commit (or commits) that stem from the patch set are identified with a Git commit hash. There is potentially a many-to-many relationship between these message IDs and commit hashes, which leads to a need for a tool to extract that information from the mailing lists and Git repositories.

They have a tool, Patch Stack Analysis (PaStA), that can do that work. It originally was written to detect similar patches between different branches in order to quantify the upstreaming efforts of various out-of-tree projects (e.g. vendor kernels, realtime patches). It has now been extended to work with mailing lists.

He gave an example of two patches to show how they are processed to determine how similar they are. The patches are tokenized and then string distance measurements are used to generate a similarity score. If that score exceeds a threshold, the patches are considered likely to be related and a similarity graph is created, which gets matched up with patches in the repository. More information about the techniques used can be found in two papers (PDF 1, PDF 2) they have written, Ramsauer said.

The Git data to be used is easy to come by simply by cloning the repository of interest, but the mailing list data is not as easy. Up until a few years ago, they used the data from Gmane, but that has become unreliable after the demise of Gmane (mostly) in 2016. Lore.kernel.org is a reasonable substitute, with some data going back to 1996 and beyond, but it has a limited subset of mailing lists. It also has imported some of its data from Gmane, which improperly handled some headers, so it can be difficult to work with.

Due to that, they started collecting their own archive of around 200 kernel mailing lists in May. In doing so, they ran into the same problems that anyone dealing with a large pile of email encounters: email is full of all sorts of weirdness. There are broken encodings, bad dates, badly formatted Message-ID headers, and more. Once that all gets cleaned up, that data will be ready to use, he said.

Some results

Duda then stepped up to the podium to report on what had been found in their analysis. They looked at the roughly 610,000 commits to the mainline starting with the v2.6.39 tag along with around three-million emails from the public lists on lore.kernel.org; 34 mailing lists were used, starting in May 2011 and running until the end of 2018. Of the three-million emails, though, not all are patches; eliminating the non-patches left 1.15m patches. But not all patches posted to the lists are actually patches to the kernel; beyond that, there is a fair amount of traffic that comes from bots, pull requests, stable reviews, and the like. In the end, they winnowed it all down to around 800,000 emails with patches from humans that were meant to be applied to the current kernel at the time they were posted. That is what they ran through the analysis program.

They set out to see if they could identify unmaintained subsystems based on patches that get ignored on the mailing lists. As it turns out, they were not able to do so because the two do not correlate. In the process, though, they found other interesting things. First they defined an ignored patch as one where the posting thread has no responses other than by the author, where the patch itself does not get accepted upstream, and that any related (similar) patches for, say, other kernel versions were also ignored.

Based on that criteria, 2.5% of patches were ignored over the time period. Interestingly, the percentages seemed to drop over time: 2011 had 3.9%, 2015 had 2.1%, and 2018 had 1.6%. When graphing that, which can be seen in slide 18 or around 17:50 in the YouTube video of the talk, a large spike appears in the ignored patch line midway through 2016. That comes from a single 1300-patch posting that, perhaps unsurprisingly, got ignored. Since it throws off the rest of the graphs, they simply eliminated that outlier in subsequent graphs.

What the corrected graph shows is that there are roughly 30-50 patches that are ignored each week throughout the time frame. But the number of patches submitted each week rose over that same time frame, which leads to the percentage decline. At a talk given at the Linux Plumbers Conference (LPC), some maintainers asked for a way to find out how many patches they were ignoring. To try to help answer that question, Duda said that they have also run the tool on individual mailing lists.

He presented four graphs of the linux-arm-kernel, linux-mips, linux-wireless, and netdev mailing lists. One thing that stands out right away is that all show a steady number of ignored patches. That number is quite low in all four, ten or less per week, and it remains flat even when the overall number of patches posted to the mailing list grows.

It is interesting that over the time period, the number of patches to linux-kernel-arm went from around 150 per week to as many as 700 per week, but that the ignored patches remained below ten per week, Ramsauer said. That trend persists with most mailing lists, though there a few that look different, he said. They chose the linux-kernel-arm and linux-kernel-mips lists because both are architectures, but the Arm list has grown substantially over the years, while MIPS has stayed roughly the same. Likewise for linux-wireless and netdev, though linux-wireless has actually declined slightly over the years.

Another analysis that they did, Ramsauer said, is to try to see if the likelihood of being ignored is correlated to when in development cycle it is sent. As it turns out, being ignored is largely independent of when in the cycle a patch is sent. There is, however, a slightly higher chance of a patch being ignored during the merge window.

Off-list patches

During this work, they also discovered some "off-list" patches, which have been included into Linus Torvalds's Git tree, but were not posted to a public mailing list prior to that. They analyzed the stabilization phase of the 5.1 kernel, patches from v5.1-rc1 to v5.1, which was around 1800 commits. They mapped those commits back to the mailing list and found 60 that did not have a mailing-list thread identified. Some of those were errors in their tool, but a manual review showed that 24 patches were off-list patches.

Some of those were reverts, where doing the revert was discussed on the list, but the actual patch doing the revert was not posted. Some were also simply fixups of various sorts by the maintainers that were never discussed. There are some subsystems where the maintainers often have off-list patches, he said. In addition, of course, some of those patches will be security fixes that were discussed, but not publicly. For example, he pointed to a commit from Greg Kroah-Hartman that never appeared on a public list; when asked, Kroah-Hartman said that it had been discussed on the closed kernel security mailing list. Thus this kind of analysis might provide a way to find security fixes before they become publicly known, Ramsauer said.

Several audience members were interested to know which subsystems tended to have off-list patches, but the researchers did not want to point fingers. It is entirely possible for others to do the same analysis, however, as an attendee pointed out, which could lead to a lucrative, if possibly illegal, "business" of disclosing them in various ways. The clear takeaway is that off-list patches are likely to become more visible, rather than quietly lurking in the sea of other patches, before too long.

As Mauerer noted at the end of the talk, this kind of research is not well funded, either by industry or by normal academic research channels. Presenting results at a conference like ELCE is not deemed important in academic circles, while presenting it in a workshop to half-a-dozen people who have never submitted a single kernel patch is seen as valuable in that realm. He encouraged companies and other organizations to consider funding this kind of research in the future.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for travel assistance to attend Embedded Linux Conference Europe in Lyon, France.]

Comments (none posted)

Statistics from the 5.4 development cycle

As of this writing, just over 14,000 non-merge changesets have found their way into the mainline repository for the 5.4 release; that is a bit less than we saw for 5.3, but more than most of the other recent kernels. The final 5.4 release is approaching, so it must be time for our usual look at where the code merged in this development cycle came from. It's mostly business as usual in the kernel community, modulo an appearance from none other than Hulk Robot.

Those 14,000 changesets were contributed by 1,802 developers, which is just short of the 1,846 who contributed to 5.3; there is still time, though, for 5.4 to set a new record for the number of contributors — a surprising number of developers wait until the end of the release cycle to fix something. Of the developers seen so far, 266 made their first contribution to the kernel in this cycle. The combined work from these developers increased the size of the kernel by 393,000 lines.

The most active developers this time around were:

Most active 5.4 developers

By changesets Yue Haibing 270 1.9% Chris Wilson 238 1.7% Arnaldo Carvalho de Melo 182 1.3% Christoph Hellwig 165 1.2% Jiri Olsa 165 1.2% Nishka Dasgupta 150 1.1% Andy Shevchenko 145 1.0% Colin Ian King 140 1.0% Masahiro Yamada 121 0.9% Ard Biesheuvel 116 0.8% Kuninori Morimoto 103 0.7% Andrey Smirnov 99 0.7% Thomas Gleixner 98 0.7% Greg Kroah-Hartman 97 0.7% Linus Walleij 92 0.7% Xiaojie Yuan 91 0.6% Chuhong Yuan 89 0.6% Arnd Bergmann 89 0.6% Daniel Vetter 83 0.6% Maxime Ripard 80 0.6%

By changed lines Le Ma 92046 10.1% Bhawanpreet Lakha 89403 9.8% Christoph Hellwig 45376 5.0% Arnd Bergmann 35746 3.9% Ard Biesheuvel 17750 2.0% Chris Wilson 12697 1.4% Valdis Klētnieks 10941 1.2% Alex Vesker 10937 1.2% Shannon Nelson 9740 1.1% Eric Biggers 8120 0.9% Pawel Laszczak 7824 0.9% Neil Armstrong 7370 0.8% Jun Lei 7338 0.8% Jiri Olsa 7334 0.8% Yan-Hsuan Chuang 7087 0.8% Andrew Jeffery 6827 0.8% Paul Cercueil 6217 0.7% Leo Liu 6122 0.7% Larry Finger 6077 0.7% Mauro Carvalho Chehab 5945 0.7%

At the top of the "by changesets" column is Yue Haibing, who fixed a large number of small problems all over the kernel. The bulk of these problems, it seems, have been found using a system at Huawei called Hulk Robot, which uses various Coccinelle scripts and reports the results. Chris Wilson continues to contribute a lot of work to the Intel i915 graphics driver. Arnaldo Carvalho de Melo worked almost exclusively on the perf tool, Christoph Hellwig worked throughout the memory-management and I/O subsystems (and also removed a lot of obsolete code), and Jiri Olsa worked on perf as well.

Le Ma added 92,000 lines of code which was, as is often the case for the weightiest patches, mostly register definitions for the amdgpu graphics driver; Bhawanpreet Lakha also worked on the AMD graphics subsystem. Arnd Bergmann, in addition to fixing problems across the kernel tree, removed a number of obsolete drivers, and Ard Biesheuvel worked almost exclusively within the crypto subsystem.

Work on 5.4 was supported by 234 employers that we were able to identify; the most active of those were:

Most active 5.4 employers

By changesets Intel 1714 12.2% Red Hat 1048 7.4% (Unknown) 931 6.6% AMD 859 6.1% (None) 818 5.8% Google 759 5.4% Huawei Technologies 650 4.6% IBM 556 3.9% Linaro 546 3.9% Mellanox 357 2.5% NXP Semiconductors 334 2.4% Renesas Electronics 307 2.2% SUSE 280 2.0% (Consultant) 273 1.9% ARM 225 1.6% Linux Foundation 182 1.3% BayLibre 174 1.2% Canonical 173 1.2% Broadcom 162 1.2% Oracle 161 1.1%

By lines changed AMD 229309 25.3% Intel 75357 8.3% Linaro 66064 7.3% (Consultant) 51674 5.7% Red Hat 39670 4.4% IBM 33541 3.7% Google 33363 3.7% (None) 33362 3.7% (Unknown) 32469 3.6% Mellanox 27268 3.0% NXP Semiconductors 12879 1.4% Realtek 11655 1.3% BayLibre 11447 1.3% Cisco 11378 1.3% Facebook 11010 1.2% Virginia Tech 10941 1.2% Huawei Technologies 10291 1.1% Pensando 9740 1.1% SUSE 8887 1.0% Code Aurora Forum 8716 1.0%

As usual, there is not much in the way of surprises here. The 650 changesets from Huawei are a new high for that company; Huawei may be having significant political problems in the market, but its contributions at the code level continue unabated.

The testing and reviewing numbers this time around look like this:

Test and review credits in 5.4
Tested-by Andrew Bowers 162 16.9% Himanshu Madhani 60 6.3% Arnaldo Carvalho de Melo 33 3.4% Artur Rojek 25 2.6% Kevin Hilman 18 1.9% Ralph Campbell 18 1.9% Aaron Brown 16 1.7% Geert Uytterhoeven 13 1.4% Sedat Dilek 12 1.3% Mathieu Malaterre 12 1.3%	Reviewed-by Chris Wilson 219 3.5% Rob Herring 201 3.2% Christoph Hellwig 143 2.3% Alex Deucher 140 2.3% Hawking Zhang 115 1.8% Sam Ravnborg 88 1.4% David Sterba 87 1.4% Andrew Lunn 86 1.4% Emil Velikov 85 1.4% Simon Horman 71 1.1%

Andrew Bowers still has never contributed a patch to the kernel, but he collects Tested-by credits for patches from many other Intel developers for patches to the company's network drivers. The patches tested vary from significant changes to version-number tweaks. The top reviewer, Chris Wilson, is also based at Intel; when he is not writing i915 patches he is busily reviewing i915 patches written by others. Those patches are mostly, but not exclusively, from Intel developers as well.

The situation with Reported-by credits, given to those who report bugs, looks like this:

Reported-by credits in 5.4
Recipients Hulk Robot 263 25.4% Syzbot 85 8.2% kbuild test robot 71 6.8% Dan Carpenter 28 2.7% Stephen Rothwell 26 2.5% Christoph Hellwig 25 2.4% Qian Cai 10 1.0% Geert Uytterhoeven 9 0.9% Andrew Lunn 8 0.8% Arnd Bergmann 6 0.6% Russell King 6 0.6% Randy Dunlap 6 0.6% Guenter Roeck 6 0.6% Michal Kubecek 6 0.6%	Creditors Yue Haibing 236 22.8% Eric Dumazet 34 3.3% Gao Xiang 30 2.9% zhengbin 25 2.4% Will Deacon 14 1.4% Chris Wilson 14 1.4% Gustavo A. R. Silva 14 1.4% Jens Axboe 14 1.4% Nathan Chancellor 13 1.3% Stephen Boyd 10 1.0%

It is worth noting that, for previous kernels, the number of report credits for Syzbot, the fuzzing tool run at Google, has been significantly underreported. Syzbot requests credit in a unique format that, among other things, uses a different email address for each one; for 5.4, the gitdm tool has been enhanced to recognize and coalesce those credits to yield the correct number.

What jumps out here, though, is Hulk Robot, which is credited with reporting fully 25% of the bugs fixed in this development cycle. In the other column, we see Yue Haibing as the developer who added the most Reported-by tags to patches. Those two numbers are not unrelated: Haibing is the developer using this system to find bugs to fix. It is arguably not normal practice for developers to give report credits to bugs they find and fix themselves, but it does shine a light on how one developer is fixing large numbers of problems.

The increased use of automated testing systems is clearly having an impact; three such systems are now earning 40% of the bug-report credits, and they all far outpace any human reporters. Note that, contrary to some rumors, neither Dan Carpenter nor Stephen Rothwell is a robot. More seriously, there have been worries that Intel's 0day testing service ("kbuild test robot" above) is not proving as useful as it once was, but it still generated a fair number of bug reports during this cycle.

Overall, the numbers show that the kernel community continues to generate change at a high rate, and to bring in a significant number of new developers with each release. The fixing of problems seems to be happening at a high rate; hopefully they are being fixed more quickly than ones are being added. Either way, it looks like there will be plenty for the development community to do for the foreseeable future.

Comments (7 posted)

Emulated iopl()

Operating systems and computing hardware both carry a lot of their history with them. The x86 I/O-port mechanism is one piece of that history; it is rarely used by hardware designed in the last 20 years, but it must still be supported. That doesn't mean that this support can't be cleaned up and improved, though, especially when the old implementation turns out to have some unpleasant properties. An example can be seen in the iopl() patch set from Thomas Gleixner.

On most architectures, I/O is handled through memory-mapped I/O (MMIO) regions. A peripheral device will make a set of registers available as a range of memory; that range is then mapped into the processor's address space. Device drivers can then interact with the device by reading from and writing to those registers using normal memory accesses (or something close to that). This mechanism is flexible and it allows, for example, a set of registers to be mapped into a user-space process if the need arises; user-space drivers generally depend on this capability.

Back in the early days of the x86 architecture, though, things were done a little differently. A separate address space was created for up to 65536 I/O ports, which have to be accessed via special instructions. Even devices that could map memory ranges for other purposes would use I/O ports for their control interfaces. The instructions for accessing I/O ports are necessarily privileged, so user-space code cannot normally use them.

Once again, though, there is value in driving devices from user space at times. To support this functionality, the x86 designers created two separate ways to give an otherwise unprivileged process access to I/O ports:

• The I/O privilege level (IOPL) is a two-bit variable that controls how much privilege a process must have to access I/O ports. It is normally set to zero, meaning that this access is only available when running in kernel mode. Setting it to three makes I/O-port operations available to ordinary user-space processes. Changing the I/O privilege level for a specific process (done with the iopl() system call) can thus make all I/O ports available to that process.
• The I/O port permissions bitmap stored in the task state segment (TSS) can be used to grant access to specific ports. If the bit corresponding to a given port is zero, then the running task is allowed to access that port. The ioperm() system call is used to manipulate this bitmap.

A privileged process can call either iopl() or ioperm() to gain access to I/O ports. Calling ioperm() will increase the process's context-switch time, though, since the 8KB bitmap must be copied during a switch; for that reason, some applications use iopl(), even though it opens access to far more ports than needed.

There is, however, one other little problem with iopl(): an elevated I/O privilege level also allows the current process to disable and enable interrupts. That, as Gleixner pointed out, is less than ideal. A rogue process could easily lock up the CPU by disabling interrupts and looping, but the real issue is that there are no defined semantics for user space disabling interrupts. Kernel developers tend to assume that interrupts will be enabled while user space is running, but a process with an elevated IOPL can violate that assumption. Nothing good can be expected to come from a process actually exercising this privilege, but it simply comes as part of an elevated IOPL.

The most pleasing solution, Gleixner said, would be to just get rid of iopl() entirely, but there are still applications that depend on it so that cannot be done. But, perhaps, there is another solution: emulating iopl() by using the bitmap instead. If a process has an I/O privilege bitmap with all bits cleared, it has access to all I/O ports, just like it would with an elevated IOPL. But the ability to disable interrupts would be taken away.

Even doing that would be a problem if there were any applications that depend on the ability to disable interrupts in user space. Gleixner searched for such applications, but the only thing he found was a "really ancient X implementation". That code wouldn't run on current systems anyway, so it is not a concern. Hopefully there is nothing else out there that eluded Gleixner's search.

Switching to using the bitmap for iopl() solves the interrupt problem, but there is still the issue of the performance hit. A couple of optimizations in the patch set take care of that issue, though. Most processes don't use the bitmap at all; rather than set the bitmap to all ones for such processes, it is enough to change the pointer to the bitmap in the TSS to an invalid value and access to I/O ports will be denied. In the case of a context switch where both the incoming and outgoing processes are using the bitmap, only the portion with cleared bits needs to be copied, speeding that operation as well. In the end, the overhead of emulated iopl() is not zero, but it seems to be close enough.

Linus Torvalds pointed out that performance could be improved further by just leaving the I/O bitmap in place until something forces it to be changed. This optimization is aimed at the case where there is only one process running with access to I/O ports — a case that is likely to hold much of the time. Gleixner indicated that he would look at implementing this change.

Willy Tarreau suggested taking another step and just using an all-zeroes bitmap for any process that has called ioperm(). The result would be that a call that currently only grants access to specific ports would instead grant access to all ports. The calling process already has the privilege to request access to those ports, he said, so there wouldn't really be a security issue with that change. Eric Biederman pointed out, though, that DOSEMU actually counts on ioperm() not giving access to more ports than requested, so this idea is not workable in the end.

There was no opposition to the patch set in general, so a version of it is likely to be merged sometime in the near future. Then the kernel will have managed to leave behind a little piece of inconvenient legacy behavior, which can only be a good thing.

Comments (8 posted)

The 2019 Automated Testing Summit

This year saw the second edition of the Automated Testing Summit (ATS) and the first that was open to all. Last year's ATS was an invitation-only gathering of around 35 developers (that was described in an LWN article), while this year's event attracted around 50 attendees; both were held in conjunction with the Embedded Linux Conference Europe (ELCE), in Edinburgh, Scotland for 2018 and in Lyon, France this year. The basic problem has not changed—more collaboration is needed between the different kernel testing systems—but the starting points have been identified and work is progressing, albeit slowly. Part of the problem, of course, is that all of these testing efforts have their own constituencies and customers, who must be kept up and running, even while any of this collaborative development is going on.

Setting the stage

As with the first ATS, this edition was organized by Tim Bird and Kevin Hilman. Bird welcomed everyone to the conference then turned things over to Hilman for something of an overview of the "kernel testing landscape". Hilman started by noting that there were some gatherings and discussions at the Linux Plumbers Conference (LPC) in September, which he described in an email to the automated-testing mailing list. There were some themes that came out of those discussions, he said, which led to the title of his talk (slides [PDF]): "The bugs are too fast (and why we can't catch them)".

He gave a brief summary of the new kernel unit-testing frameworks that were discussed at LPC in order to bring attendees up to date on what kernel developers have been up to. The existing kernel test efforts, including kselftest, Linux Test Project (LTP), syzbot, and others, are likely pretty familiar to attendees, he said. The KUnit framework (LWN article from March) has been merged into linux-next; it is a fast way to test kernel functionality in an architecture-independent way and can be run in user space with user-mode Linux (UML). The Kernel Test Framework (KTF) is another unit-test framework that has been posted for comments. Since KUnit is headed for the mainline, though, the KTF project will need to figure out how to add its functionality to KUnit, Hilman said, since there won't be multiple unit-test frameworks in the mainline.

He then turned to the various testing initiatives that are currently active. The Intel 0-Day test service is probably the longest running; it is "mostly Intel focused". The Linaro Linux kernel functional testing (LKFT) has "quite a bit of in-depth testing but on a narrower set of hardware". The Red Hat continuous kernel integration (CKI) project has been around for a while, but has only recently been seen more publicly, he said; it is focused on testing stable kernels. And, of course, there is KernelCI that he cofounded; it was officially announced as a Linux Foundation project earlier in the week.

There is lots of testing going on, Hilman said, but there are a number of problems with that. One is that everyone is doing this testing off in their own corner; there is little collaboration between the efforts, which is the reason for the existence of ATS. All of the different players are testing areas that they care about, vendors are testing their hardware or software, developers test the platforms they care about, and so on, but the testing coverage of the kernel is concentrated; much of the kernel is tested, but everyone is generally testing the same parts of the kernel over and over. Broadening that coverage is an area that needs work.

But even with the fragmentation, these test efforts are finding "lots and lots of bugs"; "can we actually keep up and fix the bugs as we find them?" He referred to an LPC talk from Dmitry Vyukov (who was also present at ATS) that outlined some of the parameters of the size of the bug problem. Hilman used the statistics from that talk to note that more than 10% of the patches going into the kernel over the last few years carried a "Fixes:" tag, which means they are fixing a known bug that is identified in the tag. Not all patches that fix bugs use that tag, however, so the percentage of actual fixes is higher.

Beyond that, the raw numbers of bugs being found are mind-boggling. In two years, syzbot has found 5,800 crashes by fuzzing the kernel; in doing so, it has only exercised around 7% of the kernel. Does the kernel community actually have the capacity to handle "all" of the bugs that could be found? Vyukov estimated that there are 20,000 new bugs added in each major kernel release.

The LPC talk led to discussion at the Maintainers Summit, which was held a few days later. Every kernel developer has their own workflow to handle and track patches, which does not scale when the number of bugs being reported grows rapidly. There are efforts underway to figure out ways to automate some of those of processes, Hilman said. The workflows mailing list was created to discuss that; there are a lot of ideas on it, but "there is not a ton of consensus yet".

Hilman said that the fragmentation in the testing landscape (and elsewhere) is one of the things that is preventing the community from digging out from under all of these bugs. He noted that the reason for ATS is to help fix that fragmentation, so that "we can actually collaborate on fixing the issues rather than dealing with understanding each others' frameworks [...] or competing with each others' frameworks". The ultimate goal is to stop as many bugs as possible from even entering the released kernels. Defragmenting testing is only addressing one half of the problem; the kernel workflow also needs to work to get there.

An attendee noted that Hilman had talked a lot about kernel testing, but there is other testing being done as well. Bootloaders, user-space programs, and other components are being tested too; that kind of testing should continue. Hilman agreed; he focused on kernel testing mostly because the last few events that he was reporting on were also kernel-focused. There is definitely fragmentation in those other kinds of testing efforts, but "we gotta start somewhere" and kernel testing seemed like a good common denominator.

Status updates

A series of "lightning talks" was next on the agenda. These were short project overviews and updates for some of the different testing frameworks. There was more covered in each talk than is reported below; the portions that introduced the testing system were particularly helpful.

Milosz Wasilewski was first, describing LKFT, which targets both the Arm and x86 architectures. It focuses on testing the stable kernels; LKFT was originally set up to help Greg Kroah-Hartman maintain the long-term stable (LTS) kernels, so it tests the LTS branches and the most recent stable branch as well. It has also expanded into testing the mainline and linux-next kernels over time.

LKFT runs a variety of test suites, including LTP, kselftests, tests for libhugetlbfs, and various performance tests, he said. That adds up to around 25,000 tests that get run for every release and around one million tests that get run every every week. In addition, there has recently been some testing of Android kernels, though that is somewhat "less advertised".

Bird talked about the Fuego test system, which is targeted at testing on embedded devices. Fuego has its own Linux "distribution" that is based on Debian, with the Jenkins automation server, a test execution core, and a bunch tests installed on it. That is all wrapped up inside of a Docker container that runs on the host that is controlling the testing.

The focus for Fuego is not on testing the upstream kernel, but is instead for doing "product testing"—"high-level integration testing and benchmarking". Various facilities needed for testing (e.g. the netperf server) are integrated into the Fuego distribution, as is a cross-platform toolchain to build the tests from source. There are scripts to control test execution, including handling results parsing, analysis, and visualization. Multiple transports for talking with the device under test (DUT) are supported, including TCP, serial ports, and Android Debug Bridge (adb).

Fuego does not handle provisioning the device with the code to test; Bird hopes to be able to reuse the provisioning support from some other tool. His users mostly handle the provisioning as another Jenkins job in the test pipeline. The DUT is not required to have much in the way of capabilities in order to function with Fuego: just a shell with a limited set of commands, not including awk or sed, though grep is needed. All of those are available in BusyBox, he said. Some tests may require additional support on the DUT, however.

Bird was followed by another familiar face as Hilman stepped up to talk about KernelCI. That project's goal is to test on a wide range of hardware platforms, he said, which makes it different from many of the other frameworks. Today, it is running tests on more than 250 boards, with systems-on-chip (SoCs) from more than 35 vendors, and for multiple architectures: x86_64, arm, arm64, mips, arc, and riscv.

KernelCI tests multiple trees, including the mainline, linux-next, and the stable trees (as well as the stable release candidates). It also tests subsystem trees, maintainer and developer trees, the android-common tree, and the chrome-platform tree. It tests multiple kernel configurations, including all of the upstream defconfigs, and builds with multiple versions of both GCC and Clang.

For the most part, KernelCI is simply doing boot testing; it boots to a shell and does a few simple commands. For a subset of the boards, more functional tests are being added. The main thing is that the project does not want to develop and maintain its own test suites, but it is working with subsystems that have their own test suites to add them to KernelCI.

Veronika Kabatova was up next to describe CKI, which has the goal of preventing bugs from getting into the kernel in the first place; finding existing bugs is nice, she said, but not the goal of CKI. Getting there is hard, however, so the project has started by testing trees in the upstream kernel. It is testing the trees of the newest stable branch, stable-next, arm-next, RDMA, and rt-devel; some testing of other trees is being done, like net-next and mainline, but the results are not being sent out. It runs tests for the x86_64, arm64, ppc64, ppc64le, and s390x architectures. Overall, CKI is meant to complement the other existing continuous integration (CI) solutions for the kernel.

Unlike others, CKI does not use Jenkins; it uses the GitLab CI system instead. For actually running the tests, Beaker is used. Various kinds of input can trigger the system, including patches from Patchwork, Git commits, Koji builds, and more. Reports from CKI are sent to a mailing list.

Kcidb

Several of the framework talks referred to kcidb, which is something that came out of the CKI hackfest that was held right after LPC. In his talk, Hilman mentioned that there is a huge corpus of test results that KernelCI has squirreled away but is never used. It would be nice to be able to do something with all of that data. Other projects have the same problem, of course.

At the hackfest, KernelCI, LKFT, and CKI, got together to work on a simple JSON schema to describe test results. It is simply a starting point that can be used to gather all of this data into a common place, so that it can be processed with common tools to see what can be found within. There are tools to submit and query kcidb data in a BigQuery instance in the Google cloud. The "founding" three test frameworks are also working on adding a kcidb client to their reporting—as are other frameworks (Bird mentioned it for Fuego, at least).

Plenty more

This article is only scratching the surface of the all-day conference, covering much of the first quarter of the day. ATS soon split into two tracks, as can be seen in the schedule. There is a lot of information exchange going on at this point, trying to bring everyone up to speed on all of the other efforts, tools, frameworks, test hardware, and so on, which is reflected in the talk topics. But it seems abundantly clear that there is an enormous amount of defragmentation work to do.

The current focus on kcidb was not even two month's old when ATS was held on October 31—because kcidb itself was brand new. There is some interest in working on common test definitions that can be shared among frameworks. Bird's vision is that some day there would be a "test store" that is akin to an app store, where users could browse through thousands of tests to choose the ones that fit their needs. But for now, that effort is on the back burner, simmering while the test-results piece is solidified—starting with kcidb.

In the end, there is a lot of work to do and only a limited number of folks to do it—as is so often the case in the free-software world. In addition, it seems likely that LPC (and the CKI hackfest) being held so near in time to ATS split the participation to some extent. At the wrapup session at the end of the day, it was decided to try to focus on a single event, next year's LPC, rather than to reprise ATS for next year.

During that wrapup session, the "key decisions" were gathered, which can be seen on the ATS 2019 page of the Embedded Linux Wiki at elinux.org. In addition, Pengutronix folks took notes that they posted to the automated-testing mailing list.

Over the years, kernel testing has most certainly gotten a lot better, but there is still a long way to go. Collaboration seems like it will be pivotal in pushing kernel testing to the next level—and beyond. Gatherings like ATS, LPC, the CKI hackfest, and others will play an important role.

[I would like to thank Linaro for travel assistance to attend the Automated Testing Summit in Lyon, France.]

Comments (none posted)