Welcome to the LWN.net Weekly Edition for April 29, 2021

This edition contains the following feature content:

• Rethinking Fedora's compiler policy: will relaxing the compiler rule be good or bad for portability?
• Toward signed BPF programs: the BPF subsystem gets a program loader.
• Avoiding unintended connection failures with SO_REUSEPORT: fixing a longstanding flaw in how SO_REUSEPORT works.
• Preventing information leaks from ext4 filesystems: personal information can show up in surprising places.
• Some 5.12 development statistics: where the contributions to 5.12 came from.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Rethinking Fedora's compiler policy

Now that the Fedora 34 release is out the door, the Fedora project is turning its attention to Fedora 35, which is currently scheduled for release on October 26. One of the changes under consideration for Fedora 35 is this proposal allowing maintainers to choose whether to build their packages with GCC or Clang. This policy change may give maintainers some welcome flexibility, but it has not proved entirely popular in the Fedora community.

It is a longstanding practice for distributors to pick a specific toolchain to be used to build the full set of packages. Mixing compilers and linkers always presents the possibility of incompatibilities leading to subtle bugs, which is the sort of outcome distributors tend to go out of their way to avoid. Adhering to that practice, Fedora has long used GCC as its compiler of choice — a practice dating back to the time when there were no other compilers to use in any case.

Some packagers prefer LLVM

Over the years, though, the LLVM compiler suite and its Clang C compiler have grown in capability and become more popular. There are some programs that will not build with anything else at this point. For such programs, Fedora allows Clang to be used; for everything else, though, Fedora's policy requires building with GCC. This, according to the change proposal, creates unneeded difficulties for Fedora maintainers:

Upstream, the Firefox project builds primarily with Clang/LLVM. Yet we force the Fedora package owner to find and fix issues building with GCC then either carry those custom fixes forward in Fedora or negotiate with upstream to get those changes upstreamed. While this process can be helpful in finding non-portable code, this is ultimately a poor use of the packager's time.

The new proposed policy states that maintainers could use a non-default compiler for their package if they have a "valid technical reason" to do so. Beyond avoiding the above-mentioned Firefox problem, this change, it is argued, will enable the use of the compiler that yields the best result for any given package, thus improving the user experience overall. Beyond that, building with the compiler an upstream project uses for its own development and testing is, perhaps, less likely to tickle compiler-related bugs.

The real motivation behind this proposal, though, might be found in places like this RHEL bug report. The Ruby language has a just-in-time (JIT) compilation feature, and users would like to take advantage of it. But, due to GCC, when built as a position-independent executable, being incompatible with pre-compiled headers, JIT compilation fails with an unhelpful error message. There are various ways that this problem could be addressed; the morbidly curious can find them in the bug report. The conclusion, though, is that none of the options for making this feature work with GCC are deemed acceptable. If, however, LLVM were used, the problem would go away. Thus, unsurprisingly, the Ruby developers would like the freedom to use LLVM when building the Ruby package.

Questioning the change

The first question to arise in this posting of the proposal (it has actually been under discussion for nearly a year) was: who decides what's a "valid technical reason"? Will there be some sort of committee review? The consensus would appear to be that maintainers will be trusted to make the best decisions for their packages. The current version of the proposal differs from the original, which said maintainers should use the compiler that is preferred by the upstream project they are building.

The first posting of the new compiler policy (targeted at Fedora 33) drew a number of complaints, many of which resurfaced this time around. Neal Gompa described it as "an excuse to avoid doing the right thing and leveraging the toolchain that offers the highest quality code generation" and said that, at a minimum, the justifications for this change needed to be spelled out more clearly. Jakub Jelinek (a GCC developer) argued that subtle ABI incompatibilities still exist between the two compilers, and that mixing them increases the chance of introducing difficult bugs.

Mark Wielaard worried about the extra workload for developers of other parts of the toolchain:

For the packages I work on, elfutils, valgrind, debugedit, etc. we have enough trouble keeping up with gcc versions, new flags, optimizations, etc. I don't think we will have time to debug issues because packages are now also built with another compiler. Leaving the system as a whole in a worse state because of it.

Kevin Kofler argued that the best compiler overall should be used for the entire distribution — and that GCC is the best compiler for now.

A few participants, including Wielaard (linked above) and Florian Weimer, pointed out that even if Fedora adopts Clang when building projects that prefer it, the Fedora build will still differ in many ways, including the specific compiler version, the compilation flags used, and the setup of the system as a whole. The end result may well not be a better-tested build environment than was had using the default compiler.

Compiler diversity

One other concern mentioned by Jelinek in the above-linked message merits a mention. He agrees that competition between toolchains is good for everybody involved. While this proposal, on its surface, would appear to facilitate this competition by allowing the best compiler to be used everywhere, he says, the end result is more likely to be an increase in the number of packages that can only be built with LLVM. Upstream projects, especially those dominated by a single large company, often do not have portability across compilers as one of their important objectives. If distributors do not insist on that portability, it may well go away.

This would appear to be a valid concern. The growth of the Chrome-only web shows how this sort of capture can take place. GCC's slow rate of progress in the years before LLVM took off, and the acceleration of development since, highlight the value of this sort of competition. A return to a single-compiler world is not in anybody's interest.

Distributors, being the organizations that wield compilers for the benefit of vast numbers of users, are an obvious place to try to draw a line against a compiler monopoly. Whether distribution policies are the best way to promote compiler diversity is a different question, though. There may come a point where the pain of building with an out-of-favor compiler becomes more than people are willing to deal with, especially in volunteer-driven distributions. The apparent imminent end to Java support in Fedora shows what can happen when the work gets too overwhelming. Burning out package maintainers in the name of portability does not seem like a good way to achieve that goal.

The outcome of this particular change proposal is unclear at this point. Even if it is not adopted this time around, though, it seems likely that the pressure to make that sort of change will only increase in the future. Unless the forces that are driving projects to only support one compiler are somehow reduced, the cost of going against the stream will only grow.

Comments (34 posted)

Toward signed BPF programs

The kernel's BPF virtual machine is versatile; it is possible to load BPF programs into the kernel to carry out a large (and growing) set of tasks. The growing body of BPF code can reasonably be thought of as kernel code in its own right. But, while the kernel can check signatures on loadable modules and prevent the loading of modules that are not properly signed, there is no such mechanism for BPF programs; any sufficiently privileged process can load any program that will pass the verifier. One might think that adding this checking for BPF would be straightforward, but that subsystem has some unique characteristics that make things more challenging than one might expect. There may be a solution in the works, though; fittingly, it works by loading yet another BPF program.

Loadable kernel modules are stored as executable images in the ELF format. When one is loaded, the kernel parses that format and does the work needed to enable the module to run within the kernel; this work includes allocating memory for variables, performing relocations, resolving symbols, and more. All of the necessary information exists within the ELF file. Applying a signature to that file is simply a matter of checksumming the relevant sections and signing the result.

BPF programs have similar needs, but the organization of the requisite information is a bit more, for lack of a better word, messy. The code itself is compiled as an executable section that is then linked into a loader program that runs in user space and invokes the bpf() system call to load the BPF program into memory. But BPF programs, too, need to have data areas allocated in the form of BPF maps, and they need relocations (of a sort) applied to be able to cope with different structure layouts on different systems. The necessary maps are "declared" as special ELF sections in the loader program; the libbpf library finds those sections and turns them into more bpf() calls. The BPF program itself is then modified (before loading into the kernel) so that it can find its maps when it runs.

This structure poses a challenge for anybody wanting to implement signed BPF programs. The maps are a part of the program itself; if they are not established as intended, a BPF program might misbehave in interesting ways. But the kernel has no way to enforce any specific map configuration, and thus cannot ensure that a signed BPF program has been properly set up. Additionally, the need to modify the BPF program itself will break signature verification; after all, modifications to BPF programs are just the sort of thing this mechanism is expected to prevent. So, somehow, the kernel has to take a more active role in the loading of BPF programs.

In-kernel BPF loading

The old-timers among us will remember that, once upon a time, the kernel's module loader lived in user space. Moving it into the kernel was one of many causes of extended pain during the 2.5 development cycle; 20 years later, some developers still hold a grudge against Rusty Russell for that experience. But those problems are long past and the in-kernel loader has long since ceased to create problems. So one might logically expect that moving the user-space BPF loader into the kernel would be a sensible approach to take.

According to Alexei Starovoitov in the cover letter to a new patch set, that approach has been tried in a couple of forms and "discarded after months of work". Evidently an attempt was made to move libbpf into the kernel; it is not entirely surprising that this complex body of code did not fit comfortably there. Another idea was to create a new executable file format that contained, in essence, a series of system calls needed to set up a specific BPF program.

The problems that were encountered while implementing that second approach are not spelled out. But the third approach, found in Starovoitov's patch set, can be thought of as a variant on that idea. Rather than have the kernel step through a series of system calls, though, user space loads a special BPF program that does that work instead.

Specifically, the patch set creates yet another type of BPF program — one that exists to execute system calls. This program will run in the context of the process that runs it, and will be limited to a small set of system calls; only bpf() and close() are allowed in the proposed patch set. This program will be expected to make the necessary set of bpf() calls to load and set up the BPF program that the user really wants to run.

The generation of this "loader program" is done by watching what libbpf does to load the BPF program of interest and capturing each of the resulting bpf() calls. Those calls are then collected to generate the loader program, which will reproduce that work, from within the kernel, at the right time. So the kernel is, indeed, stepping through a canned series of system calls to load the program; it's just that this series is formatted as a BPF program in its own right.

The problem of patching the BPF program to find its maps is addressed in the usual way: adding another layer of indirection. An array of file descriptors is set up, and the BPF program references maps by way of that array. When the program is loaded, this array can be populated with the actual file descriptors corresponding to the necessary maps.

Next steps

As Starovoitov noted in the cover letter, this work is not yet a complete solution to the problem; it is a first step to show the direction that this work is taking. A big remaining piece is the offset relocations needed for BPF programs to access structure fields in a configuration-independent way. These relocations, too, require changing the BPF program text, so the solution may not be entirely trivial; more indirection-based schemes run the risk of imposing more of a performance cost than some users may want to pay.

There is also, of course, the little matter of signing BPF programs and checking those signatures; this problem is not addressed in this patch set either. There is a brief mention of putting together a skeleton that would allow BPF programs to be packaged into a kernel module; if that were done, then the existing system for checking module signatures could be used for BPF programs as well.

At a first glance, the BPF loader looks like a bit of a convoluted solution to the problem. It is worth noting, though, that this mechanism is not all that far removed from what happens in user space, where running a program usually involves launching ld.so to assemble the required pieces for that program to run. So there are well-established precedents to this sort of solution. Whether this design will make it into the mainline kernel is yet to be seen, though; this work is young and has not yet seen much review.

Comments (3 posted)

Avoiding unintended connection failures with SO_REUSEPORT

Many of us think that we operate busy web servers; LWN's server, for example, sweats hard when keeping up with the comment stream that accompanies any article mentioning the Rust programming language. But some organizations run truly busy servers and have to take some extraordinary measures to keep up with levels of traffic that even language advocates cannot create. The SO_REUSEPORT socket option is one of many features that have been added to the network stack to help these use cases. SO_REUSEPORT suffers from an implementation problem that can cause connections to fail, though. Kuniyuki Iwashima has posted a patch set addressing this problem, but there is some doubt as to whether it takes the right approach.

In normal usage, only one process is allowed to bind to any given TCP port to accept incoming connections. On busy systems, that process can become a bottleneck, even if all it does is pass accepted connections off to other processes for handling. The SO_REUSEPORT socket option, which was added to the 3.9 kernel in 2013, was meant to address that bottleneck. This option allows multiple processes to accept connections on the same port; whenever a connection request comes in, the kernel will pick one of the listening processes as the recipient. Systems using SO_REUSEPORT can dispense with the dispatcher process, improving scalability overall.

SO_REUSEPORT does its work when the initial SYN packet (the connection request) is received; at that time, a provisional new socket is created and assigned to one of the listening processes. The new connection will first wait for the handshake to complete, after which it will sit in a queue until the selected process calls accept() to accept the connection and begin the session. On busy servers, there may be a fair number of connections awaiting acceptance; the maximum length of that queue is specified with the listen() system call.

When SO_REUSEPORT misbehaves

Most of the time, SO_REUSEPORT works just as intended, but that can change when one of the listening processes exits. If a process quits with open network connections, those connections will be closed — not a surprising result. But the kernel will also "close" (by resetting) any incoming connections that are still in the accept queue. In the absence of SO_REUSEPORT this behavior makes sense; if the (single) listening process goes away, there is no longer anybody who can accept those connections.

If SO_REUSEPORT is being used, and if there are multiple listening processes, incoming connections do not necessarily have to be closed in this way. There are, after all, other processes running that would happily handle those connections. But once the kernel has committed an incoming connection to a specific process, it will not change its mind later; either that connection will be accepted by the chosen process, or it will be closed.

There are a number of reasons why a listening process might exit on a busy system. Perhaps it simply crashed. But, more likely, the server is being restarted to effect a configuration change or to switch to a new certificate. Such restarts can be phased across a pool of server processes so that they don't all exit at once; that should allow incoming connections to be handled without any apparent interruption of service. But when the above-described behavior comes into the picture, users can be turned away, which tends to have an unpleasant effect on their mood. The depressive effect on the operator of the site, who may have just lost the opportunity to learn that the would-be user is in the market for a new pair of socks, can be even worse.

There are ways around this problem, such as using a BPF program to steer incoming connections away from a server process that is about to exit, then being sure that it drains any queued connections before it bows out. But Iwashima makes the point that there is a better way: when a process exits, just take all of the queued incoming connections and reassign them to a different process for handling. After all, there is no state yet associated with an unaccepted request; one process can handle it just as well as another, and moving it will avoid causing the request to fail.

Migrating the accept queue

Getting there requires an eleven-part patch set, though. The first step is to add a new sysctl knob (net.ipv4.tcp_migrate_req; there does not appear to be an IPv6 version) controlling whether incoming connections should be moved to a new listener if the one they were assigned to exits. By default, this new behavior is disabled to avoid interfering with deployments where other arrangements have been made.

Actually migrating incoming connections away from an exiting server process is a bit more complicated than one might think because the "accept queue" is a bit more complicated than has been discussed so far. Remember that TCP connections go through a three-way handshake before being established: the connection is initiated with a SYN packet, the server side responds with SYN+ACK, and the initiator completes the connection with an ACK packet. That entire process must complete before an incoming connection can be given to a server process via accept().

Fully established connections — those that have completed the three-way handshake but which have not yet been accepted — are relatively easy to move to a new server process; they are just shunted from one queue to another. Connections that are still in the handshake are more complicated, though. They can only be moved at specific points during the sequence; when the handshake completes being the most obvious such point. It is also possible to move a connection when the SYN+ACK is retransmitted, should that be necessary. Either way, the remains of the old server process's socket structure must stay around for long enough to finish the handshake; that adds a certain amount of complexity.

One remaining question is: how is the new recipient for the connection chosen? Normally, the kernel will use the same algorithm it uses to pick a recipient in the first place, which is essentially a round-robin approach. But there will surely be users who know better and who want to be able to redirect these connections more explicitly. For those users there is, inevitably, a new BPF program type (BPF_PROG_TYPE_SK_REUSEPORT) that can be used to make decisions on where to reroute these connections.

Unacceptable?

As of this writing, the only comment on Iwashima's patch set comes from Eric Dumazet, who questioned whether it is the right approach. Since SO_REUSEPORT was added, he said, the TCP accept code has been reworked to run locklessly, which should address much of the scalability problem that SO_REUSEPORT was added to mitigate in the first place. Thus, he said, it might be better for applications to go back to a single-listener mode, perhaps helped by a new form of accept() that would allow incoming connections to be quickly directed to server processes.

That, of course, is a rather different development direction than Iwashima has taken so far and is thus unlikely to be welcome news. One could argue that, while a new accept() call might be a more pleasing solution to the overall problem, there should still be a place for a patch series making an existing kernel feature work without occasionally killing incoming connections. So it's not clear how this will play out; stay tuned.

Comments (36 posted)

Preventing information leaks from ext4 filesystems

A filesystem's role is to store information and retrieve it in its original form on request. But filesystems are also expected to prevent the retrieval of information by people who should not see it. That requirement extends to data that has been deleted; users expect that data to be truly gone and will not welcome its reappearance in surprising places. Some work being done with ext4 shows the kind of measures that are required to live up to that expectation.

In early April, Leah Rumancik posted a two-patch series making a couple of small changes to the ext4 filesystem implementation. The first of those caused the filesystem to, after a file is deleted, overwrite the space (on disk) where that file's name was stored. In response to a question about why this was needed, ext4 maintainer Ted Ts'o explained that it was meant to deal with the case where users were storing personally identifiable information (PII) in the names of files. When a file of that nature is removed, the user would like to be sure that the PII is no longer stored on the disk; that means wiping out the file names as well.

Dave Chinner quickly objected to this explanation. The real problem, he argued, is that users are storing PII as clear text; wiping directory entries is not a real solution to that problem:

This sounds more and more like "Don't encode PII in clear text anywhere" is a best practice that should be enforced with a big hammer. Filenames get everywhere and there's no easy way to prevent that because path lookups can be done by anyone in the kernel. This so much sounds like you're starting a game of whack-a-mole that can never be won.

From a security perspective, this is just bad design. Storing PII in clear text filenames pretty much guarantees that the PII will leak because it can't be scrubbed/contained within application controlled boundaries. Trying to contain the spread of filenames within random kernel subsystems sounds like a fool's errand to me, especially given how few kernel developers will even know that filenames are considered sensitive information from a security perspective...

The problem with that approach, as Ts'o explained, is that the people involved may not even know what their "legacy workloads" are doing in this regard. Rather than risk the possibility of exposing PII that nobody even knew was there, he said, it is better to simply clear the file names.

Of course, if a file's name constitutes PII, its contents are likely to be interesting as well. It is possible, though expensive, to overwrite the data in files when they are deleted. An alternative, on modern storage devices at least, is to use the FITRIM ioctl() command to tell the drive to discard the data. Even if this operation does not physically erase that data, it should make the data inaccessible afterward. For this reason (and others), administrators who are concerned about the persistence of deleted data tend to arrange for FITRIM operations to be run regularly.

There is one little remaining issue, though, for the truly paranoid. Ext4, like many contemporary filesystems, uses journaling to ensure that the filesystem remains consistent even if the system is interrupted at an inopportune time. To implement that sort of robustness, metadata written to ext4 filesystems is also written to the journal (and file data can optionally be written there too), so the possibility exists that the journal will contain PII even after all traces of it have been removed from the rest of the filesystem. A bad actor who gains access to the disk could harvest that data from the journal, even though it had already been deleted from the filesystem.

To address this problem, Rumancik's second patch adds a new ioctl() command, called FS_IOC_CHKPT_JRNL, that forces the journal to be flushed out to persistent storage. There is an optional flag, CHKPT_JRNL_DISCARD, which causes the filesystem to run the equivalent of a FITRIM operation on the journal itself once the flush is complete. That ensures that no PII remains in the journal itself.

Chinner (in the same email linked above) suggested that this behavior should be an option on the FITRIM operation rather than a separate command — before noting that FITRIM has no "flags" field and, thus, cannot be extended. Perhaps, he suggested, it is time for a separate fstrim() system call that could also trim the journal on request. A separate system call would also, by default, make the functionality available to all filesystems rather than being an ext4-specific feature.

The two patches together are intended to help administrators ensure that data that has been deleted from a filesystem truly disappears. It looks like they will be taking separate paths into the kernel from here, though. Rumancik recently posted a new version of the file-name overwrite patch on its own, and Ts'o subsequently applied it. The journal side of the problem, Ts'o said, is going to require some more discussion. Eventually, though, one can expect solutions to both problems to find their way into the kernel. That will help prevent the accidental disclosure of sensitive information from ext4 filesystems, even if the user is storing it in ill-advised ways.

Comments (61 posted)

Some 5.12 development statistics

By the time the 5.12 kernel was finally released, some 13,015 non-merge changesets had been pulled into the mainline repository for this development cycle. That makes 5.12 the slowest development cycle since 5.6, which was released at the end of March 2020. Still, there was plenty of work done for 5.12. Read on for our traditional look at where that work came from and how it got into the kernel.

Patches were contributed to 5.12 by 1,873 developers, 262 of whom were first-time contributors; those are typical numbers, especially given the (relatively) small size of this cycle. The most active 5.12 developers were:

Most active 5.12 developers

By changesets Lee Jones 256 2.0% Chris Wilson 167 1.3% Pavel Begunkov 158 1.2% Vladimir Oltean 120 0.9% Christoph Hellwig 117 0.9% Jens Axboe 113 0.9% Arnd Bergmann 109 0.8% Andy Shevchenko 94 0.7% Ben Skeggs 91 0.7% Paul E. McKenney 85 0.7% Zheng Yongjun 81 0.6% Ard Biesheuvel 77 0.6% Dan Carpenter 75 0.6% Hans de Goede 70 0.5% Alexandre Belloni 69 0.5% Geert Uytterhoeven 65 0.5% Sean Christopherson 64 0.5% Jiri Olsa 61 0.5% Chuck Lever 60 0.5% Colin Ian King 59 0.5%

By changed lines Arnd Bergmann 65277 9.4% Po-Hao Huang 21723 3.1% Viresh Kumar 16782 2.4% Maximilian Luz 14520 2.1% Andy Shevchenko 13160 1.9% Pawel Laszczak 10605 1.5% Ard Biesheuvel 9837 1.4% Sean Wang 9392 1.4% Chris Wilson 9255 1.3% Damien Le Moal 8949 1.3% Vladimir Oltean 7948 1.1% Hannes Reinecke 7854 1.1% Srujana Challa 7786 1.1% Thomas Zimmermann 6983 1.0% Bjorn Andersson 6978 1.0% Lorenzo Bianconi 6282 0.9% Srinivas Kandagatla 6100 0.9% Paul Kocialkowski 6050 0.9% Yevgeny Kliteynik 5967 0.9% Rob Herring 5870 0.8%

As with 5.11, Lee Jones was the most active changeset contributor this time around; he continues to work to fix compiler and docs-build warnings throughout the tree. Chris Wilson worked on the Intel i915 graphics driver. Pavel Begunkov worked mostly within the io_uring subsystem, Vladimir Oltean made a lot of improvements within the networking subsystem, and Christoph Hellwig continues to clean up the code in (mostly) the block layer and filesystems.

Arnd Bergmann deleted support for a number of obsolete architectures and their associated drivers, reaching the top of the "lines changed" column by virtue of the amount of code removed. Po-Hao Huang contributed eight patches, mostly updating some machine-generated tables. Viresh Kumar removed support for the old, unused "oprofile" profiling mechanism, Maximilian Luz added support for Microsoft Surface devices, and Andy Shevchenko removed some unneeded Intel driver code.

Report, test, and review credits

The kernel community depends heavily on contributions beyond just code; that includes those who report problems, and those who test and review code. Kernel developers try to credit these contributions through the addition of tags to the relevant patches. The most active credited reporters of bugs fixed in 5.12 were:

Most active 5.12 bug reporters
kernel test robot	184	16.1%
Syzbot	111	9.7%
Abaci Robot	107	9.4%
Dan Carpenter	44	3.9%
Hulk Robot	41	3.6%
Stephen Rothwell	28	2.5%
Randy Dunlap	19	1.7%
Kent Overstreet	12	1.1%
Guenter Roeck	11	1.0%
TOTE Robot	11	1.0%
Colin Ian King	9	0.8%
Andrii Nakryiko	8	0.7%
Juan Vazquez	7	0.6%
Arnd Bergmann	6	0.5%

Here we see the increasing presence of automated testing systems, which are finding bugs before (one hopes) they affect users.

The most active testers and reviewers this time around were:

Test and review credits in 5.12
Tested-by Daniel Wheeler 67 7.1% Tony Brelinski 62 6.6% Matt Merhar 33 3.5% Nicolas Chauvet 32 3.4% Peter Geis 31 3.3% Arnaldo Carvalho de Melo 29 3.1% Dmitry Osipenko 26 2.8% Wolfram Sang 21 2.2% Karthik B S 19 2.0% Marek Szyprowski 16 1.7% Sean Nyekjaer 16 1.7% Boqun Feng 15 1.6%	Reviewed-by Christoph Hellwig 220 3.6% Rob Herring 146 2.4% Laurent Pinchart 132 2.2% David Sterba 121 2.0% Florian Fainelli 109 1.8% Lyude Paul 91 1.5% Linus Walleij 87 1.4% Josef Bacik 82 1.4% Andrew Lunn 79 1.3% Hans de Goede 70 1.2% Darrick J. Wong 69 1.1% Chris Wilson 64 1.1%

The top folks in the Tested-by column appear to be routinely testing patches from their work colleagues — something that is probably much more widely done without applying tags to that effect. Matt Merhar, Nicolas Chauvet, and Peter Geis have almost the same number of credits, which is not a coincidence: all three tend to show as having tested the same patches, almost all from a single author (example). The Reviewed-by column is more diverse, with a mix of active maintainers and others who clearly put a lot of time into the review task.

Employer support

Work on 5.12 was supported by 211 employers that we were able to identify; that is a small decrease from previous releases but is in line with the lower patch volume in general. The most active companies this time were:

Most active 5.12 employers

By changesets Intel 1425 10.9% (Unknown) 1012 7.8% Red Hat 872 6.7% Linaro 868 6.7% Google 752 5.8% Huawei Technologies 525 4.0% Facebook 477 3.7% (None) 475 3.6% AMD 403 3.1% NVIDIA 402 3.1% IBM 372 2.9% SUSE 358 2.8% (Consultant) 333 2.6% Oracle 306 2.4% Renesas Electronics 252 1.9% Arm 248 1.9% NXP Semiconductors 237 1.8% Pengutronix 191 1.5% MediaTek 173 1.3% Alibaba 170 1.3%

By lines changed Linaro 120694 17.4% Intel 80056 11.5% Red Hat 38249 5.5% Google 29108 4.2% (Unknown) 28976 4.2% NVIDIA 28766 4.1% (None) 26298 3.8% Realtek 22572 3.3% SUSE 20388 2.9% MediaTek 19872 2.9% Arm 16063 2.3% Marvell 15400 2.2% AMD 14753 2.1% Pengutronix 13744 2.0% Western Digital 12827 1.8% Facebook 11885 1.7% Code Aurora Forum 11768 1.7% NXP Semiconductors 11669 1.7% IBM 10799 1.6% Texas Instruments 10744 1.5%

Unsurprisingly, these numbers look a lot like what we saw for 5.11 — and numerous kernels before that. The biggest change is the result of the flurry of code-removal patches coming from Linaro, which ended up at the top of the "lines changed" column as a result.

The path into the mainline

While Linus Torvalds is ultimately responsible for pulling all of the patches discussed here into the mainline, he handles few of them directly; by far the bulk of all patches going into the kernel go through at least one subsystem maintainer's repository. The path taken by patches can draw an interesting picture of how our community actually works; how closely does it match the hierarchical model that is often used to describe the kernel community?

An email message typically passes through a number of servers between sender and recipient; each server normally adds a "Received" header marking that passage. Git commits, instead, are immutable and accumulate no such markings from the repositories through which they pass. What does happen, though, is the addition of merge commits when a set of commits from one repository is pulled into another. At least, that happens most of the time. Some projects suffer from a sort of merge phobia; they require rebasing and fast-forward merges that do not leave any signs in the history. The kernel does not suffer from this particular syndrome, though, so most pulls do result in the creation of a merge commit.

Git will, when creating this commit, put the beginnings of a message in the changelog that includes the tree from which the commits are being merged. Assuming the maintainer leaves that information in place, this message makes it possible to recreate the path taken by those commits; happily, almost all kernel maintainers leave that information in place.

The merge commit also contains information about the GPG key (if any) that was used to sign the tag at the head of the set of commits to pull. This, too, is useful information. In theory, all pull requests should include signed tags as a way of ensuring that the associated commits were actually marked, by a recognized kernel maintainer, for merging into the mainline. In practice, not all maintainers have bothered to sign their commits in this way. Torvalds does not insist on signatures for pulls from trees hosted on kernel.org, but does generally require them for repositories stored anywhere else.

The treeplot program, part of the gitdm suite of ugly data-mining hacks, can collect all of this information and turn it into a graph. The result of this work for the 5.12 kernel can be seen on the right; clicking on that image and displaying the result in a large window should yield something that is actually legible. Rectangles indicate repositories, and arrows show the flow of commits between them; the number next to each arrow indicates how many commits took that path during the 5.12 cycle. Black indicates the use of signed tags, red shows their absence.

The shape of this graph has changed slowly over the years; it is still quite flat with a large number of repositories being pulled directly into the mainline by Torvalds, but we are seeing more going through mid-level maintainers as well. A number of the larger subsystems — networking, graphics, and Arm systems, for example — feed patches through multiple layers of maintainers. The largest stream of commits into the mainline continues to pass through the networking repositories, which were the source of over 2,400 commits going into 5.12.

The networking repositories are also the largest source of commits that are not protected with a signed tag; this situation has persisted for some years. All told, though, of the 126 repositories that fed commits into 5.12, all but 15 were using signed tags, a significant increase from past years. It is somewhat ironic that one of the remaining trees eschewing cryptographic signatures is the crypto tree.

While Torvalds requires signed tags for pulls from sites like GitHub, it can be seen that some other maintainers do not. There are, as a result, a few unsigned pulls from public hosting sites that are making it into the mainline kernel by way of an intermediate maintainer.

In the end, though, the picture that emerges from all of the above is a community that, for all of its difficulties, still manages to incorporate large numbers of changes in a reliable and predictable way. There are few projects out there that can sustain the scale of the kernel community and produce a stable and usable result. As of this writing, there are 13,677 changesets queued in linux-next, most of which will be destined for 5.13, so the process seems unlikely to slow down anytime soon.

Update: the employer numbers have been adjusted slightly since the initial publication of this article in response to a report from Leon Romanovsky.

Comments (none posted)

Quote of the week

The integrity and trust of the entire software industry has sharply declined due to cryptocurrency. It sets up perverse incentives for new projects, where developers are no longer trying to convince you to use their software because it’s good, but because they think that if they can convince you it will make them rich. I’ve had to develop a special radar for reading product pages now: a mounting feeling of dread as a promising technology is introduced while I inevitably arrive at the buried lede: it’s more crypto bullshit.

— Drew DeVault

Comments (54 posted)

Kernel release status

The 5.12 kernel is out, released on April 25. Quoth Linus: "Thanks to everybody who made last week very calm indeed, which just makes me feel much happier about the final 5.12 release."

Headline features in 5.12 include the removal of a number of obsolete, (mostly) 32-bit Arm subarchitectures, atomic instructions for BPF, conditional file lookups with LOOKUP_CACHED, support for zoned block devices in the Btrfs filesystem, threaded NAPI polling in the network stack, filesystem ID mapping, support for building the kernel with Clang link-time optimization, the KFENCE kernel-debugging tool, and more. See the LWN merge-window summaries (part 1, part 2) and the (in-progress) KernelNewbies 5.12 page for more information.

Stable updates: 5.11.17, 5.10.33, 5.4.115, 4.19.189, 4.14.232, 4.9.268, and 4.4.268 were released on April 28.

Comments (none posted)

A statement on the UMN mess

Speaking for the Linux Foundation Technical Advisory Board, Kees Cook has posted a brief statement on the controversy over patches submitted from the University of Minnesota.

The LF Technical Advisory Board is taking a look at the history of UMN's contributions and their associated research projects. At present, it seems the vast majority of patches have been in good faith, but we're continuing to review the work. Several public conversations have already started around our expectations of contributors.

Stay tuned for more.

Full Story (comments: 69)

A letter from the UMN researchers

The University of Minnesota researchers who have stirred up the kernel community with various types of bad patches have sent an open letter to the linux-kernel list. "This current incident has caused a great deal of anger in the Linux community toward us, the research group, and the University of Minnesota. We apologize unconditionally for what we now recognize was a breach of the shared trust in the open source community and seek forgiveness for our missteps."

Comments (36 posted)

"Full disclosure" from the University of Minnesota

The researchers at the University of Minnesota have posted a description of the work they did [PDF] as part of their "hypocrite commits" project. It includes a list of the buggy commits they posted and how they were handled.

In the following we will show two parts: (1) the message log of our disclosure of the findings to the community, and (2) the patches we submitted. By showing the details of the patches and the exchange of messages, we wish to help the community to confirm that the buggy patches were "stopped" during message exchanges and not merged into the actual Linux code. No other interactions with the Linux Kernel team has involved intentional deception or intentionally misleading or bad patches. This misguided behavior on our part was limited to the patches described and clarified in this document.

Amusingly, one of their attempts to submit a buggy commit was, itself, buggy, yielding a valid change overall.

Comments (17 posted)

An Interview With Linus Torvalds: Linux and Git (Tag1)

The Tag1 Consulting site has posted an interview with Linus Torvalds.

So I think the GPLv2 is pretty much the perfect balance of "everybody works under the same rules", and still requires that people give back to the community ("tit-for-tat"). And everybody knows that all the other people involved are bound by the same rules, so it's all very equitable and fair.

Of course, another part of that is that you also get out what you put in. Sure, you can try to "coast" on the project and be just a user, and that's ok. But if you do that, you also have no control over the project. That can be perfectly fine too, if you really just need a basic operating system, and Linux already does everything you want. But if you have special requirements, the only way to really affect the project is to participate.

Comments (23 posted)

Quote of the week

If the result of the exercise is that a typical device driver will contain more unsafe code than not, then device drivers are not a good starting point for Rust in the Linux kernel. In that case I would recommend that Rust start at a point where there is a lot of abstract code that is prone to the kind of problems that Rust is trying to solve. My intuition would be such things as network protocols. But I may be wrong.

— Linus Walleij

Comments (2 posted)

Fedora Linux 34 released

The Fedora 34 release is now available. "This release features GNOME 40, the next step in focused, distraction-free computing. GNOME 40 brings improvements to navigation whether you use a trackpad, a keyboard, or a mouse. The app grid and settings have been redesigned to make interaction more intuitive." LWN recently reviewed the Fedora 34 Workstation release.

Comments (none posted)

Ubuntu 21.04 released

The Ubuntu 21.04 distribution release is available. "Today, Canonical released Ubuntu 21.04 with native Microsoft Active Directory integration, Wayland graphics by default, and a Flutter application development SDK. Separately, Canonical and Microsoft announced performance optimization and joint support for Microsoft SQL Server on Ubuntu."

Comments (20 posted)

Yocto Project 3.3 (hardknott-25.0.0) released

Yocto Project, a system to build embedded Linux distributions, released version 3.3 "Hardknott". In this version all OE-Core recipes build reproducibly regardless of host distro/build location except golang recipes and ruby's docs package. There are many more new features, upgrades, and bug fixes. The release notes have more details.

Full Story (comments: 4)

GCC 11.1 released

Version 11.1 of the GCC compiler suite is out. "This release switches the default debugging format to DWARF 5 on most targets and switches the default C++ language version to -std=gnu++17. It makes great progress in the C++20 language support, both on the compiler and library sides, adds experimental C++23 support, some C2X enhancements, various optimization enhancements and bug fixes, several new hardware enablement changes and enhancements to the compiler back-ends and many other changes."

Full Story (comments: 1)

Development quotes of the week

We welcome the participation of researchers of all kinds in the Julia community. However, please remember that open source developmnent is a social process and that the participants on the other side of any interaction are (obviously) humans. If you are doing research *on the community or its processes*, you are likely performing Human Subject Research (HSR) and are expected to abide by the highest standards of ethics and professional practice. Meeting this expectation is your responsibility.

— The Julia language's amended code of conduct

Anyway. The moral of all of this is that sometimes history works out such that you write some software that a huge number of people run without any idea of who you are, and also that this can happen without you having any [...] idea what you're doing.

— Matthew Garrett

Comments (none posted)

Computer security world in mourning over death of Dan Kaminsky (The Register)

The Register reports on the death of security researcher Dan Kaminsky. "Though Kaminsky rose to fame in 2008 for identifying a critical design weakness in the internet's infrastructure – and worked in secret with software developers to mitigate the issue before it could be easily exploited – he had worked behind the scenes in the infosec world for at least the past two decades."

Comments (2 posted)

EuroPython 2021: Call for Proposals Open

EuroPython 2021 will take place July 26-August 1 online. "We're looking for proposals on all aspects of Python from novice to advanced level, including but not limited to: applications, frameworks, Data Science, internals or topics which you're excited about, your experiences with Python and its ecosystem and creative or artistic things you've done with Python." The call for proposals closes May 9.

Full Story (comments: none)

CFP Deadlines: April 29, 2021 to June 28, 2021

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
April 30	October 17 October 19	All Things Open	Raleigh, NC, and virtual
April 30	June 3	Live Embedded Event	Online
May 2	June 18 June 25	Akademy 2021	Online
May 4	June 18 June 20	openSUSE conference	online
May 12	June 1 June 30	Ceph Month June	Virtual
May 15	September 15 September 17	International Workshop on OpenMP	Bristol, UK
May 16	July 28 July 30	EuroPython 2021	Online
May 22	September 17 September 20	PyCon India 2021	Online
May 26	September 16 September 19	EuroBSDCon 2021	online
June 10	July 19 July 23	Netdev 0x15 Main Conference	Online
June 13	September 27 September 29	OSPOCon	Seattle, WA, and online
June 13	September 27 September 30	Open Source Summit + Embedded Linux Conference	Seattle, WA, and online
June 13	September 28 September 30	Qt Developer Conference [POSTPONED]	Berlin, Germany
June 14	September 13 September 14	OpenMP Developer Conference	Bristol, UK
June 20	August 22 August 29	DebConf	Online
June 20	October 11 October 12	Open Networking & Edge Summit + Kubernetes on Edge Day	Virtual

If the CFP deadline for your event does not appear here, please tell us about it.

Netdev 0x15 going virtual

The Netdev conference will be held July 20-23. There was hope that an in-person event in San Francisco, CA would be possible, but the NetDev Society board has decided that the Netdev 0x15 will be fully virtual. The decision was based on this evaluation [PDF] conducted by the IETF.

Full Story (comments: none)

GNOME Foundation announces GUADEC Keynote speakers

The GNOME conference, GUADEC 2021, will take place July 21-25 online. "Our keynote speakers for this year are Hong Phuc Dang and Shauna Gordon-McKeon. Hong Phuc Dang chairs the annual FOSSASIA Summit and organizes Open Tech Summits in countries from Vietnam, India, China, and Sri Lanka, to Germany. She is the board member of the Open Source Business Alliance Europe and also serves as Vice President of the Open Source Initiative. Shauna Gordon-McKeon is a writer, programmer and community organizer who focuses on the intersection of technology and governance. She co-organizes the Washington DC chapter of the Tech Workers Coalition. She serves on the board of Tech Inquiry and is an advisor to Metagov."

Full Story (comments: none)

Events: April 29, 2021 to June 28, 2021

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
May 2 May 3	PyCon Israel 2021	Online
May 4 May 6	sambaXP 2021	Online
May 13 May 15	Linux App Summit	online
May 14 May 15	PyCon US 2021	Online
May 25 May 26	Yocto Project Virtual Summit 2021	Online
June 1 June 30	Ceph Month June	Virtual
June 3	Live Embedded Event	Online
June 15 June 16	stackconf online 2021	Online
June 18 June 20	openSUSE conference	online
June 18 June 25	Akademy 2021	Online
June 22 June 23	Postgres Vision 2020	online
June 23	Linaro and Arm CCA Tech Event	Online
June 24 June 25	Build Meetup 2021	Virtual

If your event does not appear here, please tell us about it.

Alert summary April 22, 2021 to April 28, 2021

Dist.	ID	Release	Package	Date
Debian	DSA-4906-1	stable	chromium	2021-04-27
Debian	DLA-2637-1	LTS	drupal7	2021-04-23
Debian	DLA-2633-1	LTS	firefox-esr	2021-04-23
Debian	DLA-2644-1	LTS	gst-libav1.0	2021-04-27
Debian	DSA-4901-1	stable	gst-libav1.0	2021-04-24
Debian	DLA-2642-1	LTS	gst-plugins-bad1.0	2021-04-27
Debian	DSA-4902-1	stable	gst-plugins-bad1.0	2021-04-24
Debian	DLA-2641-1	LTS	gst-plugins-base1.0	2021-04-27
Debian	DSA-4903-1	stable	gst-plugins-base1.0	2021-04-24
Debian	DLA-2640-1	LTS	gst-plugins-good1.0	2021-04-26
Debian	DSA-4900-1	stable	gst-plugins-good1.0	2021-04-24
Debian	DLA-2643-1	LTS	gst-plugins-ugly1.0	2021-04-27
Debian	DSA-4904-1	stable	gst-plugins-ugly1.0	2021-04-24
Debian	DLA-2638-1	LTS	jackson-databind	2021-04-25
Debian	DLA-2635-1	LTS	libspring-java	2021-04-23
Debian	DLA-2639-1	LTS	opendmarc	2021-04-25
Debian	DSA-4899-1	stable	openjdk-11	2021-04-23
Debian	DLA-2634-1	LTS	openjdk-8	2021-04-23
Debian	DLA-2636-1	LTS	pjproject	2021-04-23
Debian	DSA-4905-1	stable	shibboleth-sp	2021-04-27
Debian	DLA-2632-1	LTS	thunderbird	2021-04-22
Debian	DSA-4897-1	stable	thunderbird	2021-04-22
Debian	DSA-4896-1	stable	wordpress	2021-04-22
Debian	DSA-4898-1	stable	wpa	2021-04-22
Fedora	FEDORA-2021-ec00da7faa	F33	buildah	2021-04-24
Fedora	FEDORA-2021-e29c1ee892	F34	ceph	2021-04-27
Fedora	FEDORA-2021-ec00da7faa	F33	containers-common	2021-04-24
Fedora	FEDORA-2021-ec00da7faa	F33	crun	2021-04-24
Fedora	FEDORA-2021-26a293c72b	F32	curl	2021-04-21
Fedora	FEDORA-2021-d8b1386945	F32	firefox	2021-04-26
Fedora	FEDORA-2021-5ed46601f6	F33	firefox	2021-04-21
Fedora	FEDORA-2021-6eb9bbbf0c	F33	java-11-openjdk	2021-04-24
Fedora	FEDORA-2021-8cd093f639	F32	kernel	2021-04-27
Fedora	FEDORA-2021-e6b4847979	F33	kernel	2021-04-27
Fedora	FEDORA-2021-8cd093f639	F32	kernel-headers	2021-04-27
Fedora	FEDORA-2021-e6b4847979	F33	kernel-headers	2021-04-27
Fedora	FEDORA-2021-8cd093f639	F32	kernel-tools	2021-04-27
Fedora	FEDORA-2021-e6b4847979	F33	kernel-tools	2021-04-27
Fedora	FEDORA-2021-f4223b6684	F33	mediawiki	2021-04-21
Fedora	FEDORA-2021-9bd201dd4d	F33	mingw-binutils	2021-04-21
Fedora	FEDORA-2021-1ffffa0251	F33	nextcloud-client	2021-04-24
Fedora	FEDORA-2021-242ef81244	F33	openvpn	2021-04-24
Fedora	FEDORA-2021-186bca5b58	F33	os-autoinst	2021-04-21
Fedora	FEDORA-2021-ec00da7faa	F33	podman	2021-04-24
Fedora	FEDORA-2021-b6b6093b3a	F32	python3-docs	2021-04-23
Fedora	FEDORA-2021-067c9deff1	F33	python3-docs	2021-04-23
Fedora	FEDORA-2021-067c9deff1	F33	python3.9	2021-04-23
Fedora	FEDORA-2021-eadfc56b95	F33	rpm-ostree	2021-04-21
Fedora	FEDORA-2021-ec00da7faa	F33	runc	2021-04-24
Fedora	FEDORA-2021-d7f74f0250	F32	rust	2021-04-27
Fedora	FEDORA-2021-b1ba54add6	F33	rust	2021-04-27
Fedora	FEDORA-2021-5aaebdae8e	F33	salt	2021-04-28
Fedora	FEDORA-2021-f7b4c97879	F32	xorg-x11-server	2021-04-24
Mageia	MGASA-2021-0195	7	connman	2021-04-24
Mageia	MGASA-2021-0196	7, 8	krb5-appl	2021-04-24
Mageia	MGASA-2021-0197	7, 8	virtualbox	2021-04-24
openSUSE	openSUSE-SU-2021:0606-1	15.2	ImageMagick	2021-04-23
openSUSE	openSUSE-SU-2021:0605-1	15.2	apache-commons-io	2021-04-23
openSUSE	openSUSE-SU-2021:0621-1	15.2	firefox	2021-04-26
openSUSE	openSUSE-SU-2021:0595-1	15	irssi	2021-04-23
openSUSE	openSUSE-SU-2021:0620-1		jhead	2021-04-26
openSUSE	openSUSE-SU-2021:0594-1	15.2	jhead	2021-04-23
openSUSE	openSUSE-SU-2021:0619-1	15.2	libdwarf	2021-04-26
openSUSE	openSUSE-SU-2021:0618-1	15.2	nim	2021-04-26
openSUSE	openSUSE-SU-2021:0601-1	15.2	nodejs-underscore	2021-04-23
openSUSE	openSUSE-SU-2021:0592-1	15.2	opera	2021-04-22
openSUSE	openSUSE-SU-2021:0597-1	15	python-django-registration	2021-04-23
openSUSE	openSUSE-SU-2021:0600-1	15.2	qemu	2021-04-23
openSUSE	openSUSE-SU-2021:0607-1	15.2	ruby2.5	2021-04-24
openSUSE	openSUSE-SU-2021:0598-1	15.2	shim	2021-04-23
openSUSE	openSUSE-SU-2021:0602-1	15.2	sudo	2021-04-23
Oracle	ELSA-2021-1363	OL7	firefox	2021-04-26
Oracle	ELSA-2021-1363	OL7	firefox	2021-04-27
Oracle	ELSA-2021-1360	OL8	firefox	2021-04-27
Oracle	ELSA-2021-1301	OL8	java-1.8.0-openjdk	2021-04-21
Oracle	ELSA-2021-1307	OL8	java-11-openjdk	2021-04-21
Oracle	ELSA-2021-1242	OL8	mariadb:10.3 and mariadb-devel:10.3	2021-04-27
Oracle	ELSA-2021-1350	OL7	thunderbird	2021-04-27
Oracle	ELSA-2021-1353	OL8	thunderbird	2021-04-27
Oracle	ELSA-2021-1354	OL7	xstream	2021-04-26
Oracle	ELSA-2021-1354	OL7	xstream	2021-04-27
Red Hat	RHSA-2021:1407-01	EL7	etcd	2021-04-27
Red Hat	RHSA-2021:1363-01	EL7	firefox	2021-04-26
Red Hat	RHSA-2021:1360-01	EL8	firefox	2021-04-26
Red Hat	RHSA-2021:1362-01	EL8.1	firefox	2021-04-26
Red Hat	RHSA-2021:1361-01	EL8.2	firefox	2021-04-26
Red Hat	RHSA-2021:1373-01	EL7.3	kernel	2021-04-27
Red Hat	RHSA-2021:1376-01	EL7.6	kernel	2021-04-27
Red Hat	RHSA-2021:1379-01	EL7	kernel-alt	2021-04-27
Red Hat	RHSA-2021:1377-01	EL7.6	kpatch-patch	2021-04-27
Red Hat	RHSA-2021:1384-01	EL7	nss	2021-04-27
Red Hat	RHSA-2021:1389-01	EL7	openldap	2021-04-27
Red Hat	RHSA-2021:1350-01	EL7	thunderbird	2021-04-26
Red Hat	RHSA-2021:1353-01	EL8	thunderbird	2021-04-26
Red Hat	RHSA-2021:1351-01	EL8.1	thunderbird	2021-04-26
Red Hat	RHSA-2021:1352-01	EL8.2	thunderbird	2021-04-26
Red Hat	RHSA-2021:1354-01	EL7	xstream	2021-04-26
Scientific Linux	SLSA-2021:1363-1	SL7	firefox	2021-04-26
Scientific Linux	SLSA-2021:1384-1	SL7	nss	2021-04-27
Scientific Linux	SLSA-2021:1389-1	SL7	openldap	2021-04-27
Scientific Linux	SLSA-2021:1350-1	SL7	thunderbird	2021-04-26
Scientific Linux	SLSA-2021:1354-1	SL7	xstream	2021-04-26
SUSE	SUSE-SU-2021:1315-1	SLE12	apache-commons-io	2021-04-26
SUSE	SUSE-SU-2021:14707-1	SLE11	curl	2021-04-28
SUSE	SUSE-SU-2021:1396-1	SLE12	curl	2021-04-28
SUSE	SUSE-SU-2021:1325-1	OS8 OS9 SLE12	firefox	2021-04-27
SUSE	SUSE-SU-2021:1307-1	SLE15	firefox	2021-04-23
SUSE	SUSE-SU-2021:1401-1	OS8 OS9 SLE12	gdm	2021-04-28
SUSE	SUSE-SU-2021:1314-1	SLE12	java-11-openjdk	2021-04-26
SUSE	SUSE-SU-2021:1301-1	SLE15	kernel	2021-04-21
SUSE	SUSE-SU-2021:14706-1	SLE11	kvm	2021-04-23
SUSE	SUSE-SU-2021:1399-1	OS8 OS9 SLE12	libnettle	2021-04-28
SUSE	SUSE-SU-2021:1310-1	SLE15	librsvg	2021-04-23
SUSE	SUSE-SU-2021:1292-1	SLE15	pcp	2021-04-21
SUSE	SUSE-SU-2021:1313-1	SLE15	python-aiohttp	2021-04-26
SUSE	SUSE-SU-2021:1305-1	SLE12	qemu	2021-04-22
SUSE	SUSE-SU-2021:14705-1	SLE11	tomcat6	2021-04-21
Ubuntu	USN-4924-1	16.04	dnsmasq	2021-04-22
Ubuntu	USN-4927-1	16.04 18.04 20.04 20.10	file-roller	2021-04-26
Ubuntu	USN-4926-1	16.04 18.04 20.04 20.10	firefox	2021-04-26
Ubuntu	USN-4916-2	14.04 16.04 18.04 20.04	linux, linux-aws, linux-gke-5.3, linux-hwe, linux-kvm, linux-lts-xenial, linux-oem-5.6, linux-raspi2-5.3, linux-snapdragon	2021-04-21
Ubuntu	USN-4892-1	16.04 18.04 20.04 20.10	openjdk-8, openjdk-lts	2021-04-27
Ubuntu	USN-4922-2	20.04	ruby2.7	2021-04-26
Ubuntu	USN-4925-1	20.04	shibboleth-sp	2021-04-22
Ubuntu	USN-4913-2	21.04	underscore	2021-04-28

Full Story (comments: none)