Welcome to the LWN.net Weekly Edition for July 7, 2022

This edition contains the following feature content:

• The 2022 embedded Linux update: Tim Bird gives a whirlwind tour of the embedded Linux landscape at the 2022 Embedded Linux Conference.
• An Ubuntu kernel bug causes container crashes: the chain of events that led to shipping a catastrophically buggy kernel.
• A BPF-specific memory allocator: a proposed free-memory cache for BPF programs.
• Removing the scheduler's energy-margin heuristic: an energy-saving mechanism turns out not to help.
• The end of CONFIG_ANDROID: a confusing configuration option is escorted out of the kernel.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

The 2022 embedded Linux update

A regular feature of the Embedded Linux Conference (ELC) has been an update on the state of embedded Linux from conference organizer Tim Bird. It has been quite a few years since I had the opportunity to sit in on one, so I took one at the 2022 Open Source Summit North America (OSSNA) in Austin, Texas. OSSNA is an umbrella conference that contains ELC and a whole lot more these days. Bird gave a look at recent kernel features from an embedded perspective, talked a bit about some different technology areas and their impact on embedded Linux, and also tried to answer a question that Andrew Morton posed in a keynote at ELC in 2008.

Among his many hats, Bird was the program committee chair for ELC and he is a principal software engineer at Sony Electronics. He started by saying that he was trying to squeeze a talk that he gives fairly regularly at other events down from its usual hour and a half—to 40 minutes. So he warned attendees that he would be moving fast. His goals with the talk were to introduce new technologies that have been added to the Linux kernel, so that attendees could perhaps incorporate them into their products, but also to start a conversation in the ecosystem about areas that need attention.

Past

Bird began his journey with Linux in 1993 and switched over to embedded Linux in 1998 at Lineo. He moved to Sony in 2003, where he was hired to help found the Consumer Electronics Linux Forum (CELF), which started the "Embedded Linux Technical Conference" in 2005. That is the direct predecessor to ELC. CELF moved under the Linux Foundation umbrella in 2010 as the Core Embedded Linux Project.

In the late 1990s, a lot of time and effort went into convincing companies that Linux was a good choice for embedded projects. The competition in those days was the commercial realtime operating systems (RTOSes) and "whole lot of roll-your-own bare-metal stuff that companies were doing on their own". The main architectures at the time, Intel, Arm, and MIPS, were supported by Linux, but no major chip vendor was working on the architecture support for the kernel; that work was done by volunteers and the embedded Linux companies. Distributions were either roll-your-own or came from one of a few vendors (e.g. MontaVista, Lineo, TimeSys)

One of the first things that CELF did once it got started was to analyze Linux to identify areas in need of work in order to make it suitable for embedded devices. Five specific areas were identified by CELF in 2003: system size, boot time, power management, realtime, and security. The organization then funded several kernel features in those areas (and some others, like filesystems).

So, a lot of work has been done, and it has been almost 20 years, "are we done yet?" That question came from Morton's keynote in 2008, perhaps as a bit of a joke, but it still haunts Bird to this day. Incidentally, he noted that video of the talk still exists thanks to Bootlin, which was known as Free Electrons back when the company recorded the talk (and many others in those days). Morton's talk is still relevant; "the stuff he said back then still applies today, it's amazing".

Kernel features

Then Bird went on a bit of a whirlwind tour of the last year's kernel releases from the perspective of features and changes of interest to embedded developers. There are, of course, lots of features going into each release for other use cases, which he would not be talking about. In addition, he would not be mentioning much about the different drivers or new system-on-chip (SoC) support that have also come with the new kernel releases, he said.

He began with Linux 5.14 from August 2021, which saw the addition of the osnoise and timerlat tracers that are useful for investigating realtime kernels, new Qualcomm and MediaTek drivers for a variety of devices, and a simpledrm driver that provides a direct-rendering manager (DRM) interface for simple framebuffer devices like those often found in embedded systems. The headline embedded feature in 5.14 was the addition of the memfd_secret() system call. It creates a region of memory that is inaccessible by other processes; even the kernel will have difficulty accessing it, so it is a good place to store secrets, like cryptographic keys. Bird recommended an LWN article and the commit for more information.

Kernel 5.15 in October 2021 brought sleeping spinlocks, which is a core piece of the realtime (i.e. PREEMPT_RT) patch set; he would have more to say about realtime later in the talk. The scheduler gained support for asymmetric systems, where cores on the same chip are not able to run both 32-bit and 64-bit code, so the scheduler needs to keep that in mind as it moves processes to different cores. Those working on network-attached storage (NAS) devices may want to look at the in-kernel SMB server, ksmbd, that was merged for 5.15.

Another feature that came in that kernel was printk() indexing, which gives the developers of tools that process log files a way to know when printk() messages change; that allows the tools to adapt rather than to silently fail when regular expressions suddenly fail to match. Since many embedded developers use older kernels—often provided by an SoC or board vendor—they may be surprised someday when they move to a 5.15 or later kernel because the mainline now builds with the -Werror flag by default. That flag turns compiler warnings into errors, so code that builds with warnings will cause the build to fail when it is used on more recent kernels.

In addition, the "data access monitor" (DAMON) tool was added. It provides a way to record and visualize data access patterns in the kernel. It is a diagnostic tool, like other tracers in Linux, "but is it actually more?" And, in fact, we see in 5.16 that DAMON has other functions beyond just monitoring, Bird said.

The 5.16 kernel in January 2022 also saw the addition of features for the Enhanced Read-Only File System (EROFS). In particular, it got support for multiple devices. The io_uring facility for asynchronous I/O has come on rapidly; it may be worth a look for embedded products. In 5.16, io_uring got support for security policy enforcement for SELinux and Smack. DAMON operating schemes (DAMOS) was added to help with proactive reclaim; DAMOS can offer hints to the kernel about what memory regions should be targeted. That may be particularly useful for embedded products with limited memory, he said.

In March, the 5.17 kernel was released. It can now decompress kernel modules within the kernel itself, instead of having to rely on user space to do it. This is important for certain secure boot scenarios and for the LoadPin security module. The realtime Linux analysis tool (RTLA) was also added; it uses the osnoise and timerlat tracers that were added in 5.14. In addition, some flag fields in the FUSE_INIT command for the Filesystem in Userspace (FUSE) facility have changed. So, systems using FUSE filesystems should be checked to ensure compatibility when moving to 5.17 and beyond.

The 5.18 kernel was released in May with several things of interest to the embedded community. For example, support for older Arm MMU-less architectures (ARMv4 and ARMv5) was removed. "It is not always true that the kernel just keeps getting bigger and bigger, sometimes stuff is removed." There is still support for MMU-less systems in the ARMv7 series, however. Beyond that, lots of RISC-V support was added, the tracing subsystem added user_events for dynamic tracepoints in user-space applications, and the kernel is now built with C11, rather than C89.

The big news in the upcoming 5.19 kernel is the near completion of the Arm multi-platform kernel work. Over the last 12 years, there has been an effort to consolidate all of the Arm kernels so that a single kernel image can be booted on all Arm systems just by providing a devicetree file specific to the board. There is support for a new RISC architecture, LoongArch, that was added, as well as a new hardware timestamp engine that he would be talking about further later on.

Bird spent a bit of time going over some kernel development statistics that will be familiar to LWN readers. That is because most of the stats, and many of the links he used in his slides come from LWN, he said.

Technology

Apologizing again for moving rapidly, he moved on to different technology areas that would be of interest to embedded developers. He said that another set of slides from a longer version of the talk might help fill in some of the gaps. He started with audio, noting that PipeWire is an up-and-coming project that embedded companies may want to look into, but that PulseAudio is still a big player, with a release of version 16.0 in May.

He listed a few "core kernel" changes that he had already talked about, but did expand a bit on the switch to C11. That came about because of the problems with speculative execution vulnerabilities (e.g. Spectre) and their interaction with kernel macros. It turns out that being able to declare variables inside loops, which is not possible using C89, alleviates some types of problems of that nature.

The other "big news in the core of the kernel" is the Rust for Linux project. It is currently an out-of-tree patch set of around 35,000 lines of code, which is considered experimental. The patches were recently moved to a newer version of the Rust language (1.59); "there's some grousing" about how fast the language version changes, he said. There have been a number of debates on various aspects of the project, including one going on that week on the ksummit-discuss mailing list. Most kernel developers have a "wait and see" attitude toward the project, Bird said.

For filesystems and I/O, various things are maturing, including io_uring; there are some zero-copy networking patches for io_uring that are "flying around", he said. Meanwhile, EROFS and Flash-Friendly File System (F2FS) have better support for compression and for extended attributes (xattrs).

In graphics, there are "a couple of small things and a couple of big things". In the former category are the addition of the simpledrm driver in 5.14 and that the framebuffer device (fbdev) subsystem got a new maintainer. The latter is particularly important for the embedded space where there are older devices that still need to be supported.

In the big-things department, the Mali GPUs now have a fully compliant OpenGL ES 3.1 open-source driver, called Panfrost. Also, NVIDIA has announced that it is releasing some of its code as open source. "This represents a big shift for NVIDIA". He is not sure what effect that will have on the nouveau project that has been working on open-source drivers for NVIDIA hardware, however.

"There is a steady stream of weird networking stuff going on." There is not a lot of work on new protocols and the esoteric ones that do get added are not really relevant to the embedded world, he said. He did note that the new mechanism for providing reasons for dropping packets will help in diagnosing problems of that nature.

Realtime and more

The progress with the PREEMPT_RT patch set is "really, really big news", Bird said. More and more patches for realtime Linux have been going into the mainline over the last few releases, starting with the sleeping spinlocks added in 5.15. Sleeping spinlocks allow another task to be scheduled while a lock is being held, which is how the realtime kernel can provide the low-latency guarantees. They are "probably the core feature of the PREEMPT_RT patch set".

At this point, the realtime patches have been "extensively tested and verified" over the 17-year history of the project. A natural question is: "what's left?" The remainder is about 1300 lines of code in around 50 patches. While that may sound like a lot, it actually is not; "compared to where this was before, this is unbelievable". In fact, 1700 lines of code have been merged from the patch set just since February.

In the security realm, he noted some control-flow integrity (CFI) changes that went into 5.13 to help ensure that jumps in the kernel went to places where they are expected to go. The memcpy() bounds checking went in for 5.16 to help keep memory copies from overrunning their buffers. There is also a steady stream of Spectre mitigations being added, though he was interested to note that a Spectre mitigation was removed in 5.16, as well, since it was not actually buying much in the way of increased security and it had a substantial performance cost.

In the kernel testing world, there is lots of testing going on, with many reports being made to the upstream kernel developers, coming from the 0-day test robot, KernelCI, Syzbot, and others. A new automated test framework, Compass CI, is one to keep an eye on, Bird said. It is being developed by Wu Fengguang, now at Huawei, who developed the 0-day test robot. The test suites, such as LTP, kselftest, and Kunit, all have new releases and new tests as well.

For toolchains, GCC released version 12.1 in May. It comes with better handling for uninitialized variables, along with some static-analysis features. LLVM also had a release in May (version 14.0.4). At this point, some people are using LLVM for building a whole distribution, not just the kernel, Bird said.

The hardware timestamping engine support that came in 5.18 is a nice addition to tracing features in Linux. It provides a mechanism to access hardware that can automatically attach a timestamp to a hardware event, without requiring a software interrupt, which is useful for low-overhead tracing.

That concluded his whirlwind tour, which was necessarily brief, but the idea is to give the audience some ideas of search terms they can use for various features. That should allow them to seek out more information from different sources to help as they develop new products and move to newer kernels.

Mars

Bird turned to an "interesting embedded Linux use case" that had been in the news quite a bit of late: the Mars Ingenuity helicopter. Bird said that for years he has been asking: "have we got Linux on Mars yet?; it turns out we finally did." The helicopter has some commercial off-the-shelf (COTS) hardware that is running Linux.

Ingenuity landed in February 2021 and its first five flights in April and May of that year were a "technology demonstration", but it worked so well, more flights have been done—29 in all at this point. One of those flights was able to take photos of the debris field of its "backshell" that came off as part of its descent to the surface. That was the first time NASA has been able to get such detailed pictures of a crash site, Bird said. Not only does it make for a interesting photo, but it is detailed enough to be analyzed to learn more about Mars.

Bird seemed particularly excited about the helicopter's ability to survive in the harsh environment; for example, it is now the Martian winter, and dust has reduced the amount of charging the solar panel can provide, so the device has to completely shut down at night. That means the COTS CPU is running well out of spec because it gets to -80C overnight; amusingly, some of the hardware in the helicopter came from the SparkFun hobbyist-electronics company, he said to laughter.

The cold has led to a loss of communications at times, due to clock problems, and it has also led to a broken inclinometer. NASA plans to send a software patch that will allow the helicopter to use other sensors to replace the inclinometer data. "They are going to update in the field on Mars." His slides have lots of links for more information. "I think it's one of the most interesting uses of embedded Linux in the solar system," he said with a chuckle.

Scorecards

Bird had three "scorecards" to judge the progress of embedded Linux over the years, for technology, development, and markets. Back in 2003, CELF had identified five areas that needed attention and, based on contributions to the kernel over the last few years, those areas are apparently now "done". There are few patches for system size, boot time, and power management these days, realtime is "done" as he said earlier, and security is always a work in progress. "So congratulations everybody, we did it."

Things are not quite that way, of course. There are few system-size patches anymore "because we kind of gave up". Linux is never going to run on one-cent processors, he said, so the 10-trillion IoT sensors will be running something else. A similar situation is at play for boot time; devices these days are using a variety of tricks, such as suspend/resume or going into a low-power state, to avoid booting at all. "You think your TV is off, it's not off."

Power management is more of a gray area, he said. Most of the needed features are upstream, but it is up to board vendors to actually integrate those features. Realtime is basically done, though it, like most of the rest of these areas, will require maintenance to ensure that it continues to work well.

On the development scorecard, he said that the community is doing quite well, overall. There are plenty of build system and distribution options for embedded systems, lots of different companies providing training and consulting, the toolchain support is in good shape, and the debugging options are plentiful.

Two development areas are still in progress, however. There are gaps in the test coverage, particularly in the areas of automated testing and hardware testing. In addition, while there has been enormous progress in terms of hardware support, much of the code from SoC and other vendors is not actually getting upstream. That means that companies making embedded Linux devices have to carry a lot of out-of-tree patches, thus increasing their technical debt.

Looking at markets, he had identified a non-comprehensive list of different types of embedded products, which shows that Linux is doing quite well. For example, Walmart and Amazon are working on drone deliveries using drones that run Linux; lots of commercial drones run it as well. Robots running the Linux-based Robot Operating System (ROS) are projected to be 55% of the commercial robot sales in 2024. Software in cars is largely Linux-based these days, as well.

One area that has surprised him is the use of Linux in space. He expected it to take a lot longer, but Linux use is improving quickly in that realm as well. Starlink and SpaceX use Linux, as does the Ingenuity helicopter, and some cubesats. As more commodity hardware goes into space, Linux will go with it, he thinks. Routers, mobile phones, and consumer electronics are dominated by Linux. In fact, in some areas, like TVs and DVRs, Linux has nearly 100% market share. "In consumer electronics, we're just killing it."

Overall, the embedded Linux community is "doing really well". Linux is widely deployed, at "world domination" levels. It is not going to be in everything though, and we won't see Linux running on a cereal box, perhaps ever. The core kernel has the support needed for embedded use cases, though there will always be a need for new drivers and other code to support new hardware.

He concluded that the bottom line is: "We're not done yet, but that's OK." Linux is ready to be used for lots of different kinds of devices. He hoped that his whirlwind had helped attendees get ready to design and build their next product and to hopefully become active and productive members of the ecosystem and community.

[I would like to thank LWN subscribers for supporting my travel to Austin for ELC.]

Comments (9 posted)

An Ubuntu kernel bug causes container crashes

Some system administrators running Ubuntu 20.04 had a rough time on June 8, when Ubuntu published kernel packages containing a particularly nasty bug that was caused by an Ubuntu-specific patch to the kernel. The bug led to a kernel panic whenever a Docker container was started. Fixed packages were made available on June 10, but there are questions about what went wrong with handling the patch; in particular, it is surprising that kernel 5.13, which has been beyond its end-of-life for months, made it onto machines running Ubuntu 20.04, which is supposed to be a long-term support release.

Ubuntu's kernel release lifecycle

Unless it is following a rolling-release model, a Linux distribution project will often pick a kernel branch and stick with it for the lifetime of a distribution release. For example, a release that ships with a 5.4 kernel, as Ubuntu 20.04 did, might receive updates to later 5.4.x kernels, but is unlikely to be upgraded to 5.15 until the next major release of the distribution. For this reason, such projects often prefer or even require a branch that has been designated as a long-term maintenance branch by the stable kernel team. It's easier for a distribution maintainer to sleep at night knowing that the version of the software they are shipping is supported upstream.

Debian and Red Hat both adhere to these rules when picking kernels for their releases; Ubuntu claims to do so as well, at least for its LTS (long-term support) releases. Those releases are made every two years, and supported for five. They ship with a long-term stable kernel; Ubuntu provides updates to that for the lifetime of the release.

Ubuntu also makes non-LTS releases at six month intervals in between the LTS releases. In contrast to the LTS releases, these releases are only supported for about six nine months, and are declared end-of-life (EOL) a month three months after a newer release is made available. Because of their relatively short shelf-life, Ubuntu does not restrict itself to long-term kernels for these releases. The most recent non-LTS release, Ubuntu 21.10, shipped with Linux 5.13, which is not a long-term branch. In fact, the 5.13 branch was declared EOL on September 18, 2021, almost a month before Ubuntu 21.10 was released on October 14.

Users who prioritize stability highly value the long window of support that comes with an LTS release, but five years is a relative eternity in the world of hardware, particularly in fast-moving areas like graphics. In order to support newer hardware, Ubuntu periodically publishes new hardware enablement (HWE) stacks for its LTS releases. These are comprised of packages backported from the latest (possibly non-LTS) release. The HWE stack includes updated kernel packages, and may also include updated Xorg and Mesa packages.

According to Ubuntu, the HWE stack is enabled by default for new desktop installs of Ubuntu, but needs to be explicitly chosen for server installs. This opt-in policy also seems to only apply to users installing from the ISO image; the default Ubuntu 20.04 images on Amazon AWS, Azure, and Google Cloud all come with the HWE kernel pre-installed. Many system administrators (including me) choose the HWE stack for their servers as well, either out of a desire for features only available in newer kernels or out of a need for a kernel that works with their hardware.

When considered independently from each other, the decision to bypass long-term kernels for non-LTS releases and the decision to publish HWE kernels to extend the hardware support of LTS releases both seem reasonable. In combination, though, these two decisions can lead to a somewhat surprising situation; users running a "long-term support" distribution can end up running a version of Linux which is considered end-of-life by the kernel developers.

As of this writing, users running the HWE kernel on Ubuntu 20.04 will get a 5.13 kernel backported from 21.10. Ubuntu 22.04, which is the next LTS release, includes the 5.15 kernel, which is a long-term stable branch. This is currently available to 20.04 users under the name "hwe-20.04-edge". It will presumably replace the kernel from 21.10 as hwe-20.04 sometime before July 14, when Ubuntu 21.10 is itself EOL. For now, though, and for the past few months, anyone running the HWE kernel on 20.04 is running a kernel based on 5.13. Since the HWE kernel is the default kernel on all three major clouds, problems with it can affect a large slice of Ubuntu's users.

A tale of four filesystems

The HWE kernel allows using newer hardware and kernel features with Ubuntu LTS, but it seems that this may come with some cost to stability. The root cause of the kernel crash lies at the intersection between no less than four different filesystems, although none of them are filesystems in the traditional sense of something that writes data to persistent storage.

The first is overlayfs. As the name might suggest, overlayfs allows overlaying the files in one directory (the "upper" directory, in overlayfs parlance) on top of the files in another (the "lower" directory.) This results in a mount point that contains all of the files in both the upper and the lower directories; if both directories contain a file with the same name, overlayfs presents the version present in the upper directory. Any changes made to an overlayfs mount are reflected in the upper directory. The functionality provided by overlayfs is particularly valuable to container runtimes such as Docker that store container images as a series of layers; overlayfs provides an efficient way of constructing a container's root directory from these layers. It has been a part of the kernel since version 3.18 in 2014.

The second filesystem involved is AUFS, which does everything that overlayfs does, and a lot more, but its implementation is significantly more complex. AUFS weighs in at about 35,000 lines of code, whereas overlayfs is about 12,000. AUFS was first submitted for inclusion into the kernel in 2008, but was never merged; since then, it has continued to be maintained out-of-tree. Ubuntu included AUFS in its kernels through version 20.10, but dropped it in 21.04.

The third filesystem is shiftfs, which was originally created by James Bottomley in 2018 to allow remapping the user and group IDs in a mounted filesystem, and while it has never been merged upstream, it has been included in Ubuntu's tree since the 5.0 kernel series. Canonical's LXD project can use shiftfs to speed up the creation of unprivileged containers, where the root user inside the container is mapped to a user other than root outside of it; otherwise, filesystems would need to have their user and group IDs changed to be used in that way. It is unlikely that shiftfs will ever land in Linus Torvalds's tree, though, as its functionality is entirely duplicated by the ID-mapped mounts that were added to the kernel in version 5.12. LXD has since been updated to use ID-mapped mounts when available.

The fourth and final filesystem in our story is procfs. As is generally known, each process running on a Linux system has a corresponding directory in /proc. Among a great many other things, each of these directories contains a subdirectory named map_files, which has a collection of symbolic links. Each link corresponds to a range of addresses in the process's address space that has been mapped to a file; the name of each link indicates the range of addresses that are mapped, and the destination is the file that is mapped to that range. For example:

    $ ls -l /proc/$$/map_files/
    total 0
    lr-------- 1 jordan everybody 64 Jun 22 16:21 55e0cc120000-55e0cc14d000 -> /usr/bin/bash
    lr-------- 1 jordan everybody 64 Jun 22 16:21 55e0cc14d000-55e0cc1fe000 -> /usr/bin/bash
    ...

The most prominent user of the map_files subdirectory is perhaps the Checkpoint/Restore In Userspace (CRIU) tool, which allows for "checkpointing" a process by serializing its entire state to disk, and later "restoring" it by recreating the process from its serialized state.

What does the patch do?

The patch that caused the kernel panic when creating Docker containers was intended to correct a problem when using overlayfs and shiftfs together. If a process mapped a file from such a mount, the symbolic link in map_files would point to the original "unshifted" version of the file, instead of the path inside the shiftfs mount. This broke checkpointing and restoring Docker containers, because the files the symbolic links in map_files were pointing to were in filesystems that weren't mounted inside the container.

This problem was discovered early in 2020, and fixed shortly after the release of Ubuntu 20.04. At the time, AUFS was included in Ubuntu's kernel. The developers of AUFS had also faced challenges related to differentiating between the real name of a file and its alias inside of an AUFS mount. To address this, the AUFS patch introduces an additional field called vm_prfile to the kernel's vm_area_struct, which is populated with AUFS's name for the file. To fix the problem with overlayfs and shiftfs, Ubuntu's developers needed to keep track of a file's alias inside of a synthetic mount, and, since AUFS had already added vm_prfile for a similar purpose, they chose to reuse it instead of introducing another field. Knowing that their fix was dependent on AUFS being enabled, they also chose to guard it in an #ifdef block — if AUFS was not configured into the kernel, then the patch became a no-op.

How things went wrong

When Ubuntu's developers ported the shiftfs-related patches from their 5.8 kernel branch to their 5.13 and 5.15 kernels, the patch that corrected the problem with map_files and shiftfs was left out, because it depended on AUFS, which had been dropped from Ubuntu's kernel. When those kernels were backported to Ubuntu 20.04, where AUFS continues to be supported, the missing patch was noticed, and it was applied to Ubuntu's 5.13 and 5.15 trees as well.

Unfortunately, the internals of overlayfs changed over time in a way that eventually caused the patch to be incorrect. As a result, when a file on an overlayfs is mapped into memory, the function added by the patch attempts to release a reference to a struct file using fput(), but the structure had already been freed due to an earlier fput() call. That causes the kernel to panic.

On Ubuntu 21.10, where 5.13 is the default kernel, this didn't cause any problems. Since AUFS is not enabled, the #ifdef block around the code introduced by the patch prevented it from being compiled into the kernel. The problem occurred when 5.13 and 5.15 were rebuilt for Ubuntu 20.04. Since an HWE kernel needs to support all of the features that are supported by the kernel it is replacing, AUFS was enabled in these builds, and the code containing the extraneous fput() was compiled in.

The problem was noticed in May, almost immediately after the patch was added back in. However, it appears that 5.13 was overlooked; the patch was reverted in Ubuntu's 5.15 branch and replaced with a version that did not call fput(), but the incorrect version remained in the 5.13 branch and made it into the 5.13 HWE kernel.

According to the changelog, the problematic kernel package was built on June 3, although it may not have been published to Ubuntu's package repositories for some time afterward. The problem was reported on June 8. Until updated packages were made available on June 10, the only recourse available to affected users was to manually roll back to a previous kernel.

Conclusion

Maintaining an out-of-tree kernel patch for any length of time is an arduous task. As much as Linux has an iron-clad guarantee of user-space compatibility, it provides zero assurances about the stability of internal kernel interfaces between versions. Things that do not get merged often quickly fall by the wayside, due to the sheer level of effort required to keep up with changes elsewhere in the kernel.

When Ubuntu ships out-of-tree patches with its LTS releases, it is signing its kernel developers up for the task of maintaining them for at least five years, often across multiple branches of the kernel simultaneously. Sometimes these bets pay off; Ubuntu included overlayfs in its kernel before it was merged, and now it is maintained upstream. On the other hand, even though Ubuntu dropped support for AUFS in 2021, because the distribution shipped it in 20.04, they are on the hook for supporting it until 2025. Their latest LTS release, 22.04, still contains support for shiftfs; those patches will be hanging around in Ubuntu's tree until at least 2027. As the problem with the patch demonstrates, keeping these patches up-to-date is no simple task; changes in other parts of the kernel can and will cause problems, which requires careful attention.

Based on those timelines, it doesn't seem like things are set to get any easier for Ubuntu's kernel developers anytime soon. Indeed, things may actually be destined to become harder; as the kernel now provides equivalent functionality, interest in these out-of-tree alternatives is likely to wane, which will place the burden of maintenance even more squarely upon Ubuntu's shoulders. The bets that don't pay off turn into debt, with compound interest.

In the end, it appears that Ubuntu fell victim to at least some level of self-inflicted complexity. Ubuntu's developers quickly caught and fixed the problem, but only in one of the affected branches. Unfortunately, the branch that was missed is the one that was shipped to users.

Comments (34 posted)

A BPF-specific memory allocator

The kernel does not lack for memory allocators, so one might well question the need for yet another one. As this patch set from Alexei Starovoitov makes clear, though, the BPF subsystem feels such a need. The proposed new allocator is intended to increase the reliability of allocations made within BPF programs, which might be run in just about any execution context.

Allocating memory in the kernel can be tricky in the best of situations. Depending on the execution context at the time, the memory-management subsystem may or may not have various options available to find memory if an allocation request cannot be immediately satisfied. For example, memory can be freed by pushing its contents out to persistent storage, but if memory is requested from within a filesystem, calling back into that filesystem to write out data could cause deadlocks and is thus not an option. In many kernel contexts, it is not possible to sleep to wait for memory to become free. If the kernel is currently handling a non-maskable interrupt (NMI) from the CPU, the options are even more limited.

Most kernel code is written to run within a specific context and with an awareness of the available memory-allocation options; that information is passed to the memory-management subsystem via the GFP flags supplied with allocation requests. When a specific function can be invoked in multiple contexts, it generally must allocate memory as if it were always running in the most restrictive possible context; this can be inconvenient for developers.

Over the years, mechanisms like memory pools ("mempools"), which pre-allocate a certain amount of memory to ensure that it will be available when it is needed, have been developed to make life easier. Naturally, mempools quickly created a new problem: kernel developers adopted mempools as a way of insulating themselves from memory-allocation failures. Before long, much of the kernel's memory was tied up in mempools and unavailable where it was actually needed. Over the years, a balance has mostly been found between overenthusiastic mempool use and being unable to allocate memory in critical situations.

Mempools for BPF

BPF programs can be run in just about any context imaginable; that is especially true of tracing programs. A given BPF program might be attached to a function that runs in atomic context, in response to hardware interrupts, or even one that handles NMIs. That makes allocation of memory an uncertain affair and that, in turn, makes life harder for the BPF verifier, which is meant to ensure that BPF programs will be well-behaved in all situations. BPF programs do not normally allocate memory directly, but they can do so indirectly by, for example, storing data in a BPF map. Having such an operation fail can create any number of follow-on problems.

The proposed BPF-specific allocator can be thought of as a sort of mempool that is specialized for the needs of BPF programs. As with mempools, the purpose is to create a cache of pre-allocated memory in a relatively relaxed context so that it is readily available in more restrictive times. While a mempool is, at its core, a cache made up of a single list of available objects of a single size, the BPF caches are somewhat more elaborate. At a minimum, a BPF cache consists of two lists of free objects of a single size (one for most allocations, one for allocations while handling NMIs) — for every CPU in the system. A more elaborate variant has eleven pairs of per-CPU lists for objects of eleven different sizes; that is 22 free-object lists for each CPU.

The core mechanism is straightforward enough: when a BPF program needs to allocate memory, it calls into the cache much as if it were calling kmem_cache_alloc() (for a single-size cache) or kmalloc() (for objects of variable size). A free object will be removed from the appropriate list for the current CPU and returned to the caller; an NMI-specific list will be used only if an NMI is being handled at the time. When objects are freed, they are returned to the appropriate per-CPU list for the current CPU, which may not be the CPU on which they were allocated. These operations are performed with interrupts and migration disabled, so they can happen without the need for any other sort of locking. Assuming that the cache is not exhausted, BPF programs will be able to safely allocate memory in any context.

When a cache is first set up, it will be populated with a maximum of four objects of each size in each regular list, and one of each size in each NMI list (for each CPU, of course). Whenever a object is allocated from a list, though, the number of available items is compared against a low watermark, which is 32 objects; if it is below that value, the irq_work mechanism is used to call a function to expand the list to 64 items. On non-realtime kernels, this work will happen in a kernel thread, where memory allocations are relatively unconstrained.

Similarly, whenever an object is returned to the BPF allocator, the length of the list is compared against the high watermark (96 items); if the list is too long, some objects will be returned to the kernel to bring the number of items down to 64. This prevents a cache from accumulating too many objects; it also deals with imbalances that may result if objects are allocated on one CPU and released on a different one.

Memory use

It is worth noting that these caches are not global objects; they are allocated independently for each user. For example, one cache is established for every hash map used by BPF programs, of which there could be many. It is thus natural to wonder about just how much memory might be consumed by these caches in a busy system. The behavior described above is clearly intended to make memory available for ready allocation while limiting the total amount of memory that sits in the cache itself. It is logical to expect that most users will not allocate objects of every available size on every available CPU, and most of them will never run in response to an NMI. Keeping hundreds of objects around just in case something like that does happen would be wasteful, so the BPF memory caches start with a minimal number of objects and each list will only be filled when actual allocations occur from that list.

Still, it seems reasonable to worry that these caches could grow to consume an excessive amount of memory, especially if there are a lot of them. Cache allocations are counted against the memory control-group limit, if one is in place, but there is no other limit on the memory that can be dedicated to BPF caches. There is no shrinker mechanism that could reduce the size of the caches should the kernel find itself in need of memory elsewhere, and there is no accounting to show how many of these caches exist or how much memory is currently allocated to them.

The only user of this mechanism added by this patch set is the hash-map implementation; it uses the fixed-size option. The existence of the kmalloc()-style option, though, suggests that there are other near-term users in mind.

In response to the posting, Christoph Hellwig suggested that Starovoitov talk with the maintainers of the existing slab allocators before proceeding. There have been few responses from the memory-management developers to patch series thus far, though, so it is not clear what their thoughts on the matter are. This new allocator is entirely contained within the BPF code and does not touch the memory-management subsystem, so it could, in principle, proceed without their involvement. That said, the memory-management developers have some expertise in this area and are likely to have some useful thoughts on how this problem should be solved.

Comments (none posted)

Removing the scheduler's energy-margin heuristic

The CPU scheduler's job has never been easy; it must find a way to allocate CPU time to all tasks in the system that is fair, allows all tasks to progress, and maximizes the throughput of the system as a whole. More recently, it has been called upon to satisfy another constraint: minimizing the system's energy consumption. There is currently a patch set in circulation, posted by Vincent Donnefort with work from Dietmar Eggemann as well, that changes how this constraint is met. The actual change is small, but it illustrates how hard it can be to get the needed heuristics right.

Reduction of energy use is, of course, a worthy goal; energy that is not wasted becomes available for the mining of more cryptocurrency, after all. There are some smaller considerations as well, such as environmental benefits, that justify the effort, but the proliferation of battery-powered devices has added more urgency to the task. If batteries can be made to last longer, doomscrolling interruptions will be fewer and users will be happier.

These pressures have led to the addition of energy-aware scheduling to the kernel. When the scheduler considers the placement of tasks in the system, it will work to reduce the amount of energy consumed overall; this work includes running the CPUs at the power level that is the most efficient for the current load and powering down processors entirely when possible. For example, if a CPU that is currently running at a given power level can accept another task without having to move to a higher power level, it may make sense to move a task there from another CPU.

In 2018, this patch from Quentin Perret (which was part of the energy-aware scheduling patch set) added a function called find_energy_efficient_cpu() to the scheduler; its job was to find the best place (from an energy-consumption point of view) for a given task. The heuristic used, at its core, is to find the least-busy CPU within each "performance domain" (cluster of CPUs whose energy usage is tied together) and estimate the energy cost (or savings) that would result from putting the task on that CPU. The least-busy CPU is the most likely to stay in a low-power state, so it makes a logical target for some extra work.

There is, however, a cost to moving a task from one CPU to another; that task may leave some or all of its memory caches behind, which will slow it down. That affects performance and is not good for energy use either, so it should be avoided whenever possible. As a way of preventing excess task movement between CPUs, find_energy_efficient_cpu() would only move a task if the result would be a savings of at least 6% of the energy used by the task's previous CPU.

The calculation of the best CPU was expensive, though, to the point where it was adding unwanted latency to scheduling decisions. So Perret reworked it for the 5.4 kernel release in 2019. The intent was to get the same results for less CPU cost; the patch changelog said "no functional changes intended". It turns out that there was a subtle change, though, that apparently escaped review: the 6% rule now compared against the energy used by the entire system, rather than just the previous CPU a task was running on. That is a relatively high bar to movement that, on a system with enough CPUs, could become impossible for a task to meet.

Even on smaller systems, the new rule effectively prevents task movement in many cases. This is especially true in situations where there are a relatively large number of small tasks running — a situation that is often found on Android devices, where energy efficiency is a real concern. If it is no longer able to move tasks to save energy, all of the work done by find_energy_efficient_cpu() is wasted and the device runs less efficiently than it otherwise would.

An obvious solution would be to undo the 5.4 change to the algorithm but, Donnefort said, "the original version didn't have strong grounds either". Indeed, there was never any reasoning given for the 6% number, which was 1.5% until it was raised to 6% in version 4 of the patch set. Its best feature may be that it is relatively easy to approximate with a right-shift operation. The conclusion Donnefort reached is that it would be better to simply remove that test entirely and migrate a task whenever it appears that the move would result in reduced energy consumption.

According to benchmarks posted with the patch set, the result is indeed better energy performance — up to a 5.6% reduction on a video benchmark. CPU performance is reduced slightly in some tests, but the change does not seem to be significant. As Donnefort put it:

The margin removal lets the kernel make the best use of the Energy Model, tasks are more likely to be placed where they fit and this saves a substantial amount of energy, while having a limited impact on performance.

One possible drawback of this change could be increased bouncing of tasks between CPUs, but Donnefort said that testing "showed no issue".

This patch set is in its eleventh revision, having seen a number of changes in response to review comments. In response to this version, scheduler maintainer Peter Zijlstra had just one word to say: "Thanks!". So it would appear that removal of the 6% heuristic makes sense, and that it will be finding its way into the mainline sooner rather than later.

Comments (1 posted)

The end of CONFIG_ANDROID

The kernel has thousands of configuration options, many of which can change the kernel's behavior in subtle or surprising ways. Among those options is CONFIG_ANDROID, which one might expect to be relatively straightforward; its description reads, in its entirety: "Enable support for various drivers needed on the Android platform". It turns out that this option does more than that, to the surprise of some users. That has led to a plan to remove this option, but that has brought a surprise or two of its own — and some disagreement — as well.

The discussion started when Alex Xu reported a read-copy-update (RCU) error that was appearing on his system after resuming from suspend. Shortly thereafter, Xu realized that the problem was tied to the fact that his kernel had been built with CONFIG_ANDROID enabled; among other things, that option significantly reduces the time that can elapse before RCU starts putting out stall warnings. RCU maintainer Paul McKenney was not entirely sympathetic after this was revealed:

And let's face it, the intent and purpose of CONFIG_ANDROID=y is extremely straightforward and unmistakable. So perhaps people not running Android devices but wanting a little bit of the Android functionality should do something other than setting CONFIG_ANDROID=y in their .config files. Me, I am surprised that it took this long for something like this to bite you.

This response comes from a part of the discussion that does not appear directly in the archives, but can be seen quoted in Xu's answer, where he points out that both Debian and Fedora ship kernels with CONFIG_ANDROID enabled, since that is the only way to make the binder module available. Xu suggested that the intent of this option is not quite as "straightforward and unmistakable" as one might think; the one-line description mentions nothing about changing internal RCU timeout values. "If major distro vendors are consistently making this 'mistake', then perhaps the problem is elsewhere".

Christoph Hellwig was quick to show up with a patch removing CONFIG_ANDROID altogether, describing it as "obviously a bad idea". Greg Kroah-Hartman was equally quick to agree and queue the patch for the next merge window. Nobody else objected — until Jason Donenfeld pointed out that this option has other surprising effects, and that removing it could create problems on Android systems.

Specifically, both the random-number generator and the WireGuard VPN tunnel implementation will make changes in response to the system being suspended. The random-number generator will reseed itself after the system resumes, while WireGuard will clear its key material just prior to suspending. Both actions are intended to improve security, but they can be problematic on Android systems due to how power management is handled there. Devices running Android are narcoleptic; they will go to sleep at any opportunity in order to save power. Resetting the random-number generator that frequently is inefficient at best, and clearing the WireGuard keys that often may disrupt communications entirely. To avoid such problems, these actions are not taken if the kernel has been built with CONFIG_ANDROID.

The removal of CONFIG_ANDROID also removes that special behavior; this is a change that, Donenfeld feared, could create regressions in the future. He asked Hellwig for either an assurance that these problems would not result, or for an updated version of the patch that fixed those problems. There followed a not-entirely-pleasant discussion over whose responsibility it was to fix any problems, whether that use of CONFIG_ANDROID was correct, whether the removal constitutes a user-space ABI regression, and so on.

Eventually Kroah-Hartman signaled his agreement with Hellwig. Any problems experienced by Android devices, he said, would be long since found and fixed by the time a patched kernel actually is shipped to such a device, but the change might fix desktop-related problems now. So this change appears to be headed toward the mainline.

At the core of the debate was the use of CONFIG_ANDROID as an indicator that the system will suspend and resume frequently. But, as has been seen, there are many systems with CONFIG_ANDROID enabled that do not exhibit that behavior, but which are getting the related changes anyway. There may also be systems that suspend frequently and that should see that behavior, but which are not running Android and do not have CONFIG_ANDROID enabled. The consensus seems to be that using CONFIG_ANDROID to regulate RCU stall timeouts or cryptographic power-management responses is a bug that is in need of fixing.

So, for the purposes of random numbers and WireGuard, some other way to indicate that the system will suspend frequently will be needed. There was talk of a new configuration option or a sysfs knob that could be written to from user space, which would allow the behavior to be changed at run time. Your editor's suggestion that the kernel could observe actual suspend behavior and do the right thing on its own was fairly quickly dismissed.

What will happen instead, it seems, is the addition of a new configuration option, called CONFIG_PM_USERSPACE_AUTOSLEEP, that prepares the kernel for a system that will suspend frequently and enables the (formerly) Android-specific behavior. This option has a more extensive help text describing what it actually does and warning that it "should not be enabled just for fun". The necessary Android changes have already been created, and this appears to be a solution that everybody involved is happy with.

The way this solution came about could have been better, though. The kernel community works best when developers work together toward a common goal rather than argue over who is doing things incorrectly. That did eventually happen here, but it took some time to get to that point. It took multiple developers to endow CONFIG_ANDROID with its somewhat confusing semantics; it is unsurprising that it took more than one person to straighten it out.

Comments (18 posted)

Kernel release status

The current development kernel is 5.19-rc5, released on July 3. Linus said: "So everything looks ok - we certainly have some issues still being looked at, but on the whole 5.19 looks normal, and nothing particularly bad seems to be going on".

Stable updates: 5.18.9, 5.15.52, 5.10.128, 5.4.203, 4.19.250, 4.14.286, and 4.9.321 were released on July 2. The 5.18.10, 5.15.53, 5.10.129, 5.4.204, 4.19.251, 4.14.287, and 4.9.322 updates are in the review process; they are due on July 7.

Comments (none posted)

Debian 9 Long Term Support reaching end-of-life

The Debian Long Term Support (LTS) team has announced that Debian 9 ("stretch") has "reached its end-of-life on July 1, 2022, five years after its initial release on June 17, 2017". There will be further updates for a subset of the packages in the release through the Extended LTS project. Meanwhile, the LTS team is moving on to Debian 10 ("buster"):

The LTS Team will prepare the transition to Debian 10 buster, which is the current oldstable release. The LTS team will take over support from the Security Team during August, while the final point update for buster will be released during that month.

Debian 10 will also receive Long Term Support for five years after its initial release with support ending on June 30, 2024. The supported architectures will be announced at a later date.

Full Story (comments: 1)

Distributions quote of the week

After installing or upgrading your Fedora or RHEL system, you have to accept a "do you trust this official Fedora project key" prompt or you cannot install packages from the official repos. So all our users have been trained to ignore warnings about untrusted packages because it's mandatory to do so. If few users think twice about accepting a key as long as it purports to be from "Fedora" or "Red Hat"... well, the whole system is subverted. This needs a rethink.

— Michael Catanzaro

Comments (4 posted)

Darktable 4.0.0 released

Version 4.0.0 of the darktable raw photo editor has been released. "The UI has been completely revamped again to improve look and consistency. Padding, margins, color, contrast, alignment, and icons have been reworked throughout". Other changes include new exposure and color-calibration modules, a reworked "filmic" color-mapping module, guided laplacian highlight reconstruction, and more. (LWN looked at darktable in January).

Comments (none posted)

Rust 1.62.0 released

Version 1.62.0 of the Rust language has been released. Changes include a new cargo add command, default enum variants, an improved Linux mutex implementation, a number of stabilized APIs, and more.

Comments (none posted)

Amazon's CodeWhisperer

There has been a fair amount of concern recently about Microsoft's Copilot system, which many see as possibly putting its users in violation of free-software licenses. But, naturally, Copilot is not the only offering of this type; Amazon has put out a preview version of "CodeWhisperer", which is also a machine-learning-based coding tool that was trained on (unspecified) open-source code. From the FAQ:

CodeWhisperer’s reference tracker detects whether a code recommendation may be similar to particular CodeWhisperer training data, and can provide those references to you. This allows you to easily find and review that reference code and how it is used in the context of another project.

Comments (28 posted)

Software Freedom Conservancy: Give Up GitHub: The Time Has Come!

The Software Freedom Conservancy (SFC) has issued a strong call for free software projects to give up GitHub and to move their repositories elsewhere. There are a number of problems that SFC has identified with the GitHub code-hosting service and, in particular, with its Copilot AI-based code-writing tool that was trained on the community's code stored in the company's repositories. Moving away from GitHub will not be easy, SFC said, but it is important to do so lest the free-software community repeat the SourceForge mistake.

Specifically, we at Software Freedom Conservancy have been actively communicating with Microsoft and their GitHub subsidiary about our concerns with "Copilot" since they first launched it almost exactly a year ago. Our initial video chat call (in July 2021) with Microsoft and GitHub representatives resulted in several questions which they said they could not answer at that time, but would "answer soon". [...] Last week, after we reminded GitHub of (a) the pending questions that we'd waited a year for them to answer and (b) of their refusal to join public discussion on the topic, they responded a week later, saying they would not join any public nor private discussion on this matter because "a broader conversation [about the ethics of AI-assisted software] seemed unlikely to alter your [SFC's] stance, which is why we [GitHub] have not responded to your [SFC's] detailed questions". In other words, GitHub's final position on Copilot is: if you disagree with GitHub about policy matters related to Copilot, then you don't deserve a reply from Microsoft or GitHub. They only will bother to reply if they think they can immediately change your policy position to theirs. But, Microsoft and GitHub will leave you hanging for a year before they'll tell you that!

Comments (207 posted)

CFP Deadlines: July 7, 2022 to September 5, 2022

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
July 8	October 16 October 21	DjangoCon US 2022	San Diego, USA
July 10	September 14 September 15	Git Merge 2022	Chicago, USA
July 15	September 16 September 18	Ten Years of Guix	Paris, France
July 29	November 11	Software Supply Chain Offensive Research and Ecosystem Defenses 2022	Los Angeles, USA
July 31	November 14 November 16	Open Source Monitoring Conference 2022	Nürnberg, Germany
August 1	August 13 August 14	The Raku Conference	Online
August 19	October 13 October 14	PyConZA 2022	Durban, South Africa
August 29	September 9 September 11	Qubes OS Summit	Berlin, Germany
September 1	October 15 October 16	OpenFest 2022	Sofia, Bulgaria
September 1	October 11 October 13	Tracing Summit 2022	London, UK

If the CFP deadline for your event does not appear here, please tell us about it.

Events: July 7, 2022 to September 5, 2022

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
July 7 July 9	Free Silicon Conference 2022	Paris, France
July 10 July 15	Debcamp 2022	Prizren, Kosovo
July 11 July 17	SciPy 2022	Austin, TX, USA
July 17 July 24	Debconf 2022	Prizren, Kosovo
July 19 July 20	stackconf 2022	Berlin, Germany
July 20 July 25	GUADEC 2022	Guadalajara, Mexico
July 21	Icinga Camp Berlin	Berlin, Germany
July 22 July 24	TUG 2022 - Online Presentations Covering the World of TeX	Online
July 23 July 24	IndiaFOSS 2022	Bangalore, India
July 28 July 31	Southern California Linux Expo	Los Angeles, CA, USA
August 4 August 7	Fedora Nest / Hatch 2022	Online
August 13 August 14	The Raku Conference	Online
August 18 August 20	Devconf.US 2022	Boston, USA
August 20 August 22	FrOSCon 2022	Sankt Augustin, Germany
August 20 August 26	Free and Open Source Software for Geospatial	Florence, Italy
August 23 August 24	Open Source Summit Latin America	Online

If your event does not appear here, please tell us about it.

Alert summary June 30, 2022 to July 6, 2022

Dist.	ID	Release	Package	Date
Debian	DSA-5176-1	stable	blender	2022-07-04
Debian	DLA-3064-1	LTS	firefox-esr	2022-07-01
Debian	DSA-5172-1	stable	firefox-esr	2022-06-29
Debian	DLA-3061-1	LTS	firejail	2022-06-29
Debian	DSA-5174-1	stable	gnupg2	2022-07-03
Debian	DLA-3066-1	LTS	isync	2022-07-01
Debian	DLA-3065-1	LTS	kernel	2022-07-01
Debian	DSA-5173-1	stable	kernel	2022-07-03
Debian	DSA-5177-1	stable	ldap-account-manager	2022-07-05
Debian	DLA-3063-1	LTS	systemd	2022-06-30
Debian	DSA-5175-1	stable	thunderbird	2022-07-04
Debian	DLA-3062-1	LTS	ublock-origin	2022-06-29
Fedora	FEDORA-2022-bcb096166f	F35	chromium	2022-07-01
Fedora	FEDORA-2022-7416607232	F36	chromium	2022-06-30
Fedora	FEDORA-2022-8bd3bf5b40	F36	curl	2022-07-01
Fedora	FEDORA-2022-5d052cb6e0	F35	firefox	2022-07-01
Fedora	FEDORA-2022-59e474cd5c	F36	firefox	2022-06-30
Fedora	FEDORA-2022-fae3ecee19	F36	golang-github-apache-beam-2	2022-07-04
Fedora	FEDORA-2022-fae3ecee19	F36	golang-github-etcd-io-gofail	2022-07-04
Fedora	FEDORA-2022-fae3ecee19	F36	golang-github-intel-goresctrl	2022-07-04
Fedora	FEDORA-2022-fae3ecee19	F36	golang-github-spf13-cobra	2022-07-04
Fedora	FEDORA-2022-e674d52438	F36	golang-github-vultr-govultr-2	2022-07-01
Fedora	FEDORA-2022-fae3ecee19	F36	golang-k8s-pod-security-admission	2022-07-04
Fedora	FEDORA-2022-3b7d0abd0b	F36	openssl1.1	2022-07-06
Fedora	FEDORA-2022-99d02c5100	F35	thunderbird	2022-07-06
Fedora	FEDORA-2022-01b6d6f289	F36	thunderbird	2022-06-30
Fedora	FEDORA-2022-bb7f3cacbf	F35	vim	2022-07-04
Fedora	FEDORA-2022-719f3ec21b	F36	vim	2022-06-30
Fedora	FEDORA-2022-925fc688c1	F36	xen	2022-07-01
Fedora	FEDORA-2022-cbc7bfd88c	F36	yubihsm-connector	2022-07-06
Mageia	MGASA-2022-0250	8	curl	2022-07-05
Mageia	MGASA-2022-0247	8	cyrus-imapd	2022-07-05
Mageia	MGASA-2022-0251	8	firefox	2022-07-05
Mageia	MGASA-2022-0242	8	kernel	2022-06-29
Mageia	MGASA-2022-0243	8	kernel-linus	2022-06-29
Mageia	MGASA-2022-0246	8	openssl	2022-06-30
Mageia	MGASA-2022-0245	8	python-bottle	2022-06-30
Mageia	MGASA-2022-0244	8	python-pyjwt	2022-06-30
Mageia	MGASA-2022-0248	8	ruby-git	2022-07-05
Mageia	MGASA-2022-0252	8	ruby-rack	2022-07-05
Mageia	MGASA-2022-0249	8	squid	2022-07-05
Mageia	MGASA-2022-0253	8	thunderbird	2022-07-05
Oracle	ELSA-2022-4588	OL9	.NET 6.0	2022-07-01
Oracle	ELSA-2022-5050	OL9	.NET 6.0	2022-07-01
Oracle	ELSA-2022-5239	OL7	389-ds-base	2022-06-29
Oracle	ELSA-2022-5239	OL7	389-ds-base	2022-06-29
Oracle	ELSA-2022-5326	OL8	compat-openssl10	2022-07-01
Oracle	ELSA-2022-4899	OL9	compat-openssl11	2022-07-01
Oracle	ELSA-2022-4990	OL9	cups	2022-07-01
Oracle	ELSA-2022-5313	OL8	curl	2022-07-01
Oracle	ELSA-2022-5245	OL9	curl	2022-07-01
Oracle	ELSA-2022-5314	OL8	expat	2022-07-01
Oracle	ELSA-2022-5244	OL9	expat	2022-07-01
Oracle	ELSA-2022-5479	OL7	firefox	2022-07-01
Oracle	ELSA-2022-5479	OL7	firefox	2022-07-01
Oracle	ELSA-2022-5469	OL8	firefox	2022-07-01
Oracle	ELSA-2022-4590	OL9	firefox	2022-07-01
Oracle	ELSA-2022-4765	OL9	firefox	2022-07-01
Oracle	ELSA-2022-4873	OL9	firefox	2022-07-01
Oracle	ELSA-2022-5481	OL9	firefox	2022-07-04
Oracle	ELSA-2022-5337	OL8	go-toolset:ol8	2022-07-01
Oracle	ELSA-2022-5099	OL9	grub2,	2022-07-01
Oracle	ELSA-2022-4582	OL9	gzip	2022-07-01
Oracle	ELSA-2022-2137	OL9	java-1.8.0-openjdk	2022-07-01
Oracle	ELSA-2022-1728	OL9	java-11-openjdk	2022-07-01
Oracle	ELSA-2022-1729	OL9	java-17-openjdk	2022-07-01
Oracle	ELSA-2022-5316	OL8	kernel	2022-07-04
Oracle	ELSA-2022-5249	OL9	kernel	2022-07-01
Oracle	ELSA-2022-5252	OL9	libarchive	2022-07-01
Oracle	ELSA-2022-5311	OL8	libgcrypt	2022-07-01
Oracle	ELSA-2022-5331	OL8	libinput	2022-07-01
Oracle	ELSA-2022-5257	OL9	libinput	2022-07-01
Oracle	ELSA-2022-5317	OL8	libxml2	2022-07-01
Oracle	ELSA-2022-5250	OL9	libxml2	2022-07-01
Oracle	ELSA-2022-5251	OL9	pcre2	2022-07-01
Oracle	ELSA-2022-4771	OL9	postgresql	2022-07-01
Oracle	ELSA-2022-5235	OL7	python	2022-07-01
Oracle	ELSA-2022-5235	OL7	python	2022-07-01
Oracle	ELSA-2022-5234	OL7	python-virtualenv	2022-06-29
Oracle	ELSA-2022-5234	OL7	python-virtualenv	2022-06-29
Oracle	ELSA-2022-4592	OL9	rsync	2022-07-01
Oracle	ELSA-2022-4795	OL9	rsyslog	2022-07-01
Oracle	ELSA-2022-5338	OL8	ruby:2.6	2022-07-01
Oracle	ELSA-2022-4591	OL9	subversion	2022-07-01
Oracle	ELSA-2022-5480	OL7	thunderbird	2022-07-01
Oracle	ELSA-2022-5480	OL7	thunderbird	2022-07-01
Oracle	ELSA-2022-5470	OL8	thunderbird	2022-07-01
Oracle	ELSA-2022-4589	OL9	thunderbird	2022-07-01
Oracle	ELSA-2022-4772	OL9	thunderbird	2022-07-01
Oracle	ELSA-2022-4892	OL9	thunderbird	2022-07-01
Oracle	ELSA-2022-5482	OL9	thunderbird	2022-07-04
Oracle	ELSA-2022-5319	OL8	vim	2022-07-01
Oracle	ELSA-2022-5242	OL9	vim	2022-07-01
Oracle	ELSA-2022-4940	OL9	xz	2022-07-01
Oracle	ELSA-2022-4584	OL9	zlib	2022-07-01
Red Hat	RHSA-2022:5326-01	EL8	compat-openssl10	2022-07-01
Red Hat	RHSA-2022:5326-01	EL8	compat-openssl10	2022-07-01
Red Hat	RHSA-2022:5313-01	EL8	curl	2022-07-01
Red Hat	RHSA-2022:5313-01	EL8	curl	2022-07-01
Red Hat	RHSA-2022:5245-01	EL9	curl	2022-07-01
Red Hat	RHSA-2022:5245-01	EL9	curl	2022-07-01
Red Hat	RHSA-2022:5314-01	EL8	expat	2022-07-01
Red Hat	RHSA-2022:5314-01	EL8	expat	2022-07-01
Red Hat	RHSA-2022:5244-01	EL9	expat	2022-07-01
Red Hat	RHSA-2022:5244-01	EL9	expat	2022-07-01
Red Hat	RHSA-2022:5479-01	EL7	firefox	2022-07-01
Red Hat	RHSA-2022:5479-01	EL7	firefox	2022-07-01
Red Hat	RHSA-2022:5469-01	EL8	firefox	2022-07-01
Red Hat	RHSA-2022:5469-01	EL8	firefox	2022-07-01
Red Hat	RHSA-2022:5477-01	EL8.1	firefox	2022-07-01
Red Hat	RHSA-2022:5477-01	EL8.1	firefox	2022-07-01
Red Hat	RHSA-2022:5474-01	EL8.2	firefox	2022-07-01
Red Hat	RHSA-2022:5474-01	EL8.2	firefox	2022-07-01
Red Hat	RHSA-2022:5472-01	EL8.4	firefox	2022-07-01
Red Hat	RHSA-2022:5472-01	EL8.4	firefox	2022-07-01
Red Hat	RHSA-2022:5481-01	EL9	firefox	2022-07-01
Red Hat	RHSA-2022:5481-01	EL9	firefox	2022-07-01
Red Hat	RHSA-2022:5415-01	EL7	go-toolset-1.17 and go-toolset-1.17-golang	2022-06-30
Red Hat	RHSA-2022:5337-01	EL8	go-toolset:rhel8	2022-07-01
Red Hat	RHSA-2022:5337-01	EL8	go-toolset:rhel8	2022-07-01
Red Hat	RHSA-2022:5316-01	EL8	kernel	2022-07-01
Red Hat	RHSA-2022:5316-01	EL8	kernel	2022-07-01
Red Hat	RHSA-2022:5249-01	EL9	kernel	2022-07-01
Red Hat	RHSA-2022:5249-01	EL9	kernel	2022-07-01
Red Hat	RHSA-2022:5476-01	EL8.2	kpatch-patch	2022-07-01
Red Hat	RHSA-2022:5476-01	EL8.2	kpatch-patch	2022-07-01
Red Hat	RHSA-2022:5252-01	EL9	libarchive	2022-07-01
Red Hat	RHSA-2022:5252-01	EL9	libarchive	2022-07-01
Red Hat	RHSA-2022:5311-01	EL8	libgcrypt	2022-07-01
Red Hat	RHSA-2022:5311-01	EL8	libgcrypt	2022-07-01
Red Hat	RHSA-2022:5331-01	EL8	libinput	2022-07-01
Red Hat	RHSA-2022:5331-01	EL8	libinput	2022-07-01
Red Hat	RHSA-2022:5257-01	EL9	libinput	2022-07-01
Red Hat	RHSA-2022:5257-01	EL9	libinput	2022-07-01
Red Hat	RHSA-2022:5317-01	EL8	libxml2	2022-07-01
Red Hat	RHSA-2022:5317-01	EL8	libxml2	2022-07-01
Red Hat	RHSA-2022:5250-01	EL9	libxml2	2022-07-01
Red Hat	RHSA-2022:5250-01	EL9	libxml2	2022-07-01
Red Hat	RHSA-2022:5251-01	EL9	pcre2	2022-07-01
Red Hat	RHSA-2022:5251-01	EL9	pcre2	2022-07-01
Red Hat	RHSA-2022:5467-01	EL8	php:7.4	2022-07-01
Red Hat	RHSA-2022:5467-01	EL8	php:7.4	2022-07-01
Red Hat	RHSA-2022:5471-01	EL8.4	php:7.4	2022-07-01
Red Hat	RHSA-2022:5471-01	EL8.4	php:7.4	2022-07-01
Red Hat	RHSA-2022:5468-01	EL8	php:8.0	2022-07-01
Red Hat	RHSA-2022:5468-01	EL8	php:8.0	2022-07-01
Red Hat	RHSA-2022:5263-01	EL9	qemu-kvm	2022-07-01
Red Hat	RHSA-2022:5263-01	EL9	qemu-kvm	2022-07-01
Red Hat	RHSA-2022:5338-01	EL8	ruby:2.6	2022-07-01
Red Hat	RHSA-2022:5338-01	EL8	ruby:2.6	2022-07-01
Red Hat	RHSA-2022:5480-01	EL7	thunderbird	2022-07-01
Red Hat	RHSA-2022:5480-01	EL7	thunderbird	2022-07-01
Red Hat	RHSA-2022:5470-01	EL8	thunderbird	2022-07-01
Red Hat	RHSA-2022:5470-01	EL8	thunderbird	2022-07-01
Red Hat	RHSA-2022:5478-01	EL8.1	thunderbird	2022-07-01
Red Hat	RHSA-2022:5478-01	EL8.1	thunderbird	2022-07-01
Red Hat	RHSA-2022:5475-01	EL8.2	thunderbird	2022-07-01
Red Hat	RHSA-2022:5475-01	EL8.2	thunderbird	2022-07-01
Red Hat	RHSA-2022:5473-01	EL8.4	thunderbird	2022-07-01
Red Hat	RHSA-2022:5473-01	EL8.4	thunderbird	2022-07-01
Red Hat	RHSA-2022:5482-01	EL9	thunderbird	2022-07-01
Red Hat	RHSA-2022:5482-01	EL9	thunderbird	2022-07-01
Red Hat	RHSA-2022:5319-01	EL8	vim	2022-07-01
Red Hat	RHSA-2022:5319-01	EL8	vim	2022-07-01
Red Hat	RHSA-2022:5242-01	EL9	vim	2022-07-01
Red Hat	RHSA-2022:5242-01	EL9	vim	2022-07-01
Scientific Linux	SLSA-2022:5479-1	SL7	firefox	2022-07-01
Scientific Linux	SLSA-2022:5480-1	SL7	thunderbird	2022-07-01
Slackware	SSA:2022-186-01		openssl	2022-07-05
SUSE	SUSE-SU-2022:2259-1	MP4.2 SLE15 oS15.3 oS15.4	ImageMagick	2022-07-04
SUSE	SUSE-SU-2022:2263-1	SLE12	ImageMagick	2022-07-05
SUSE	openSUSE-SU-2022:10035-1	osB15	chromium	2022-06-29
SUSE	openSUSE-SU-2022:10036-1	osB15	chromium	2022-06-29
SUSE	SUSE-SU-2022:2273-1	MP4.1 SLE15 SES7	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2267-1	MP4.2 SLE15 oS15.3	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2274-1	OS8 SLE12	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2265-1	OS9 SLE12	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2271-1	SLE12	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2266-1	SLE15	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2272-1	SLE15 SES6 oS15.3 oS15.4	dpdk	2022-07-05
SUSE	SUSE-SU-2022:2277-1	SLE15 oS15.4	haproxy	2022-07-06
SUSE	SUSE-SU-2022:2275-1	SLE15 SES6 oS15.4	php7	2022-07-05
SUSE	openSUSE-SU-2022:10040-1	osB15	python-nltk	2022-07-03
SUSE	SUSE-SU-2022:2260-1	SLE15 oS15.4	qemu	2022-07-04
SUSE	SUSE-SU-2022:2253-1	SLE15	salt	2022-07-04
SUSE	SUSE-SU-2020:0026-2	OS8 OS9 SLE12	sysstat	2022-07-04
Ubuntu	USN-5496-1	18.04 20.04 21.10 22.04	cloud-init	2022-06-29
Ubuntu	USN-5499-1	14.04 16.04	curl	2022-06-30
Ubuntu	USN-5503-1	18.04 20.04 21.10 22.04	gnupg2	2022-07-05
Ubuntu	USN-5497-1	14.04	libjpeg6b	2022-06-30
Ubuntu	USN-5500-1	16.04	linux, linux-aws	2022-07-01
Ubuntu	USN-5493-2	20.04	linux-hwe-5.13	2022-07-01
Ubuntu	USN-5485-2	20.04	linux-oem-5.14	2022-07-01
Ubuntu	USN-5502-1	18.04 20.04 21.10 22.04	openssl	2022-07-05
Ubuntu	USN-5479-2	16.04	php7.0	2022-07-04
Ubuntu	USN-5498-1	16.04	vim	2022-06-30

Full Story (comments: none)