Welcome to the LWN.net Weekly Edition for October 14, 2021

This edition contains the following feature content:

• Scrutinizing bugs found by syzbot: many "low-risk" bugs turn out to be riskier than previously thought.
• A rough start for ksmbd: the new in-kernel SMB file server has potential, but it has a number of code-quality problems to solve first.
• Pulling slabs out of struct page: a proposal that foreshadows the direction of memory-management development.
• The intersection of modules, GKI, and rocket science: when is making drivers modular a bad thing?
• A QEMU case study in grappling with software complexity: how to recognize and deal with unnecessary complexity.
• Digging into Julia's package system: a look at how Julia manages packages.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Scrutinizing bugs found by syzbot

The syzbot kernel-fuzzing system finds an enormous number of bugs, but, since many of them may seem to be of a relatively low severity, they have a lower priority when contending for the attention of developers. A talk at the recent Linux Security Summit North America reported on some research that dug further into the bugs that syzbot has found; the results are rather worrisome. Rather than a pile of difficult- or impossible-to-exploit bugs, there are numerous, more serious problems lurking within.

SyzScope

The first speaker on day one of the summit was Xiaochen Zou, a PhD student at the University of California, Riverside, who described a tool that he and his colleagues have been running on bugs that syzbot—which uses the syzkaller coverage-guided fuzzer—has reported in the Linux kernel. Over the last four years, syzbot has reported around 4000 bugs, of which 3000 have been fixed. There are eight categories that these bugs mainly fall into, but only some of those categories represent security bugs, he said. These are classic, severe kernel security holes, such as use after free, double free, and out-of-bounds writes; they are mostly found by the Kernel Address Sanitizer (KASAN) as part of syzkaller runs.

There are several other classes of bugs that need fixing, but they tend to get less attention; these include kernel or sanitizer assertions and general protection faults (e.g. invalid pointer dereference). In addition, bugs that allow memory to be read can provide information leaks, such as kernel addresses, but those are generally not needed for exploits to compromise the kernel, Zou said. The researchers classified these as low-risk bugs. Bugs that are in lower-risk categories take longer to fix on average, and longer to backport to various older kernels. The sheer number of bugs that syzbot reports overwhelms the ability of the kernel community to fix them everywhere they occur.

If the bugs that are slower to be fixed are truly of low risk, that's less of a problem, but the research found that many of these bugs are not actually low risk. Beyond that, the syzbot reports do not always tell the full story of the bugs' impacts. Even worse still, there are ways to automatically find higher-risk impacts from some of these low-risk bugs. SyzScope is a tool that sets out to reveal high-risk impacts of bugs that are reported in a low-risk category. SyzScope is not meant to automatically create full exploits because there are already other research projects that are solving that part of the problem. In particular, FUZE focuses on doing so for use-after-free vulnerabilities, and KOOBE can be used for out-of-bounds-write vulnerabilities.

There were several insights that the researchers had which led to the development of SyzScope, Zou said. Syzbot only shows the first problem it finds in an execution path that it tests; it normally stops and reports the bug. That makes sense, because syzbot is looking for bugs, not necessarily security-related bugs. But it may ignore further, more risky impacts downstream from the bug it is reporting, which may downplay the risks of the bug, thus possibly delay it from being fixed promptly.

Even if you allow fuzzing to continue, though, there are several types of impacts that it will not be able to find, he said. Control-flow hijacking and the writing of values or addresses (either arbitrarily or constrained in some way) are not detected by the sanitizers, kernel assertions, or other mechanisms used by syzkaller.

He gave an example of an out-of-bounds read that KASAN detected and syzbot reported; he showed that it could be turned into a write of null to an arbitrary address. But, because it relies on specific values in out-of-bounds memory, fuzzing will not be able to find it; only if the out-of-bounds memory is specially prepared will it be useful for this purpose.

Heap spraying is a technique that an attacker could use to arrange memory in that fashion. Beyond just writing null, though, even further downstream from the syzbot-related bug is a function call made to an address that comes from the out-of-bounds memory, which can be controlled by an attacker. That leads to control-flow hijacking and a full compromise of the kernel.

But those further impacts were found by human experts, not by syzkaller or some other tool. The researchers wanted a way to discover those types of problems automatically. Fuzzers cannot control the contents of the objects that are used after they are freed or the contents of out-of-bounds memory like an attacker can.

Symbolic execution

There is a technique that can do that kind of work, however: symbolic execution. By using symbolic values rather than concrete values during program analysis, symbolic execution can effectively simulate the results of heap spraying. It can show what values need to be stored, and where to place them, to cause the function to be called at a location of the attacker's choosing, for example.

There are two modes of operation for SyzScope, both of which search for high-risk bugs lurking behind low-risk reports. In one mode, it uses static analysis and symbolic execution on open bugs reported by syzbot. In the other, it looks at bugs that have been fixed, using fuzzing, static analysis, and symbolic execution. The latter incorporates fuzzing to find downstream effects of the original bug; it uses the bug-fix patch to validate that new bugs found are consequences of the original.

In the interests of time, Zou said he would skip talking about the static-analysis piece; it is optional and simply assists the symbolic-execution step. It is, in effect, an optimization.

A restricted form of fuzzing is used to try to find additional buggy contexts that share the same root cause as the original bug; those are then used by the symbolic-execution step to see if they are actually high-risk bugs. Unlike syzbot, this fuzzing, which also uses syzkaller, does not stop when it finds a bug. Instead, it continues to run to see what else it can find. The fuzzer starts with the proof-of-concept code that syzkaller produced to demonstrate the original problem and then uses a "conservative mutating strategy" to try to find use-after-free bugs, out-of-bounds writes, and the like. It uses the bug-fix patch to determine which new contexts are related to the original fixed bug; if those new contexts do not trigger a complaint from the patched kernel, they correspond to high-risk bugs hiding underneath the report that seemed to be of a low-risk bug type.

The fuzzer augments syzkaller's code-coverage feedback with impact feedback. It tries to find bugs that have a higher impact in order to find high-risk bugs. The code-coverage feedback sometimes leads the fuzzer into finding entirely new bugs, which is not the focus of their efforts.

The symbolic-execution phase uses QEMU; it sets a breakpoint at the place where KASAN reported a problem and triggers the bug. It then launches "angr", which is the symbolic-execution engine; angr finds the out-of-bounds memory addresses from the KASAN report and "symbolizes" them. The register values in QEMU are transferred to angr, which can also retrieve memory contents dynamically from QEMU.

The engine looks for certain kinds of assembly language instructions, that operate on values from the range of out-of-bounds memory reported by KASAN. This allows it to detect ways that various types of flaws can be exploited. For example, if the value passed to kfree() comes from within the controllable memory, it is flagged as an invalid free; similarly, a call to an address that comes from out-of-bounds memory is a way to produce a control-flow-hijacking exploit. Five different types of high-risk impacts are detected in this way.

Results

The researchers ran an experiment using nearly 1200 lower-risk bugs that syzbot reported. They ran each with three hours of fuzzing and four hours of symbolic execution. Overall, the experiment was able to find high-risk impacts for 147 low-risk bugs. A lot of those low-risk bugs had multiple other impacts, many of them from high-risk categories, associated with them. For example, 51 control-flow-hijacking impacts were found, both from fixed and open bugs that syzbot had reported in lower-risk categories. Around 3200 separate impacts beyond what syzbot reported were found.

Just the fuzzing discovered around half of the high-risk bugs, with roughly 5% of the total impacts found. As noted earlier, fuzzing can only find impacts that do not require control of the out-of-bounds memory contents. For bugs that had already been fixed, thus could use the fuzzing stage, there were an average of nearly 28 further impacts per kernel bug tested; an average of roughly 17 additional impacts were found from the open bugs tested.

Symbolic execution detected the other half of the high-risk bugs found in the experiment, including all 34 that were found from bugs that were reported by syzbot but still unfixed in the kernel. It also found the vast majority (95%) of the impacts, including all of super-high-risk impacts, like control flow hijacking and arbitrary address writes.

The researchers submitted 32 high-risk bugs to the CVE maintainers, eight of which were assigned CVE numbers (which can be seen in slide 28 of Zou's slides). Their paper will be presented at the 31st USENIX Security Symposium in August 2022; it should be available soon, Zou said.

The voluminous output from syzbot and the inability of kernel developers to keep up with the onslaught has been a frequent cause for concern over the years; these findings can only heighten those fears. It would not be surprising to find out that black hats and governmental agencies are using similar techniques to turn seemingly semi-innocuous bugs into ways to fully compromise the kernel—in fact, it would be surprising if they were not. To a certain extent, these findings show that prioritizing bugs based on the impact reported by syzbot probably frequently downplays the danger, but they do tend to validate the "all bugs are security bugs" attitude that prevails within parts of the kernel-development community.

Comments (5 posted)

A rough start for ksmbd

Among the many new features pulled into the mainline during the 5.15 merge window is the ksmbd network filesystem server. Ksmbd implements the SMB protocol (also known as CIFS, though that name has gone out of favor) that is heavily used in the Windows world. The creation of an in-kernel SMB server is a bit surprising, given that Linux has benefited greatly from the user-space Samba solution since shortly after the beginning. There are reasons for this move but, in the short term at least, they risk being overshadowed by a worrisome stream of security-related problems in ksmbd.

Why create an in-kernel SMB server at this point? In a sense, ksmbd is not meant to compete with Samba; indeed, it has been developed in cooperation with the Samba project. It is, however, meant to be a more performant and focused solution than Samba is; at this point, Samba includes a great deal of functionality beyond simple file serving. Ksmbd claims significant performance improvements on a wide range of benchmarks; the graphs on this page show a doubling of performance on some tests. An in-kernel server is an easier place to support variants like SMB Direct, which uses RDMA to transfer data between systems. By drawing more eyes to the code, merging into the mainline may also facilitate faster development in general. One other reason — which tends to be spoken rather more quietly — is that a new implementation can be licensed under GPLv2, while Samba is GPLv3.

Ksmbd was first posted for review (as "cifsd") by Namjae Jeon in late March; the eighth revision came out just before the opening of the 5.15 merge window in late August. The last version received no review comments, but previous versions had clearly been looked at by a number of developers. Nobody objected when Steve French asked Linus Torvalds to pull ksmbd into the mainline on August 29.

It is not unusual for a new subsystem to receive a lot of fixes after its entry into the mainline kernel. Merging tends to draw a lot more attention to the code, and the number of testers normally goes up, leading to the discovery of more problems. That is what the stabilization period after the merge window is for, after all. That said, the nature of the fixes being applied can give some insight into the quality of the underlying code, and the indications for ksmbd are not entirely good.

The commit history for ksmbd shows a steady stream of fixes, as expected. Worryingly, though, many of the problems being fixed are clearly security issues — not a good thing in a network filesystem implementation. Examples include:

• The code to change ownership and permissions did not check existing file permissions first.
• Failure to validate data lengths could lead to access to invalid data.
• The server would blindly follow symbolic links during pathname lookup.
• Numerous failures to validate buffer lengths, such as this one or this one.

All of those fixes were applied after ksmbd landed in the mainline; there are others that came before. Currently, twelve fixes to ksmbd credit Coverity scans in their changelogs.

Again, it would not be surprising for a security issue or three to turn up in a new network-filesystem implementation. But ksmbd has shown enough problems to have raised a few eyebrows in the kernel community, though the discussion of those problems was evidently held in private for some time. When French pushed another set of ksmbd fixes in mid-September, though, Kees Cook took the discussion public:

I was looking through the history[1] of the ksmbd work, and I'm kind of surprised at some of the flaws being found here. This looks like new code being written, too, I think (I found[0])? Some of these flaws are pretty foundational filesystem security properties[2] that weren't being tested for, besides the upsetting case of having buffer overflows[3] in an in-kernel filesystem server.

I'm concerned about code quality here, and I think something needs to change about the review and testing processes.

French replied that he was surprised by some of the problems too. He pointed to a wiki page describing the ongoing security review for this code, which seems to have acquired a new urgency. A number of new procedures are being instituted, he said, and there will be testing done at various interoperability events. French said he was "pleased with the progress that is being made, but also conceded that ksmbd "is not ready for production use yet".

There are also some things to look forward to on the security front, he continued:

There is some good news (relating to security), once Namjae et al get past these buffer overflow etc. patches.

• he has already implemented the strongest encryption supported in SMB3.1.1
• he has implemented the man in the middle attack prevention features of the protocol
• strong (Kerberos) authentication is implemented
• he has removed support for weak older dialects (including SMB1 and SMB2) of the protocol
• he will be removing support for weaker authentication (including NTLMv1)

The NTLMv1 removal has since been merged into the mainline. On reading French's message, Cook responded: "Thanks for making these recent changes; I feel much better about ksmbd's direction".

The work on cleaning up ksmbd proceeds; French pushed another 11 fixes on October 1.

At this point, there is little doubt that ksmbd will be properly reviewed and cleaned up; there are eyes on the code, and ksmbd itself is small enough that a comprehensive review should be feasible. At that point, the kernel should have an SMB implementation that is feature-rich, performant, and secure. That said, waiting another kernel development cycle or two for the developers to "get past these buffer overflow etc. patches" before deploying it might well be prudent.

This is all good, but it is still a little worrisome that this code got as far as it did in the condition it was in. It seems clear that security concerns were not at the forefront when this code was being developed and that the review it received before being merged failed in this regard as well. The addition of security features is great, but they do not help much in the absence of a secure implementation. If we ever want to reach a point where we are not adding more security problems to the kernel than we are fixing, we will need to do better than this.

Comments (54 posted)

Pulling slabs out of struct page

For the time being, the effort to add the folio concept to the memory-management subsystem appears to be stalled, but appearances can be deceiving. The numerous folio discussions have produced a number of points of consensus, though; one of those is that far too much of the kernel has to work with page structures to get its job done. As an example of how a subsystem might be weaned off of struct page usage, Matthew Wilcox has split out the slab allocators in a 62-part patch set. The result may be a foreshadowing of changes to come in the memory-management subsystem.

The kernel maintains one page structure for every physical page of memory that it manages. On a typical system with a 4KB page size, that means managing millions of those structures. A page structure tells the kernel about the state of the page it refers to: how it is being used, how many references to it exist, its position in various queues, and more. The required information varies depending on how any given page is being used at the moment; to accommodate this, struct page is a complicated mess of structures and unions. The current definition of struct page makes for good pre-Halloween reading, but those who truly want a good scare may want to see what it looked like before Wilcox cleaned things up for 4.18.

One of the users of struct page is the set of slab allocators, which obtain blocks of pages ("slabs") from the kernel and hand them out in smaller, fixed-size chunks. They are used heavily, and their performance will affect the performance of the system as a whole, so it is not surprising that they reach into the memory-management subsystem at the lowest levels. To support this usage, many of the fields in struct page are there specifically for the slab allocators. Just to complicate things, the kernel has three slab allocators: SLAB (the original allocator, often used by Android), SLUB (often used for desktop and data-center systems), and SLOB (a tiny allocator intended for embedded systems). Each has its own needs for fields in struct page.

Wilcox's patch set creates a new struct slab by removing the relevant fields from struct page. The new structure is, in its entirety:

    struct slab {
	unsigned long flags;
	union {
	    struct list_head slab_list;
	    struct {	/* Partial pages */
	        struct slab *next;
    #ifdef CONFIG_64BIT
	        int slabs;	/* Nr of slabs left */
	        int pobjects;	/* Approximate count */
    #else
	        short int slabs;
	        short int pobjects;
    #endif
	    };
	    struct rcu_head rcu_head;
	};
	struct kmem_cache *slab_cache; /* not slob */
	/* Double-word boundary */
	void *freelist;		/* first free object */
	union {
	    void *s_mem;	/* slab: first object */
	    unsigned long counters;	/* SLUB */
	    struct {			/* SLUB */
	        unsigned inuse:16;
	        unsigned objects:15;
	        unsigned frozen:1;
	    };
	};

	union {
	    unsigned int active;	/* SLAB */
	    int units;			/* SLOB */
	};
	atomic_t _refcount;
    #ifdef CONFIG_MEMCG
	unsigned long memcg_data;
    #endif
    };

As can be seen, this structure still relies heavily on unions to overlay the information that each allocator needs to store with the page. Those could be eliminated by splitting the structure into three allocator-specific variants, but that would add complication to a patch set that is already large (and set to grow).

It is worth noting that struct slab is really struct page in disguise; instances of struct slab overlay the corresponding page structure in the kernel's memory map. It is, essentially, the kernel's view of struct page for pages that are owned by a slab allocator, extricated from its coexistence with all of the other views of that structure. This means that struct slab must be laid out with care; some fields (_refcount, for example) are shared with struct page, and the results of a disagreement over its location would be unfortunate. To ensure that no such calamity occurs, Wilcox has included a set of compile-time tests verifying the offsets of the shared fields.

After that, the remaining patches in the series convert various code in the slab allocators (and beyond) to use the new type. The SLUB conversion is done meticulously, in over 40 small steps. Wilcox described the conversion of the other allocators as "slapdash", done in a single patch each. Presumably a later version of the patch set will turn these proof-of-concept patches into a proper series of their own, but it's not entirely clear who will do that; Wilcox wrote in the cover letter:

I don't know the slab allocators terribly well, so I would be very grateful if one of the slab maintainers took over this effort. This is kind of a distraction from what I'm really trying to accomplish with folios, although it has found one minor bug.

As of this writing, no slab maintainers (who tend to be thin on the ground in the best of times) have responded to this request.

This might seem like a lot of work to put an old structure into a new form, but there are a number of reasons to want something like this. Just pulling the slab-specific fields out of struct page simplifies that structure significantly. Using a separate type makes it clear which variant of the page structure the code expects to deal with, and it adds a degree of type safety; it is no longer possible to accidentally access the wrong union fields.

But the biggest benefit comes simply from beginning to separate the slab allocators from struct page. Eventually it may become possible to disassociate struct slab from struct page entirely and allocate it dynamically. That would be one small step toward encapsulating struct page within the core memory-management code and hiding it from the rest of the kernel, a change that would ease the much-needed task of improving page-level memory management.

First, though, some variant of this work must make it into the mainline kernel. It should be an easier process than getting folios merged, but getting big changes into the memory-management code is never easy. The relative silence that has greeted this work so far might be a bit worrisome, especially since Wilcox has requested help, but it is the early days for this series still. Regardless of how struct slab fares, though, it provides an indication of the direction that the memory-management developers are trying to go.

Comments (32 posted)

The intersection of modules, GKI, and rocket science

One does not normally expect a lot of controversy around a patch series that makes changes to platform-specific configurations and drivers. The furor over some work on the Samsung Exynos platform may thus be surprising. When one looks into the discussion, things become more clear; it mostly has to do with disagreements over the best ways to get hardware vendors to cooperate with the kernel development community.

In mid-September, Will McVicker posted a brief series of changes for the Exynos configuration files; one week later, a larger, updated series followed. The purpose in both cases was to change a number of low-level system-on-chip (SoC) drivers to allow them to be built as loadable modules. That would seem like an uncontroversial change; it is normally expected that device drivers will be modular. But the situation is a little different for these low-level SoC drivers.

Generic kernels and essential drivers

In the distant past, kernels for Arm-based SoCs were specific to the target platform. While x86 kernels normally run on all x86 processors, Arm kernels were built for a small range of target platforms and would not boot on anything else. Over the years, the Arm developers have worked to make the 64-bit Arm kernel sufficiently generic that a single binary image can boot on a wide range of platforms. To a great extent, this portability has been achieved by building drivers as modules so that a kernel running on a given device need only load the drivers that are relevant to that device.

There is a bootstrapping problem to solve, though; before the kernel can load a single module, it must be able to boot to a point where it has the platform in a known, stable state and is able to mount a RAM-based root filesystem. That can only happen if the drivers needed to boot that far are built into the kernel itself. Thus, the generic kernel contains a long list of platform-specific drivers to configure clocks, pin controllers, and more; without them, the kernel would never boot. The maintainers' policy has long stated that any drivers which are essential for the boot process must be built into the kernel itself.

McVicker's patch set takes a number of these essential drivers and, for reasons to be discussed in detail below, removes them from the kernel image, making them into loadable modules instead. Ostensibly, this change does not violate the policy for these drivers, but only if it can be demonstrated that the drivers are, in fact, not essential for the kernel to boot on the affected hardware. Therein lies the first big problem for this patch set: McVicker made it clear in the cover letter that these changes had not actually been tested on the appropriate hardware. While he is optimistic that the systems should still boot with modular drivers, nobody has yet proved that.

Optimism only gets one so far in the kernel community. This lack of testing has caused Exynos platform maintainer Krzysztof Kozlowski to repeatedly push back on the patches; until he is sure that all Exynos systems can boot with those drivers as modules, he is unwilling to take the changes. He has offered to help with some of the needed testing. Meanwhile, Arnd Bergmann has backed up Kozlowski's reticence:

The "correctness-first" principle is not up for negotiation. If you are uncomfortable with the code or the amount of testing because you think it breaks something, you should reject the patches. Moving core platform functionality is fundamentally hard and it can go wrong in all possible ways where it used to work by accident because the init order was fixed.

The lack of testing seems like the kind of problem that should be amenable to a solution. Reaching the needed level of confidence may take a while, though. Some systems running a given SoC may boot without a specific clock driver (say) because the firmware initializes the clocks to a reasonable configuration at power-on. Counting on all firmware to have its act together in this way can be a risky endeavor, though. Even so, this testing, which should have been done before the patches were ever submitted, should be possible to fill in after the fact.

Out-of-tree code

There is still the question of why one would want to make this possibly risky change. The obvious benefit is making the core kernel image smaller; this is especially appreciated on all of the platforms that don't use the drivers in question and thus see them as dead weight. But there is another motivation here that relates to a different kernel, also called "generic".

The kernels shipped on Android devices have notoriously contained vast amounts of out-of-tree code, to the point that such code sometimes outweighs the mainline code that is in use on the device. This has led to problems throughout the ecosystem, including a lack of cooperation with upstream kernel developers, the fragmentation of the Android kernel space, the inability to apply security updates when the vendor inevitably stops doing so, and the cost of maintaining all of those kernels. To address this problem, Google has been pushing vendors of Android-based devices toward its "generic kernel image" (GKI), which is a core kernel that must be shipped by all Android devices. Vendors are able to supply their own modules to load into that kernel, but they cannot replace the kernel itself.

This policy brings a number of benefits. Vendor changes are restricted to what can be done with the module API, and Google has been pushing to restrict that somewhat as well. The days of vendors replacing the CPU scheduler should be done now. Vendors, naturally, chafe at these restrictions, but they have little alternative to compliance. If they choose to run their own system, even if it is an Android fork, they lose access to many of the Google apps and services that make Android useful for their customers.

Code that is built into the GKI kernel thus cannot be changed by device vendors. Code that is loaded from modules, instead, can be shoved aside and replaced. Viewed in this light, the desire to modularize built-in drivers becomes rather easier to understand. Even so, there are two different aspects of this situation that are worth examining. One is that vendors want to ship out-of-tree modules on their devices rather than upstreaming their drivers to hide their secret magic from competitors. As Lee Jones described it:

In order for vendors to work more closely with upstream, they need the ability to over-ride a *few* drivers to supplement them with some functionality which they believe provides them with a competitive edge (I think you called this "value-add" before) prior to the release of a device. This is a requirement that cannot be worked around.

As one might imagine, this position is seen as less than fully compelling by much of the kernel development community. It is also not entirely convincing; as Tomasz Figa put it:

Generally, the subsystems being mentioned here are so basic (clock, pinctrl, rtc), that I really can't imagine what kind of rocket science one might want to hide for competitive reasons.

Jones tried to de-emphasize this point of discussion later on, but it was a bit late; he had said (part of) the quiet part out loud.

The other piece of the puzzle is simpler to understand. Even if a set of clock drivers contains no real secrets of interest, the vendor may simply lack the desire to make the effort to get the drivers upstream. It is possible to get out-of-tree drivers built into the GKI, but Google would clearly rather not deal with that anymore, so there is a continual pressure to get drivers into the mainline. If the drivers can be supplied directly by the vendor as a module, instead, they disappear from the GKI and that pressure vanishes. With regard to the Exynos changes, a lack of desire to work upstream seems like a plausible explanation; as Kozlowski pointed out in the above-linked message, Samsung has only contributed a single change to the Exynos subsystem since 2017.

Jones has tried to characterize vendors' upstream reticence as temporary, saying "vendors are not able to upstream all functionality right away". Later, though, he said:

But [they have] no incentive to upstream code [for] old (dead) platforms that they no longer make money from. We're not talking about kind-hearted individuals here. These are business entities.

If neither new or old code can be upstreamed, then it would appear that mainline support for these platforms is at a dead end.

Better or worse?

The natural reaction for many kernel developers is to make life harder for vendors that are seemingly looking for ways to avoid engaging with the development community. That would include rejecting the patch set under consideration here. Olof Johansson's opinion was:

This patchset shouldn't go in.

GKI is a fantastic effort, since it finally seems like Google has the backbone to put pressure on the vendors to upstream all their stuff.

This patch set dilutes and undermines all of that by opening up a truck-size loophole, reducing the impact of GKI, and overall removes leverage to get vendors to do the right thing.

McVicker, instead, argued that modularizing these drivers is a way to bring vendors closer to upstream and will improve the situation overall:

We believe that if we make it easier for SoC vendors to directly use the upstream kernel during bring-up and during the development stages of their project, then that will decrease the friction of working with upstream (less downstream changes) and increase the upstream contributions.

Which of these positions is closer to the truth is hard to say; each may hold water with respect to some vendors while falling down with others. Getting vendors to engage with upstream is a constant process requiring judicious use of both carrots and sticks.

That said, the outcome of this particular discussion is not in a great deal of doubt. Making life easier for uncooperative vendors is usually not, on its own, sufficient reason to keep a patch set out of the kernel. Bergmann described it well in the above-linked message:

I understand that it would be convenient for SoC vendors to never have to upstream their platform code again, and that Android would benefit from this in the short run.

From my upstream perspective, this is absolutely a non-goal. If it becomes easier as a side-effect of making the kernel more modular, that's fine.

So, in a sense, much of the discussion was irrelevant; if the patches can be shown to work properly (which has not yet happened), then they are consistent with many of the community's long-term goals and will likely find their way into the mainline sooner or later. Whether that will encourage vendors to work upstream or, instead, make it easier for them to stay away remains to be seen. But problems with uncooperative vendors have existed for as long as the Linux kernel has; they will not go away regardless of what happens here.

Comments (28 posted)

A QEMU case study in grappling with software complexity

There are many barriers to producing software that is reliable and maintainable over the long term. One of those is software complexity. At the recently concluded 2021 KVM Forum, Paolo Bonzini explored this topic, using QEMU, the open source emulator and virtualizer, as a case study. Drawing on his experience as a maintainer of several QEMU subsystems, he made some concrete suggestions on how to defend against undesirable complexity. Bonzini used QEMU as a running example throughout the talk, hoping to make it easier for future contributors to modify QEMU. However, the lessons he shared are equally applicable to many other projects.

Why is software complexity even a problem? For one, unsurprisingly, it leads to bugs of all kinds, including security flaws. Code review becomes harder for complex software; it also makes contributing to and maintaining the project more painful. Obviously, none of these are desirable.

The question that Bonzini aimed to answer is "to what extent can we eliminate complexity?"; to do that he started by distinguishing between "essential" and "accidental" complexity. The notion of these two types of complexity originates from the classic 1987 Fred Brooks paper, "No Silver Bullet". Brooks himself is looking back to Aristotle's notion of essence and accident.

Essential complexity, as Bonzini put it, is "a property of the problem that a software program is trying to solve". Accidental complexity, instead, is "a property of the program that is solving the problem at hand" (i.e. the difficulties are not inherent to the problem being solved). To explain the concepts further, he identified the problems that QEMU is solving, which constitute the essential complexity of QEMU.

Essence and accidents of QEMU

QEMU has a large set of requirements in terms of portability, configurability, performance, and security. Besides emulating guest devices and providing ways to save and restore their state, it has a powerful storage layer and also embeds a few network servers, such as a VNC server. QEMU also has to make sure that the CPU and device models exposed to the guest remain stable, regardless of whether the underlying hardware or QEMU itself are updated. For many users it's important to use QEMU with a distribution kernel rather than a custom-built kernel. Being able to boot non-Linux operating systems is a necessary feature for many QEMU users, as well; it counts as essential complexity.

QEMU provides a management interface, usually called the monitor. In fact, there are two, HMP (human monitor protocol) and QMP (QEMU monitor protocol), because users need an easy way to interact with the monitor and won't be served by the same JSON-based interface provided by QMP that external programs use to manage QEMU. Therefore, QEMU contains an object model and a code generator that handles the marshaling and unmarshaling of C structures. Thanks to this code generator, the same code can easily operate on either JSON dictionaries or command-line options.

Developers also see another face of complexity, which is brought in by the tools that are part of the build process. Tools make common tasks easier, but they also make debugging harder when they break. For example, QEMU once had a manual configuration mechanism that required listing all of the devices, one by one, in the board that it is emulating. These days, only the board needs to be specified, and the build system will automatically enable the devices that are supported by it. It also ensures that impossible configurations don't build — this is useful, but, of course, developers have to learn how to deal with those failures.

Sources of complexity

For the sake of his presentation, Bonzini identified two main sources of accidental complexity. The first is "incomplete transitions" (inspired by a paper on GCC maintenance), which occur when a new and better way to do something is introduced, but it is not applied consistently across the codebase. This can be due to any number of reasons: the developer might not have the time or relevant expertise; or they simply fail to discover the remaining occurrences.

As an example, he cited two contrasting ways to report errors in QEMU: the propagation-based API, and ad hoc functions (e.g. error_report()) that write errors to the standard output. The propagation-based API was introduced to report errors to the QMP interface. It has two advantages: it separates the point where the errors happen versus where they're reported, and allows for graceful error recovery. Another example of an incomplete transition is that, even though these days the QEMU build system mostly uses Meson, there are preexisting compilation tests that are written in the Bourne shell and are part of QEMU's configure script.

However, QEMU also has a decent track-record of completing transitions. Several of these were done using Coccinelle — a pattern-matching and source transformation tool that allows the creation of a "semantic patch" that it can apply uniformly across the codebase. For instance, Coccinelle was used to replace obsolete APIs, to simplify code that was going through unnecessary hoops, or even to introduce whole new APIs (as was the case for the creation and "realization" of devices).

The second source of accidental complexity is duplicated logic and missing abstractions. There is a trade-off between writing code that is ad hoc, or designing reusable data structures and APIs. Bonzini pointed to the command-line parsing and contrasted ad hoc code, using functions such as strtol() or sscanf(), to QEMU-specific APIs such as QemuOpts or keyval. The latter ensure a level of consistency in the command line, and sometimes take care of printing help messages.

Another example is the recent effort to organize more parts of QEMU into shared objects that can be installed separately. As the number of such modules grew, a new mechanism was put in place to list a module's provided functionality and its dependencies in the same source file as the implementation, rather than having them scattered around the QEMU source code. As soon as a reviewer notices excessive duplication, or functionality scattered across many files, they should make a plan on how to eliminate that, Bonzini suggested.

Complexity on the QEMU command line

The talk proceeded with a case study of accidental complexity in QEMU, namely the command-line processing code. QEMU has 117 options, implemented in approximately 3000 lines of code that has "some essential complexity, but way too much accidental complexity". Bonzini outlined ways to simplify things, or how not to make them worse when working on QEMU's command-line parsing code. He began by asking: what exactly is causing accidental complexity in QEMU's command-line options? The many options vary a lot in their implementation, so the talk grouped them into six categories, and went through them in order of increasing accidental complexity: flexible, command, combo, shortcut, one-off, and legacy.

Flexible options are the most complicated, since they cater to a wide range of needs. They provide access to large parts of QEMU's essential complexity, and new features in QEMU are usually enabled through these options. Flexible options work by delegating as much as possible to generic QEMU APIs, so that enabling new features does not require writing or modifying any command-line parsing code. This is how a single option, ‑object, can configure secrets such as encryption keys, certificates for TLS, the association of virtual machines to NUMA nodes on the host, and so on. Three options, ‑cpu, ‑device, and ‑machine, configure almost all aspects of the virtual hardware. However, these options are not immune to accidental complexity: there are at least four parsers for such options: QemuOpts, keyval, a JSON parser, and a bespoke parser that is used by the ‑cpu option. "Four parsers are at least two more than there should be."

A command option is specified on the QEMU command line, but it also typically corresponds to one of the QMP commands that can be invoked at run time. An example is the option to not start the vCPU at guest boot up (qemu-kvm -S on the command line; or stop at run time), but start it only when asked to do so (via QMP cont, for "continue"). Another example is ‑loadvm to start QEMU from a file with guest state saved in it; or trace to enable trace points (this assumes QEMU is built with one of the available tracing backends). These options put a relatively small burden on the QEMU maintainer; but Bonzini suggested keeping a high bar for adding new command-line options — it's easier to invoke the options from the the QMP interface at run time.

With combo options, "we start our descent into accidental complexity hell": these options create both the frontend and backend of a device in a single command-line option. For example, QEMU's ‑drive option creates a device such as virtio-blk and a disk image for the guest in a single option. The more verbose variants of the options are unwieldy enough for casual users that the combo options do serve a genuine use case, but they have a high maintenance burden. The parsing code is complex and the options also tend to have ramifications in the rest of the code — both the backend code and the virtual-chipset creation code. These options make QEMU's code less modular, so that one cannot add support for a new board without knowing some details about the command line.

Shortcut options are syntactic sugar for the previous three groups. For example, ‑kernel path is short for ‑machine pc,kernel=path. They are handy — many users may not even realize that the longer forms exist — and they have a small maintenance burden because their implementation lives entirely within the command-line parsing code. However, given the sheer number of options that already exist, it's better to not add more.

Then there are one-off options; these are essential but their implementation is often suboptimal. Typically, they write a value to a global variable, or call a function that is not available via the QEMU monitor at run time. Bonzini pleaded with developers to avoid creating new ones and instead to refactor the existing ones into shortcut or command options, which he has been doing on and off over the past year.

Finally, with the legacy command-line options, "we hit rock bottom". Many of them are failed experiments (e.g. the ‑readconfig and ‑writeconfig options) or things that should not be in QEMU at all. For example, instead of ‑daemonize that daemonizes the QEMU process after initialization, users are better-served by tools such as libvirt. The way forward for these is to deprecate and ultimately remove them.

Ways to fight back

What lessons does the QEMU command line teach and what guidance can a developer derive? "Do not design in a void", he said — exploit the existing essential complexity. Before embarking on adding a new command-line flag, ask yourself if it is necessary. Perhaps one of the existing integrations in QEMU such as the QEMU API and QMP commands could be used. This way, one can make the most of the existing interactions between QEMU's subsystems.

Second, Bonzini highlighted the responsibilities of patch reviewers: understand the essential part of the complexity, and do not mistake it for an accident — this is a prerequisite to identify rising accidental complexity. And don't let the accidental complexity take over the project. For those working on refactoring large codebases, he encouraged learning Coccinelle.

Incomplete transitions are not always to be feared: transitioning from an old API to a new and better API is a natural part of how software is improved. In QEMU's case, sometimes a new feature requires a transition period anyway, because it affects the command line or a management tool, and thus requires a deprecation cycle. In such cases, take advantage of the incomplete transition, and work in phases. Identify the smallest chunks of work that can be considered an improvement, and plan for what comes later.

Further, ensure that the new and recommended way to perform a development task, or using a feature is documented — "there should be one obvious way to do a task. If not, one documented way to do it." Incomplete or piece-wise transitions should not deter one from making improvements to a program. Evaluate the trade-offs between duplicating code and adding more abstractions. Some situations may warrant code duplication; but when things are turning for the worse, do not aggravate the situation.

Conclusion

Building essentially-complex and maintainable software is hard enough as it is. Problems can compound over time if the elements of accidental complexity discussed here — incomplete transitions, excessive abstractions, ill-defined logical boundaries between components, and tooling complexity — are not reined in. The lessons distilled here from QEMU's experience provide ample guidance for other projects confronted with similar obstacles.

[I'd like to thank Paolo Bonzini for substantial reviews of earlier drafts of this article.]

Comments (10 posted)

Digging into Julia's package system

We recently looked at some of the changes and new features arriving with the upcoming version 1.7 release of the Julia programming language. The package system provided by the language makes it easier to explore new language versions, while still preserving multiple versions of various parts of the ecosystem. This flexible system takes care of dependency management, both for writing exploratory code in the REPL and for developing projects or libraries.

The package system

Julia's package system allows users to experiment with new versions of the language or specific packages without having to worry about conflicts or dependency issues. At the moment, I have four versions of Julia on my computer, and have been using and testing them all, with an assortment of packages under active development, and with my own evolving projects, with no difficulties. This is not due to any particular foresight on my part. The package system largely works automatically, resolving the dependency graph by installing and pre-compiling modules as needed.

In an article from a year ago, I traced Julia's popularity in the sciences in part to its unique ability to allow users to combine features from multiple third-party modules. This is enabled by Julia's type system, its use of multiple dispatch, and its optimizing just-ahead-of-time compiler. But it is made convenient by the package system, which removes the pain from managing a diverse library of software; it is similar to the way that Git revolutionized painless branching and merging, which encouraged programmers to be more willing to experiment with features.

After starting the REPL by invoking the Julia binary on the command line, the user may need functionality not provided in the standard library. The using command imports new functions, variables, and data types into the global namespace. For example, if I want to plot something, I need to execute using Plots, after which I can call plot(sin) and see a picture of the sine function.

The plot() function is one of dozens of names exported by the Plots package. A package is simply code (in a module) alongside some other information including its author, version, and a list of other packages that it depends on. A module is an ordinary Julia program containing statements listing the items that it wants to export, with everything wrapped in a module definition that provides its name. We will look deeper into modules later in the article.

The Plots package is under active development. The version that I import needs to be compatible with the version of Julia that I'm running in the REPL. If I start another REPL by running a different version of Julia, I may need a different version of Plots; and that version of Plots may need different versions of the packages that it depends on. Keeping track of these dependencies, and installing the correct versions of packages to resolve them, is the job of the package manager. It's not an add-on system, but an integral part of Julia.

The package system maintains a tree of which versions of packages to use with various versions of Julia by setting up a different environment for each version. These environments are simply directories named after the language versions, containing two files with information about that version's dependencies. Everything is stored in the user's .julia directory.

When the user starts up the REPL, it's in one of these default environments: the one that matches the version of Julia running in the REPL. The two files used for tracking dependencies are Project.toml and Manifest.toml; the file extension stands for "Tom's Obvious, Minimal Language" designed by Tom Preston-Werner. The files are maintained automatically and the user never needs to look at them.

Package manipulations are most conveniently carried out in the REPL "package mode", which is entered by pressing the ] key. The prompt changes to indicate the mode and current environment. After starting version 1.7rc1 and entering package mode, my prompt is (@v1.7) pkg>. The part within the parentheses indicates the environment; environments beginning with "@" are default environments determined by the Julia version. The package manager doesn't distinguish between a release candidate and the released Julia version, but does note the actual current version in Manifest.toml.

Adding a package uses the add command in package mode; for example, add Plots. This kicks off the process of downloading the files from an official repository on GitHub and pre-compiling the code. The package manager will not needlessly duplicate files, saving disk space by sharing resources when possible. The packages installed directly by the user are listed in the Project.toml file, while Manifest.toml records the entire dependency graph of the environment.

After adding Plots to my fresh install of version 1.7rc1, my Project.toml contained the single line for that package, and my Manifest.toml contained 815 lines: Plots pulls in many other packages.

The command update <Packagename> searches for new versions and installs them, along, of course, with any updated dependencies. The rm command deletes a package from the list of dependencies in the Project.toml file. It undoes an add command. If other packages in the active environment depend on the package, it will persist in the Manifest.toml file.

The rm command does not actually delete anything from the filesystem. An automatic garbage-collection process runs, if needed, when package commands are used. It reclaims disk space by purging packages that no other installed package depends on and that haven't been used for over 30 days. If one needs disk space right away, the garbage collector can be called manually and supplied with a time frame other than the 30-day default.

With this system I can effortlessly switch among my installed Julia versions, using any mixture of packages with any of them. Julia's package manager keeps each default environment logically separate, while avoiding file duplication.

Local packages

The package manager can also work with locally developed code and include it in the dependency management system alongside external packages. To see how this works we need to understand more precisely what packages and modules are.

A Julia module is a section of code beginning with the line module <Modulename> and ending with the keyword that terminates all blocks: end. Its purpose is to define a global namespace from which functions and variables can be imported by other programs. Here is a file defining a module called M1:

    module M1

    export plusone
    plusone(x) = x + 1
    a = 17
    
    end

The export commands list the names that can be used without namespace qualification after this module is imported. Julia doesn't hide anything, though: the caller can access the value of a in this module with M1.a.

An include statement pastes in the file directly, but for portable code reuse we would prefer to be able to simply say using M1 in the REPL or from another program file. For this to work, the module needs to be turned into a package.

A package is a collection of three things: a Project.toml file containing some metadata, such as the identity of the package author; a src directory; and, inside that, a program file like the example above that defines a module. The file should be named after the module; so for the above it should be named M1.jl.

If the M1.jl program uses functions from other packages, it will have dependencies. We can get the package manager to track these dependencies for us, just as it does for our default REPL environments. This is the purpose of the activate command: activate <path> tells the package manager to apply all subsequent commands to the project whose Project.toml file is in path. After activating M1 in this way, we could then issue the command add Plots, it will add Plots as a dependency to M1's Project.toml file, and also build a Manifest.toml file to record the dependency graph. If the latest version of Plots has not been downloaded, it will take care of that, too.

A convenient way to begin developing a Julia project is to say (in this example) generate M1 in the REPL package mode. A complete package directory with a skeleton program file defining the M1 module will be created.

A colleague can install your package (by cloning it from a Git repository or simply copying the files) and then tell the package manager to recreate its environment. This is done by using the activate command to switch to it; the instantiate command is then used to consult the project's Manifest.toml file and install everything in the dependency graph. The code can now be run in the exact environment in which it was developed.

Tracking local packages

Say we'd like to use our local package in our REPL sessions while keeping track of it as it develops with the package manager, just as we do for external packages such as Plots. If we ask the package manager to add it with add <path>, we get an error:

    ERROR: Did not find a git repository at `<path>` 

It appears that we can add dependencies to our package, but we're not yet allowed to add our package as a dependency to anything else.

Just as the public Julia ecosystem lives on GitHub, local projects must live in Git repositories if they are to fully participate in the package system. Initializing a Git repository for the package and making an initial commit is all that's required to satisfy Julia's package system. When tracking local projects, the package manager doesn't look at the file tree, but at the Git repository. An add or update command in package mode checks out the files from the local repository and caches them in the .julia/packages directory. By default this tracks the tip of the master branch, but it is also possible to track other branches.

The package manager tracks commits not through the commit hash, however, but through the tree hash. Many Git users are unaware of this hash, because it's rarely needed for anything. The tree hash encodes the actual contents of all the tracked files in the commit; it can be displayed with git log ‑‑pretty=raw. Julia's package manager uses this rather than the commit hash because it's more reliable: through rebasing or other Git operations it's possible to break the connection between the commit hash and the file state that it's supposed to represent.

Contributing to public packages: a case study

For a while I've thought that the Plots package needed a particular feature. This morning I cloned the project to my computer, added the feature, made a pull request on GitHub, made a change suggested by one of the maintainers, and got it approved. The entire elapsed time for this process was about five hours. In this section I'll describe two more package system commands that make it easier to hack on public packages.

In the REPL, I entered package mode, then executed develop Plots. This command, which can be shortened to dev, clones the named package's Git repository to the user's machine in the directory .julia/dev/<PackageName>. Since Plots is a big package with many source files, this took about two minutes.

This command also alters the environment so that using Plots imports from the version under development, rather than the official version. The command free Plots returns to using the official version. One can switch back and forth between these two incarnations of the package freely, as subsequent dev commands won't download anything, but simply switch back to the development version.

I entered the development directory and created a branch for my feature with the git checkout ‑b command. The package manager doesn't require this; it's happy to let you mangle the master branch. But I had plans to ask that my feature be merged into master, and needed to create a branch for it. Packages under develop are loaded from the file tree, not from the Git repository.

Then I wanted to edit the function to add my feature. But where is it? Plots has 37 files in its src tree. Because of multiple dispatch, each function can have dozens of methods associated with it, all with the same name. This makes finding a particular method in the source difficult to accomplish with simple grep commands.

The @edit macro comes to the rescue in just this situation. Supplied with a function call, it opens the user's default editor, right in the REPL, to the method definition that the compiler would select for that particular call. For example, there are 224 methods for the + function. One of these adds together a date and a time. If I want to edit, or look at, the method for that, I enter:

    @edit +(<date_var>, <time_var>)

The two arguments are any variables with the given data types. Vim (in my case) opens in the REPL with the cursor on the correct method definition. I used this macro to edit the plotting function that I wanted to enhance, and tested that it worked as desired. After that, I made my pull request.

Although they don't make actual programming any easier, the dev and free commands, along with the @edit macro, remove some of the barriers that stand between the possibly intimidated programmer and a large codebase.

The final essential tool is a package called Revise.jl, which indeed is listed under the "Essential Tools" heading on the Julia home page, along with the debugger and profiler. With this package imported into the REPL, every time the programmer makes an edit to code under development, the new versions of any changed functions are immediately made active. No manual re-importing nor restarting of the REPL is required, saving time and making the development experience more interactive and fluid.

The culture around the Julia development community is a big factor in encouraging new contributors. Many of the contributors to Julia packages are domain experts in areas of science or mathematics, rather than professional programmers; they are interested in improving the tools that they use in their research. Repository maintainers are welcoming and helpful, rather than dismissive of imperfect code. This generous attitude is evident in the discussions attached to pull requests on GitHub.

Modern software development gains much of its power by building solutions on top of imported code, but with this power often comes headaches when a change in some library breaks something that was working yesterday. Julia's package system eliminates most of these headaches.

In a follow-up article, we'll provide an overview of parallel and concurrent computing in Julia. It will look at multithreading, heterogeneous computing, facilities for using GPUs, and task-based parallelism, including an important improvement in multithreading behavior coming in version 1.7.

Comments (12 posted)

A study of data collection by Android devices

A group of researchers at Trinity College in Dublin has released the results of a study into the data collected by a number of Android variants. There are few surprises here, but the picture is still discouraging.

We find that the Samsung, Xiaomi, Huawei and Realme Android variants all transmit a substantial volume of data to the OS developer (i.e. Samsung etc) and to third-party parties that have pre-installed system apps (including Google, Microsoft, Heytap, LinkedIn, Facebook). LineageOS sends similar volumes of data to Google as these proprietary Android variants, but we do not observe the LineageOS developers themselves collecting data nor pre-installed system apps other than those of Google. Notably, /e/OS sends no information to Google or other third parties and sends essentially no information to the /e/OS developers.

Comments (40 posted)

Security quotes of the week

We need to fix the internet, not the tech giants. The problem isn't just that Zuck is really bad at being the unelected pope-emperor of the digital lives of 3,000,000,000 people – it's that the job of "pope-emperor of 3,000,000,000 people" should be abolished.

I believe that people who rely on digital tools should have the final say in how those tools serve them. That's the proposition at the core of the "nothing about us without us" movement for accessible tech, and the ethos of Free Software.

— Cory Doctorow

It’s why “see something, say something” doesn’t work. If you put amateurs in the front lines of security, don’t be surprised when you get amateur security.

— Bruce Schneier

Comments (none posted)

Kernel release status

The current development kernel is 5.15-rc5, released on October 10. Linus said: "So things continue to look quite normal, and it looks like the rough patch (hah!) we had early in the release is all behind us. Knock wood."

Stable updates: 5.14.10 and 4.4.287 were released on October 7, followed by 5.14.11, 5.10.72, 5.4.152, 4.19.210, 4.14.250, 4.9.286, and 4.4.288 on October 10, and 5.14.12, 5.10.73, 5.4.153, and 4.19.211 on October 13.

Comments (none posted)

Quote of the week

Well, I was thinking simple locking could work too. But I guess RCU is like Batman. You know, "Always be yourself. Unless you can be Batman, then always be Batman!". So always use locking, unless you can use RCU, then always use RCU.

— Steve Rostedt

Comments (none posted)

Jörg Schilling is gone

Jörg Schilling, a longtime free-software developer, has passed on. Most people will remember him from his work on cdrtools and the seemingly endless drama that surrounded that work. He was a difficult character to deal with, but he also contributed some important code that, for a period, almost all of us depended on. Rest well, Jörg.

Full Story (comments: 21)

LibrePlanet 2022 returns online

The Free Software Foundation has opened the call for sessions for its 2022 LibrePlanet conference, which will be held online, sometime in the (northern hemisphere) spring of 2022. The call for sessions is open until December 1. "Potential talks should examine free software through the lens of this year's theme: Living Liberation."

Comments (none posted)

CFP Deadlines: October 14, 2021 to December 13, 2021

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
October 19	November 30 December 2	Yocto Project Summit 2021-11	virtual
October 24	December 1 December 4	FlaskCon	Online
November 1	July 28 July 31	Southern California Linux Expo	Los Angeles, CA, USA
December 10	April 5 April 7	Cephalocon 2022	Portland, OR, US

If the CFP deadline for your event does not appear here, please tell us about it.

Events: October 14, 2021 to December 13, 2021

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
October 17 October 19	All Things Open	Raleigh, NC, and virtual
November 3	Qt World Summit 2021	Online
November 5 November 6	Seattle GNU/Linux conference	Online
November 9 November 10	Open Source Strategy Forum New York	New York, NY, USA
November 9 November 10	PackagingCon 2021	Online
November 10 November 12	Meeting C++ 2021	Online
November 17	SQLite & Tcl virtual conference	Online
November 27 November 28	EmacsConf	Online
November 30 December 2	Yocto Project Summit 2021-11	virtual
December 1 December 4	FlaskCon	Online
December 2 December 3	PGConf NYC	New York, USA
December 3 December 4	OLF Conference	Columbus, OH, USA

If your event does not appear here, please tell us about it.

Alert summary October 7, 2021 to October 13, 2021

Dist.	ID	Release	Package	Date
Debian	DSA-4982-1	stable	apache2	2021-10-08
Debian	DLA-2782-1	LTS	firefox-esr	2021-10-11
Debian	DSA-4981-1	stable	firefox-esr	2021-10-06
Debian	DSA-4984-1	stable	flatpak	2021-10-12
Debian	DLA-2783-1	LTS	hiredis	2021-10-12
Debian	DLA-2784-1	LTS	icu	2021-10-12
Debian	DLA-2779-1	LTS	mediawiki	2021-10-09
Debian	DLA-2781-1	LTS	neutron	2021-10-11
Debian	DSA-4983-1	stable	neutron	2021-10-10
Debian	DLA-2780-1	LTS	ruby2.3	2021-10-13
Debian	DLA-2777-1	LTS	tiff	2021-10-09
Fedora	FEDORA-2021-ab09a05562	F33	chromium	2021-10-09
Fedora	FEDORA-2021-669df5ceb9	F33	dr_libs	2021-10-10
Fedora	FEDORA-2021-450d81de95	F34	dr_libs	2021-10-10
Fedora	FEDORA-2021-2c74a1d70c	F34	firefox	2021-10-09
Fedora	FEDORA-2021-4b201d15e6	F34	flatpak	2021-10-12
Fedora	FEDORA-2021-c35235c250	F34	grafana	2021-10-10
Fedora	FEDORA-2021-2a10bc68a4	F34	httpd	2021-10-12
Fedora	FEDORA-2021-9dd76a1ed0	F33	kernel	2021-10-11
Fedora	FEDORA-2021-ffda3d6fa1	F34	kernel	2021-10-11
Fedora	FEDORA-2021-f2a020a065	F33	libssh	2021-10-07
Fedora	FEDORA-2021-56d8173b5e	F33	mediawiki	2021-10-12
Fedora	FEDORA-2021-eee8b7514f	F34	mediawiki	2021-10-12
Fedora	FEDORA-2021-8913c7900c	F33	redis	2021-10-12
Fedora	FEDORA-2021-61c487f241	F34	redis	2021-10-12
Fedora	FEDORA-2021-fbad11014a	F33	xstream	2021-10-12
Fedora	FEDORA-2021-d894ca87dc	F34	xstream	2021-10-12
Mageia	MGASA-2021-0470	8	apache	2021-10-08
Mageia	MGASA-2021-0467	8	cockpit	2021-10-06
Mageia	MGASA-2021-0464	8	fail2ban	2021-10-06
Mageia	MGASA-2021-0469	8	firefox	2021-10-08
Mageia	MGASA-2021-0468	8	libcryptopp	2021-10-06
Mageia	MGASA-2021-0471	8	libreoffice	2021-10-12
Mageia	MGASA-2021-0465	8	libss7	2021-10-06
Mageia	MGASA-2021-0463	8	nodejs	2021-10-06
Mageia	MGASA-2021-0462	8	opendmarc	2021-10-06
Mageia	MGASA-2021-0466	8	weechat	2021-10-06
openSUSE	openSUSE-SU-2021:1350-1	15.2	chromium	2021-10-12
openSUSE	openSUSE-SU-2021:1339-1	SLE15.3	chromium	2021-10-11
openSUSE	openSUSE-SU-2021:3298-1	15.3	curl	2021-10-06
openSUSE	openSUSE-SU-2021:3293-1	15.3	ffmpeg	2021-10-06
openSUSE	openSUSE-SU-2021:3331-1	15.3	firefox	2021-10-11
openSUSE	openSUSE-SU-2021:1345-1	15.2	git	2021-10-11
openSUSE	openSUSE-SU-2021:3300-1	15.3	git	2021-10-06
openSUSE	openSUSE-SU-2021:3291-1	15.3	glibc	2021-10-06
openSUSE	openSUSE-SU-2021:1342-1	15.2	go1.16	2021-10-11
openSUSE	openSUSE-SU-2021:3292-1	15.3	go1.16	2021-10-06
openSUSE	openSUSE-SU-2021:3338-1	15.3	kernel	2021-10-12
openSUSE	openSUSE-SU-2021:3387-1	15.3	kernel	2021-10-12
openSUSE	openSUSE-SU-2021:3350-1	15.3	libaom	2021-10-12
openSUSE	openSUSE-SU-2021:3301-1	15.3	libcryptopp	2021-10-06
openSUSE	openSUSE-SU-2021:3354-1	15.3	libqt5-qtsvg	2021-10-12
openSUSE	openSUSE-SU-2021:1344-1	15.2	mbedtls	2021-10-11
openSUSE	openSUSE-SU-2021:1341-1	15.2	mupdf	2021-10-11
openSUSE	openSUSE-SU-2021:1343-1	15.2	nodejs8	2021-10-11
openSUSE	openSUSE-SU-2021:3294-1	15.3	nodejs8	2021-10-06
openSUSE	openSUSE-SU-2021:3325-1	15.3	rabbitmq-server	2021-10-10
openSUSE	openSUSE-SU-2021:3348-1	15.3	systemd	2021-10-12
openSUSE	openSUSE-SU-2021:3353-1	15.3	webkit2gtk3	2021-10-12
Oracle	ELSA-2021-3755	OL8	firefox	2021-10-11
Oracle	ELSA-2021-9473	OL6	kernel	2021-10-08
Oracle	ELSA-2021-9473	OL7	kernel	2021-10-08
Oracle	ELSA-2021-9474	OL7	kernel	2021-10-11
Oracle	ELSA-2021-9474	OL8	kernel	2021-10-11
Oracle	ELSA-2021-9474	OL8	kernel	2021-10-11
Red Hat	RHSA-2021:3818-01	EL7	.NET 5.0	2021-10-12
Red Hat	RHSA-2021:3819-01	EL8	.NET 5.0	2021-10-12
Red Hat	RHSA-2021:3807-01	EL7	389-ds-base	2021-10-12
Red Hat	RHSA-2021:3791-01	EL7	firefox	2021-10-12
Red Hat	RHSA-2021:3755-01	EL8	firefox	2021-10-11
Red Hat	RHSA-2021:3757-01	EL8.1	firefox	2021-10-11
Red Hat	RHSA-2021:3756-01	EL8.2	firefox	2021-10-11
Red Hat	RHSA-2021:3771-01	EL8	grafana	2021-10-12
Red Hat	RHSA-2021:3769-01	EL8.1	grafana	2021-10-12
Red Hat	RHSA-2021:3770-01	EL8.2	grafana	2021-10-12
Red Hat	RHSA-2021:3754-01	SCL	httpd24-httpd	2021-10-11
Red Hat	RHSA-2021:3816-01	EL8	httpd:2.4	2021-10-12
Red Hat	RHSA-2021:3837-01	EL8.1	httpd:2.4	2021-10-13
Red Hat	RHSA-2021:3836-01	EL8.2	httpd:2.4	2021-10-13
Red Hat	RHSA-2021:3801-01	EL7	kernel	2021-10-12
Red Hat	RHSA-2021:3767-02	EL7.2	kernel	2021-10-12
Red Hat	RHSA-2021:3766-01	EL7.3	kernel	2021-10-12
Red Hat	RHSA-2021:3812-01	EL7.6	kernel	2021-10-12
Red Hat	RHSA-2021:3802-01	EL7	kernel-rt	2021-10-12
Red Hat	RHSA-2021:3768-01	EL7	kpatch-patch	2021-10-12
Red Hat	RHSA-2021:3814-01	EL7.6	kpatch-patch	2021-10-12
Red Hat	RHSA-2021:3810-01	EL7	libxml2	2021-10-12
Red Hat	RHSA-2021:3798-01	EL7	openssl	2021-10-12
Red Hat	RHSA-2021:3811-01	SCL	rh-mysql80-mysql	2021-10-12
Red Hat	RHSA-2021:3841-01	EL7	thunderbird	2021-10-13
Red Hat	RHSA-2021:3838-01	EL8	thunderbird	2021-10-13
Red Hat	RHSA-2021:3840-01	EL8.1	thunderbird	2021-10-13
Red Hat	RHSA-2021:3839-01	EL8.2	thunderbird	2021-10-13
Scientific Linux	SLSA-2021:3807-1	SL7	389-ds-base	2021-10-12
Scientific Linux	SLSA-2021:3801-1	SL7	kernel	2021-10-12
Scientific Linux	SLSA-2021:3810-1	SL7	libxml2	2021-10-12
Scientific Linux	SLSA-2021:3798-1	SL7	openssl	2021-10-12
Slackware	SSA:2021-280-01		httpd	2021-10-07
SUSE	SUSE-SU-2021:3299-1	OS8 OS9 SLE12	apache2	2021-10-06
SUSE	SUSE-SU-2021:3335-1	SLE15 SES6	apache2	2021-10-12
SUSE	SUSE-SU-2021:3352-1	SLE12	apache2-mod_auth_openidc	2021-10-12
SUSE	SUSE-SU-2021:3336-1	SLE12	containerd, docker, runc	2021-10-12
SUSE	SUSE-SU-2021:3351-1	OS9 SLE12	curl	2021-10-12
SUSE	SUSE-SU-2021:3332-1	SLE12	curl	2021-10-11
SUSE	SUSE-SU-2021:3298-1	SLE15	curl	2021-10-06
SUSE	SUSE-SU-2021:3297-1	SLE15 SES6	curl	2021-10-06
SUSE	SUSE-SU-2021:3293-1	SLE15	ffmpeg	2021-10-06
SUSE	SUSE-SU-2021:3331-1	SLE15 SES6	firefox	2021-10-11
SUSE	SUSE-SU-2021:3300-1	SLE15	git	2021-10-06
SUSE	SUSE-SU-2021:3289-1	OS8 SLE12	glibc	2021-10-06
SUSE	SUSE-SU-2021:3290-1	OS9 SLE12	glibc	2021-10-06
SUSE	SUSE-SU-2021:3291-1	SLE15	glibc	2021-10-06
SUSE	SUSE-SU-2021:3385-1	SLE15 SES6	glibc	2021-10-12
SUSE	SUSE-SU-2021:3292-1	SLE15	go1.16	2021-10-06
SUSE	SUSE-SU-2021:3295-1	SLE15 SES6	grilo	2021-10-06
SUSE	SUSE-SU-2021:3386-1	SLE12	kernel	2021-10-12
SUSE	SUSE-SU-2021:3388-1	SLE12	kernel	2021-10-12
SUSE	SUSE-SU-2021:3389-1	SLE12	kernel	2021-10-12
SUSE	SUSE-SU-2021:3205-2	SLE15	kernel	2021-10-13
SUSE	SUSE-SU-2021:3337-1	SLE15	kernel	2021-10-12
SUSE	SUSE-SU-2021:3338-1	SLE15	kernel	2021-10-12
SUSE	SUSE-SU-2021:3339-1	SLE15	kernel	2021-10-12
SUSE	SUSE-SU-2021:3387-1	SLE15	kernel	2021-10-12
SUSE	SUSE-SU-2021:3350-1	SLE15	libaom	2021-10-12
SUSE	SUSE-SU-2021:3301-1	SLE15	libcryptopp	2021-10-06
SUSE	SUSE-SU-2021:3333-1	SLE12	libqt5-qtsvg	2021-10-11
SUSE	SUSE-SU-2021:3354-1	SLE15	libqt5-qtsvg	2021-10-12
SUSE	SUSE-SU-2021:3294-1	SLE15	nodejs8	2021-10-06
SUSE	SUSE-SU-2021:3325-1	SLE15	rabbitmq-server	2021-10-10
SUSE	SUSE-SU-2021:3334-1	SLE12	squid	2021-10-11
SUSE	SUSE-SU-2021:3348-1	SLE15	systemd	2021-10-12
SUSE	SUSE-SU-2021:14823-1	SLE11	transfig	2021-10-06
SUSE	SUSE-SU-2021:3296-1	OS8 OS9 SLE12	webkit2gtk3	2021-10-06
SUSE	SUSE-SU-2021:3353-1	SLE15	webkit2gtk3	2021-10-12
SUSE	SUSE-SU-2021:3322-1	SLE12	xen	2021-10-08
Ubuntu	USN-5107-1	18.04 20.04 21.04	firefox	2021-10-08
Ubuntu	USN-5108-1	18.04 20.04	libntlm	2021-10-08
Ubuntu	USN-5106-1	20.04	linux-oem-5.10	2021-10-06
Ubuntu	USN-5022-3	16.04	mysql-5.7	2021-10-07
Ubuntu	USN-5105-1	18.04 20.04	python-bottle	2021-10-07
Ubuntu	USN-5078-3	20.04 21.04	squashfs-tools	2021-10-13

Full Story (comments: none)