Leading items

Welcome to the LWN.net Weekly Edition for September 16, 2021

This edition contains the following feature content:

• Revisiting NaNs in Python: the Python development community returns to the question of how to handle not-a-number values in statistics functions.
• Extended attributes for special files: the first solution to a security-related problem is not always the best one.
• The folio pull-request pushback: the discussion that led to the folio patches not being pulled for 5.15.
• The rest of the 5.15 merge window: changes applied for the 5.15 kernel in the second half of the merge window.
• Roundup: managing issues for 20 years: the past, present, and future of the Roundup issue-tracker project.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Revisiting NaNs in Python

Back in January 2020, we looked at some oddities in Python's handling of Not a Number (NaN) values in its statistics module. The conversation went quiet after that, but it has been revived recently with an eye toward fixing the problems that were reported. As detailed in that earlier article, NaNs are rather strange beasts in the floating-point universe, so figuring out how best to deal with their presence is less straightforward than it might seem.

Not a number

NaNs are defined in the IEEE 754 floating-point standard, which Python follows, as special values for quantities that cannot be represented by the hardware. This can happen as a result of operations like division by zero or taking the square root of a negative number, though Python raises an exception in those cases. Sometimes NaNs are used as a kind of special value marking missing data, especially in scientific or big-data contexts, but doing so can cause some odd behavior.

The earlier discussion centered around the statistics.median() function, which returns the "middle" value in a list of numbers; the list is sorted and either the actual middle or the average of the two middle values is returned. But median() (and others) run afoul of the perhaps counterintuitive nature of NaN comparisons. In Python (and IEEE 754), all comparisons involving NaNs are false, except for not-equal, which is always true, even when comparing the same two NaN values.

    >>> NaN = float('nan')
    >>> NaN < 0
    False
    >>> NaN >= 0
    False
    >>> NaN == NaN
    False
    >>> NaN != NaN
    True

Steven D'Aprano, who developed the statistics module for the standard library, returned to the problem with median() in an August 24 post to the python-ideas mailing list. He noted that NaN handling in the statistics module is implementation dependent, which "*usually* means that if your data has a NAN in it, the result you get will probably be a NAN". But median() behaves differently:

    >>> statistics.median([1, 2, float('nan'), 4])
    nan
    >>> statistics.median([float('nan'), 1, 2, 4])
    1.5

I've spoken to users of other statistics packages and languages, such as R, and I cannot find any consensus on what the "right" behaviour should be for NANs except "not that!".

He proposed adding a keyword argument to functions in the module to indicate what should be done when NaNs are found in their input. He saw three possible choices: raise an exception, return NaN, or filter out the NaNs and ignore them. He was looking for opinions on or objections to that plan, as well as wondering what the default should be.

Christopher Barker noted that NumPy has a set of functions (nan*()) that explicitly ignore NaNs, but:

Filtering [out] NaNs should *not* be the default. Often NaN means missing data, but could also be the result of an error of some sort. Incorrect results are much worse than errors — NaNs should never be ignored unless explicitly asked for.

Beyond that, I’d prefer returning NaN to raising an exception, but either is OK.

Marc-Andre Lemburg concurred that filtering should be an explicit choice. He likened the situation to that of the errors parameter for the codecs module; it allows the programmer to choose various types of behavior when errors are encountered in encoding or decoding the input. Finn Mason suggested having an option to warn the user if NaN values were ignored; he also thought that NaNs could perhaps be treated as zeroes in that case: "This allows calculations to still quickly and easily be made with or without NaNs, but an alternative course of action can be taken in the presence of a NaN value if desired." Treating NaNs as zeroes was a non-starter, but D'Aprano did see value in the idea of a warning.

More surprises

There are other ways that NaNs exhibit surprising behavior, as highlighted by a suggestion made by Lemburg. He said that codecs defaults to raising an exception to expose the need to make an explicit decision, but that might not be the right choice for a long-running calculation that encounters a NaN. He said that a simple test, similar to the one below, could be used to check for NaNs before doing said calculation:

    >>> a = [ 1, 2, 3, NaN ]
    >>> NaN in a
    True

While that particular test works, Peter Otten pointed out that it is not actually a solution to the problem because the container test skips the equality comparison if the objects are the same, as the following shows:

    >>> float('nan') in a
    False

Unless all of the NaNs are the same object, math.isnan() must be used instead. Lemburg thought that NaNs were treated as a singleton, but, as D'Aprano pointed out, there are an enormous number of NaN values defined by the standard; making them into a singleton would lose extra information that might be useful:

The IEEE-754 standard doesn't mandate that NANs preserve the payload, but it does recommend it. We shouldn't gratuitously discard that information. It could be meaningful to whoever is generating the data.

Lemburg thinks that the current median() behavior is a bug, but D'Aprano reiterated at some length that it was consistent with the standard. He also pointed out that there are further consequences of the comparison rules for NaNs:

So when you sort a list containing NANs, they end up in some arbitrary position that depends on the sort implementation, the other values in the list, and their initial position. NANs can even throw out the order of other values:

>>> sorted([3, nan, 4, 2, nan, 1])
[3, nan, 1, 2, 4, nan]

and *that* violates `median`'s assumption that sorting values actually puts them in sorted order, which is why median returns the wrong value.

Adding the totalOrder predicate, which provides a consistent definition of where to sort NaNs (and other special values), would perhaps be helpful, but could not change the existing NaN handling in Python. As Richard Damon noted: "Asking for the median value of a list that doesn't have a proper total order is a nonsense question, so you get a nonsense answer."

Defaulting to returning NaN would be consistent with other Python math functions, Mark Dickinson said. Brendan Barnwell also thought that was probably the best default, but he raised another issue:

One important thing we should think about is whether to add similar handling to `max` and `min`. These are builtin functions, not in the statistics module, but they have similarly confusing behavior with NAN: compare `max(1, 2, float('nan'))` with `max(float('nan'), 1, 2)`. As long as we're handling this for median and so on, it would be nice to have the ability to do NAN-aware max and min as well.

As might be guessed from earlier examples, the first max() call returns 2, while the second returns a NaN. As with the existing behavior of NaN comparisons, though, no change to the default behavior of those built-in functions can be made for backward-compatibility reasons. If desired, though, a new keyword argument could be added as D'Aprano proposed for median().

Naming

After a few days discussion, D'Aprano opened the bikeshed floodgates when he asked for opinions on the name and type of the parameter. "I'm leaning towards "nans=..." with an enum." Sebastian Berg surveyed a few other Python libraries, two of which use string types to specify what to do about NaNs. Barker liked the SciPy version, which has a nan_policy parameter with three values: 'propagate' (return NaN), 'raise' (an exception), or 'omit' (filter NaNs). In any case, he preferred a string flag rather than an Enum.

Lemburg agreed, noting that codecs uses string values; others in the thread were largely in favor of that approach as well. But Ronald Oussoren wondered what the objection to using an enum is. Barker does not see any real advantage to them over string flags, except for static typing, and noted that they are somewhat more painful to use. But for flags that can be combined in various ways (e.g. with bitwise-or, "|"), enums have a "*huge* advantage", Chris Angelico said:

It's easy to accept a flags argument that is the bitwise Or of a collection of flags, and then ascertain whether or not a specific flag was included. The repr of such a combination is useful and readable, too.

It does not make sense to combine the NaN flags, though. Oussoren pointed out that enums provide more advantages than just for static typing; static analysis and things like auto-completion in IDEs are enabled by enums. He said that he had no opinion on what should be used for statistics, but that he has been moving string flags to enums in his own code, in part to avoid bugs from typos in the strings. For Barker, though, that is more evidence of a shift in Python over the years:

Features to support static analysis are a good example -- far less important for "scripting' [than] "systems programming".

Personally, I find it odd -- I've spent literally a couple decades telling people that Python's dynamic nature is a net plus, and the bugs that static typing (and static analysis I suppose) catch for you are generally shallow bugs. (for example: misspelling a string flag is a shallow bug).

But here we are.

Anyway, if you are writing a quick script to calculate a few statistics, I think a string flag is easier. If you are writing a big system with some statistical calculations built in, you will appreciate the additional safety of an Enum. It's hard to optimize for different use cases.

The discussion largely wound down after that. D'Aprano thanked participants for their opinions along the way, so he is clearly trying to find a middle-ground position to solve the problems in the statistics module. Based on the discussion, one might guess that the default will be to return a NaN, the parameter will be called nans or maybe nan_policy, and that string flag values will be used. In any case, the open nature of Python development once again gives us a nice look inside the kinds of problems that are grappled with on the way to fixing corner cases and surprising behavior in the language.

Comments (19 posted)

Extended attributes for special files

The Linux extended-attribute mechanism allows the attachment of metadata to files within a filesystem. It tends to be little used — at least, in the absence of a security module like SELinux. There is interest in how these attributes work, though, as evidenced by the discussions that have followed the posting of revisions of this patch by Vivek Goyal, which seeks to make a seemingly small change to the rules regarding extended attributes and special files.

Specifically, extended attributes (often referred to as "xattrs") are name-value pairs that can be attached to a file. The name of an extended attribute is divided into a namespace and an identifier within the namespace. The namespace is currently one of security, system, user, or trusted; each namespace has its own special rules. As a general rule, system and trusted see little use. The security namespace, instead, is used by a number of Linux security modules. An example of a security attribute can be seen by running getfattr on a system that is set up for SELinux:

    cd /; getfattr -d -m - .
    # file: .
    security.selinux="system_u:object_r:root_t:s0"

This is the SELinux label, which is used to determine which access policies apply to this directory. A process must have the CAP_SYS_ADMIN capability to change attributes in the security namespace; even then, a running security module might have its own ideas regarding whether to allow a change to happen.

The user namespace is entirely unprivileged; users may assign whatever attributes they like to any files they are capable of writing. There is an exception, though: it is not possible to attach user extended attributes to symbolic links or device special files. The man page describes this restriction as a control on resource consumption; if any process that can write to /dev/null can attach extended attributes to it, the disk could quickly fill with attribute data. It is still possible for a suitably privileged process to attach security extended attributes to those files.

Goyal's patch relaxes that restriction slightly, so that a process that owns a symbolic link or device special file would be able to attach user extended attributes to that file. That allows users to change files they have control over anyway while avoiding the /dev/null problem. It seems to many like a reasonable change to make, but Andreas Gruenbacher opposed it:

The idea behind user.* xattrs is that they behave similar to file contents as far as permissions go. It follows from that that symlinks and special files cannot have user.* xattrs. This has been the model for many years now and applications may be expecting these semantics, so we cannot simply change the behavior.

The possibility of breaking user space in subtle ways is indeed worrisome; Goyal has suggested that the new behavior could be made opt-in to avoid creating such surprises. But the bigger sticking point has to do with the intended use case for this feature. Kernel developers will normally ask why a new behavior is needed; that information is necessary to evaluate whether the proposed solution is indeed the best way to solve the problem. In this case, not all of them were happy with the answer.

Goyal is working on virtualization and, in particular, with use cases where guests want to share a filesystem with the host. The kernel's virtiofs filesystem is designed for this application; it functions as a sort of FUSE filesystem that runs over Virtio, so it performs reasonably well. On the host side, the virtiofsd daemon performs the actual filesystem access needed by the client.

Virtiofsd tries to drop as many capabilities as possible, though it must still be able to manipulate files owned by any user and group ID. It seems to work well, except for one little problem: security extended attributes. The client may want to manipulate those attributes within the filesystem, but the host may prevent virtiofsd from carrying out those changes. In particular, if the host is running a security module like SELinux, that module will be placing its own interpretation on those attributes and may not be appreciative of the changes that virtiofsd is trying to make. Changing those extended attributes requires CAP_SYS_ADMIN in any case, and virtiofsd is trying to run without that capability.

The answer to the problem, naturally, is another layer of indirection. When properly configured, virtiofsd is able to remap extended attributes between the guest and the host; it can, for example, be made to turn the security attributes managed by the guest into user attributes on the host. No privilege is required for virtiofsd to change user attributes, so everything works as desired.

More accurately, almost everything works as desired. As Goyal notes, when the guest goes into the dreaded relabeling coma that SELinux seems to require from time to time, errors result. The problem is the restriction on user extended attributes for special files; the guest is trying to relabel its devices, but the host cannot assign those labels (remapped as user extended attributes) to the appropriate special files. That problem is what is driving Goyal's patch relaxing the restrictions, which makes this use case work.

Casey Schaufler let it be known that he had no problem with assigning user extended attributes to special files. But the remapping done in virtiofsd, he said, is "unreasonable":

As I have stated before, this introduces a breach in security. It allows an unprivileged process on the host to manipulate the security state of the guest. This is horribly wrong. It is not sufficient to claim that the breach requires misconfiguration to exploit. Don't do this.

Schaufler and Goyal have gone around on this topic numerous times over the three revisions of this patch set. Schaufler contends that the only secure way to handle a situation like this is for host-side changes to require the same level of privilege as is required on the guest. So, for example, it would be acceptable to remap the security extended attributes into the trusted namespace (which also requires CAP_SYS_ADMIN) instead. Goyal, meanwhile, continues to look for a solution that does not require additional privilege in virtiofsd.

That solution, in the end, may take a bit of a different form, inspired by this observation from David Alan Gilbert:

IMHO the real problem here is that the user/trusted/system/security 'namespaces' are arbitrary hacks rather than a proper namespacing mechanism that allows you to create new (nested) namespaces and associate permissions with each one.

Each one carries with it some arbitrary baggage (trusted not working on NFS, user. having the special rules on symlinks etc).

Miklos Szeredi suggested a next step (evidently originally proposed by Eric Biederman) to adapt trusted extended attributes for use within user namespaces. If a given user namespace is owned by user ID 1000, then trusted.foo within the namespace would be stored (and visible) in the initial namespace as something like trusted<1000>.foo. Virtiofsd could be run within a user namespace and use the trusted extended attributes without any extra privilege, and security modules on the host would still have a say in how those attributes are managed. Goyal agreed that this approach might work.

The only problem is that somebody has to implement the new behavior for user namespaces — a complex part of the system where it is easy to make security-wrecking mistakes. Biederman has already pointed out one potential problem related to nested namespaces. But, assuming that feature can be safely provided, Goyal's original problem should be solvable. This is why kernel developers tend to be inquisitive about the use cases driving a proposed change: the best solution to a problem is often not the first one that comes to mind.

Comments (23 posted)

The folio pull-request pushback

When we last caught up with the page folio patch set, it appeared to be on track to be pulled into the mainline during the 5.15 merge window. Matthew Wilcox duly sent a pull request in August to make that happen. While it is possible that folios could still end up in 5.15, that has not happened as of this writing and appears increasingly unlikely. What we got instead was a lengthy discussion on the merits of the folio approach.

The kernel's memory-management subsystem deals with memory in pages, a fundamental unit that is generally set by the hardware (and is usually 4KB in size). These "base" pages are small, though, so the kernel often needs to deal with memory in larger units. To do so, it groups sets of physically contiguous pages together into compound pages, which are made up of a head page (the first base page of many in the compound page) and some number of tail pages. That leads to a situation where kernel code that is passed a struct page pointer often does not know if it is dealing with a head or a tail page without explicitly checking.

It turns out that the "make sure this is a head page" checks add up to a certain amount of expense in a running kernel. The use of struct page everywhere also makes kernel APIs unclear — it can be difficult to know if a given function can cope with tail pages or not. To address this problem, Wilcox created the concept of a "folio", which is like a struct page but which is known not to be a tail page. By changing internal functions to use folios, Wilcox is able to speed up the kernel and clean up the API at the same time. The patch set is huge and intrusive, but it appeared to have overcome most resistance and be ready to head into the mainline kernel.

Objections

Memory-management developer Johannes Weiner quickly responded to the request to express his "strong reservations" about the folio concept. Over the course of the ensuing discussion he described his objections in a number of ways, but it seems to come down to a core point: a folio is just a collection of physically contiguous pages, and that is going to make it hard to deal with a number of challenges facing memory management.

To start with, the folio design leaks too much information about memory-management internals to other users of folios, filesystems in particular. The current page-oriented APIs have the same problem, of course, but a massive API change should, he said, be the time to address that issue. So Weiner has asked, more than once, for the creation of a more abstract representation of memory that would be used in the higher levels of the kernel. This abstraction would hide a lot of details; it would also eliminate the assumption that a folio is a physically contiguous, power-of-two-sized chunk of memory.

The assumption of physical contiguity, he continued, is a serious problem because the memory-management subsystem has never been good at allocating larger, contiguous chunks of memory. At some point fragmentation takes hold and those larger chunks simply aren't there. Techniques like page compaction can help to an extent, but that comes at the cost of excessive allocation latency. "We've effectively declared bankruptcy on this already", he said. There is no point in adopting folios to represent larger chunks of memory without thinking about how those chunks will be allocated.

The other problem, he said, is that the folio concept could make it much harder to change the memory-management subsystem to use a larger base-page size. That change would make the system more efficient in many ways, including less memory wasted for page structures and less CPU time dedicated to dealing with them. There is one problem that has kept the kernel from increasing the base-page size for many years, though: internal fragmentation.

When a file's contents (or a portion thereof) are stored in the page cache, they require a certain number of full pages. Unix-like systems have a lot of small files, but even a one-line file will occupy a full page in the page cache; all of the memory in that page beyond the end of the file is simply wasted. Increasing the size of a base page will necessarily increase the amount of memory lost to this internal fragmentation as well. In a previous folio discussion, Al Viro did a quick calculation showing just how much more memory it would take to keep the kernel source in memory with a larger page size. A 64KB size would quadruple the memory used, for example; it is not a small cost.

For this reason, Weiner argued that the kernel will need to be able to manage file caching in small units (the existing 4KB size, for example) even when the memory-management subsystem moves to a larger base-page size. In other words, the page cache will need to be able to work with sub-page units of memory. A new abstraction for memory might facilitate that; the current folio concept, being firmly tied to underlying pages, cannot.

Wilcox's answer to this criticism seems to be that it makes little sense in managing memory in units other than the allocation size. The way to use larger units in the memory-management subsystem is thus to allocate compound pages, represented by folios; if everything in the kernel is using larger pages, memory will fragment less and those pages will become easier to allocate. For cases where smaller sizes are needed, such as page-cache entries for small files, simple base pages could be used. With careful allocation, the single pages could be packed together, further avoiding fragmentation.

Weiner disagreed with that approach, though, saying that it puts the fragmentation-avoidance problem in the wrong place. The kernel has two levels of memory allocation: the page allocator (which deals in full pages and is where folios are relevant) and the slab allocator, which normally deals with smaller units. They have different strengths:

The page allocator is good at cranking out uniform, slightly big memory blocks. The slab allocator is good at subdividing those into smaller objects, neatly packed and grouped to facilitate contiguous reclaim, while providing detailed breakdowns of per-type memory usage and internal fragmentation to the user and to kernel developers.

According to Weiner, Wilcox's approach forces the page allocator to deal with problems that are currently well solved in the slab allocator. "As long as this is your ambition with the folio, I'm sorry but it's a NAK from me".

The real problem with folios

The ultimate decision on the merging of folios is, of course, up to Linus Torvalds. Early in the conversation, he wrote positively about the API improvements, but also noted that the patch set does bring a lot of churn to the memory-management subsystem. He concluded: "So I don't hate the patches. I think they are clever, I think they are likely worthwhile, but I also certainly don't love them."

He also, however, noted that he wasn't entirely happy with the "folio" name, thus touching off one of the more predictable dynamics of kernel-community discussions: when the technical side gets involved and intractable, it must be time to have an extended debate on naming. So David Howells suggested "sheaf" or "ream". Torvalds favored something more directly descriptive, like "head_page". Ted Ts'o thought "mempages" would work, or maybe "pageset". Nicholas Piggin favored "cluster" or "superpage". Given the discussion, Vlastimil Babka concluded that the only suitable name was "pageshed".

Wilcox has made it abundantly clear that he doesn't care about the name and will accept just about anything if that gets the code merged. He redid the pull request with everything renamed to "pageset" just to prove that point. Needless to say, no real conclusion came from that branch of the conversation.

At the end of August, Howells posted a plea for a quick resolution on the issue; there is a lot of other pending memory-management work that either depends on the folio patches or conflicts with them. He asked:

Is it possible to take the folios patchset as-is and just live with the name, or just take Willy's rename-job (although it hasn't had linux-next soak time yet)? Or is the approach fundamentally flawed and in need of redoing?

David Hildenbrand added that he would like to see folios move out of linux-next one way or the other; sooner would be better.

Now what?

For now, at least, the conversations have wound down. The 5.15 merge window is nearing its close and the folio patches have not been pulled. Chances are that means that folios will, at best, have to wait for another development cycle. That said, Torvalds has been known to hold controversial pulls until — or even past — the end of the merge window, when he has a bit more time to think them through. So even the closing of the merge window might not be an indication that the decision has been made.

The final chapter has not been written here, but either way it seems clear that there is a lot of work yet to be done in the memory-management subsystem. Much of what needs to happen has yet to be designed, much less written and debugged; that adds some strength to one last argument from Wilcox: "The folio patch is here now". Or, as Babka asked: "should we just do nothing until somebody turns that hypothetical future into code and we see whether it works or not?" If folios go down in flames, work to improve memory-management internals at this level will have to restart from the beginning, and it's not clear that there is anybody out there who is ready to take up that challenge.

Comments (18 posted)

The rest of the 5.15 merge window

Linus Torvalds released 5.15-rc1 and closed the merge window for this release on September 12; at that point, 10,471 non-merge changesets had found their way into the mainline repository. Those changesets contain a lot of significant changes and improvements. Read on for a summary of what came into the mainline in the roughly 7,000 changesets pulled since our first-half summary was written.

Architecture-specific

• As expected, scheduling support for asymmetric Arm systems has been merged. See this documentation patch for details.
• Arm64 kernels can now be built as Hyper-V clients.
• The ARC architecture now supports three-level and four-level page tables; this is evidently a step toward the eventual support for 64-bit processors.

Core kernel

• The printk() indexing patch set has been merged. This feature allows the extraction of a list of all messages that can be emitted by printk(); user-space monitoring utilities can use this list to detect changes.
• Memory usage accounting for memory control groups has been extended to track many more in-kernel data structures, including those created for polling, signal handling, namespaces, and more. Some of those changes have since been reverted, though, due to the performance regression they caused.
• The memory-management subsystem has gained the ability to demote pages to a slower NUMA node in favor of reclaiming them entirely. This mechanism can move less-used pages to nodes hosting nonvolatile memory, which is better than discarding them. This is one piece of the tiered memory-management scheme described in this article.
• The multi-preference memory policy patch has been merged; it allows a process to request that memory pages be allocated on a given set of NUMA nodes.
• The new process_mrelease() system call allows a supervisor process to hasten the reclaim of memory from an exiting process.
• As described in this article, the code implementing the MAP_DENYWRITE option to mmap() has been removed. As a consequence, the number of situations where a write to a file will be denied with an ETXTBSY error has been reduced; see this commit for a concise list.
• "Event probes" are a new type of kernel probe that attach to an existing trace event but which contain their own format string; they can be used to provide a new view of an existing tracepoint. This commit contains a brief description of how they are used.
• The DAMON subsystem has been merged. DAMON allows the monitoring and optimization of memory management from user space. This documentation patch contains more information.

Filesystems and block I/O

• It is no longer possible to turn off disk quota accounting in mounted XFS filesystems. Enforcement can be turned off, but tracking will continue to happen under the hood. From the pull request: "We've tried to do this in a careful enough way that there shouldn't be any user visible effects aside from quotaoff no longer randomly hanging the system".
• XFS now supports filesystems containing dates after 2038. This support has been present for a while but is no longer considered experimental.
• The EROFS filesystem now supports direct I/O on uncompressed files.
• Overlay filesystems now properly handle the immutable, append-only, sync, and noatime attributes.
• A new NTFS filesystem implementation, said to be far better than the kernel's existing version, has been merged. There were some questions about whether this filesystem would make it into 5.15, but it seems that the doubts felt by some filesystem maintainers have been adequately addressed.

Hardware support

• Clock: Qualcomm SC7280 clock controllers, Qualcomm MSM8953 global clock controllers, Qualcomm MSM8994 multimedia clock controllers, Qualcomm SM6350, SM6115 and SM4250 global clock controllers, and MediaTek MT8192 clock controllers.
• Graphics: Innolux EJ030NA 320x480 LCD panels, Widechips WS2401 DPI panels, Samsung ATNA33XC20 eDP panels, and Ilitek ILI9341 240x320 QVGA panels.
• Industrial I/O: Sensirion SGP40 gas sensors, Analog Devices AD5110 digital potentiometers, and Renesas RZ/G2L analog-to-digital converters.
• Media: Amlogic Meson IR transmitters, Sony imx335 and imx412 image sensors, and OmniVision OV9282 sensors.
• Miscellaneous: Xilinx Versal FPGAs, General Electric ACHC microcontrollers, Nintendo Wii consoles, Qualcomm SC8180X interconnects, Realtek RTL8188EU Wireless LAN NICs (replacement driver, still in staging), Intel IXP4xx expansion bus interfaces, Renesas RZ/G2L USB/PHY controllers, Apple DART I/O memory-management units, Renesas I2C-controlled synchronization management units, MediaTek CPU-frequency controllers, Renesas RZ/G2L DMA controllers, and NVIDIA Tegra30 thermal sensors.
• PCI: Rockchip DesignWare PCIe controllers, Toshiba Visconti PCIe controllers, and Keem Bay PCIe controllers.
• Pin control: Qualcomm 9607 pin controllers, Qualcomm SM6115 and SM4250 pin controllers, STMicroelectronics STM32MP135 pin controllers, NXP i.MX8ULP pin controllers, Intel Keem Bay pin controllers, and Renesas RZ/G2L pin and GPIO controllers.
• Sound: Renesas RZ/G2L serial sound interfaces.

Security-related

• The kernel can now be built with the GCC -fzero-call-used-regs=used-gpr option; in English, that means that any registers used within a function will be zeroed just before returning from that function. This helps to prevent data from leaking out of functions; it also, as Kees Cook noted in the changelog, "helps reduce the number of useful ROP [return-oriented programming] gadgets in the kernel image by about 20%".

Virtualization and containers

• The new, fully undocumented gpio-virtio driver allows guests to access GPIO lines provided by the host system.
• There is a new driver framework called "VDUSE" that allows the implementation of virtual block devices in user space, with a Virtio transport to the guest. See this documentation commit for some more information.

Internal kernel changes

• Restricted DMA pools can be used to place limits on the pages a DMA-capable device can access on systems without an I/O memory-management unit.
• Kernel builds will now use the -Werror flag by default, causing the build to fail in the presence of compiler warnings. There is a new WERROR configuration variable that can be used to turn this option off; it is meant to be used in cases where a very new (or old) compiler is generating spurious warnings. This change is causing build failures for some but, so far, Torvalds is unsympathetic:

  So the fact that *I* require a clean build, and generally have to reject a couple of build requests every merge window because of that clearly doesn't mean that we actually have clean builds.

  I've had that "no warnings" requirement for years. And that means that *my* build has been clean for years.

  I want *all* builds to be clean, or at least as far as possible.

  Even so, Torvalds eventually applied this patch setting the WERROR configuration variable to match the value of COMPILE_TEST to minimize the pain felt by kernel builders worldwide. Yet another patch silences the warning GCC issues when the name main is used for anything but a function.

• When the Clang compiler is used to build the kernel, the LLVM integrated assembler will also be used by default.
• The SLUB allocator has seen a significant rewrite to minimize its disabling of interrupts and make it compatible with realtime preemption. See this changelog for some more information.
• It is no longer possible to create executable mappings with ioremap().
• Much of the kernel documentation has been translated into traditional Chinese.

Notably not pulled was the folio patch set; see this article for more information on that subject.

In general, this merge window has been relatively difficult — or, as Torvalds put it: "This has not been one of those smooth merge windows that we occasionally have". Hopefully the stabilization period will be a bit less rough. If all goes well and the nine-week schedule sticks, we'll have the final 5.15 release as an extra Halloween treat on October 31.

Comments (4 posted)

Roundup: managing issues for 20 years

The Roundup Issue Tracker is a flexible tool for managing issues via the web or email. However, Roundup is useful for more than web-based bug tracking or help-desk ticketing; it can be used as a simple wiki or to manage tasks with the Getting Things Done (GTD) methodology. The 20th-anniversary edition of Roundup, version 2.1.0, was released in July; it is a maintenance release, but there have been a number of larger improvements in the last year or so. Here we introduce Roundup's features along with the recent developments that have helped make Roundup even more useful for tracking issues to their resolution.

Background

Roundup manages one or more "trackers", which is where the issue handling happens. Roundup and a tracker provide web, email, and command-line interfaces for interacting with issues. Each tracker managed by Roundup has its own users, schema, interfaces, and behaviors. The Roundup core code enforces few limitations on a tracker; one could support email notifications while another disables notifications. Trackers exist for managing software bugs and/or feature requests, sales leads, projects, conference-paper submission and review, interaction with a customer, and more.

A tracker "template" is an un-deployed tracker; it includes the various pieces that make up the tracker, including database schemas, configuration settings, HTML template files, library routines, initial data, enhancements to the Roundup core, and more. The roundup-admin command-line tool is used to deploy and activate templates. Roundup provides three different templates for software development projects. They add properties for tracking software version, dependencies between issues, and more. These templates address use cases similar to those of Bugzilla, Request Tracker, or Jira.

The objects in a tracker's schema store specific information. For example, the required user object stores properties (email addresses, login name, real name, password, etc.) about the people who interact with the tracker. An issue object stores a title, list of messages, list of attachments, and the "nosy" list of users who should be notified about issue updates. It also stores information about the creation and last change to an issue.

Roundup's schema is specified in Python. The hyperdatabase (hyperdb) layer implements the database handling, which automatically deals with schema changes by modifying the underlying database. It also enforces role-based access control using its permission model, which provides fine-grained control down to the property level.

Creating a tracker based on the classic template, for example, adds more issue properties. These include the user responsible for resolving the issue, the current priority selected from a list of priorities, the current status from a list of statuses (e.g. unread, chatting, need-eg, resolved), and a list of keywords/tags (selected from a list) to categorize the issue. All of these predefined lists are editable from the web interface.

A tracker processes HTML template files to generate the web interface; it should be noted that there are two distinct uses of the word "template" for Roundup, tracker templates for creating trackers versus the HTML variety mentioned here. The HTML generated from the templates is customizable to mirror changes to the schema; the operations available in the web interface can be enhanced, as well. For example, an action can be added to export a query result as a comma-separated value (CSV) file, which provides information for further analysis in a spreadsheet or database.

Tracker behaviors are not hard-coded in the core. Bits of Python called auditors and reactors add behaviors. Auditors run before changes are committed to the database to validate changes like user login names, manage the notification list, prevent tracker spam, or implement a status workflow. Reactors run after changes are committed to the database. They send notifications, update RSS feeds, and can trigger calls to webhooks on other systems. More examples are available on the wiki and in the source repository.

Roundup is easy to deploy. CGI, WSGI, Zope, or standalone server are all supported deployment methods. For low-traffic sites, using an embedded database simplifies deployment. For larger sites, Roundup can use single sign-on when placed behind a web server like Apache or NGINX and can also integrate with LDAP or Active Directory. Because trackers are easy to deploy, organizations can run more than one of them to support different sets of users or to track different types of issues.

Roundup users

Many different projects and organizations have used Roundup for more than a decade. DARCS version control has been using it for 16 years as a bug and feature tracker. For 15 years, Cendio AB has run its customer support help desk using Roundup. Intevation GmbH has used Roundup for 19 years; the company runs multiple trackers for software development (some with scrum-like support), customer relationship management, internal help desk, and customer support (via email only or email and web access).

In addition, the Department of Laboratory Medicine and Pathology at the University of Washington Medical Center has run the tool for 13 years to handle problem reporting, tracking computing resources, customer support, and recording chemistry instrument quality assurance data. The Python Software Foundation (PSF) used Roundup for Python bug and feature tracking starting in 2006. But the PSF is now transitioning to GitHub Issues since Python's version control migrated to GitHub in 2017. Lastly, Roundup itself uses it for bug and feature tracking.

As mentioned, people interact with Roundup using the web interface, but REST and XML-RPC interfaces are also available. Roundup accepts issue updates via email, which can be PGP signed; Roundup is able to verify the signatures and can also sign email that it sends. The email interface allows changing an issue's properties in addition to adding a message or attachments. The roundup-admin command-line shell and Python language support allow users to write scripts for local interaction and automation.

Origins

Ka-Ping Yee designed Roundup. His design won the "Track" category of the Software Carpentry competition in 2000. In 2001 Richard Jones was looking for a new project and chose Roundup.

Roundup was originally written in Python 2, but currently supports Python 2.7.2+ or Python 3.4+ using only the Python standard library. Extra functionality (improved full-text search, markdown support, email encryption, better HTTP compression, etc.) is enabled by installing additional modules.

Originally, Roundup used the dbm key-value store for data storage. Relational database support was added later. Today it runs with Berkeley DB, SQLite, PostgreSQL, or MySQL/MariaDB.

Code from the Zope templating attribute language (TAL) processes templates to generate HTML. Today Roundup HTML templates support Zope TAL, Chameleon (an alternate TAL implementation), and Jinja2.

Bundled HTML templates use basic HTML and CSS; the goal is to allow customization by administrators who are not developers. The customizing document provides many examples for administrators to customize the web interface. JavaScript provides progressive enhancement and isn't required for core functionality; this provides a low barrier for users, who can even use a text-based browser, to interact with Roundup.

Recent development

Around 2011, Jones stepped back as the primary developer; Ralf Schlatterbeck and Anatoly Techtonik stepped in to update and release Roundup. Four years ago, I became the release engineer and a Roundup developer.

Two improvements significantly modernized Roundup. The 2.0.0 release in July 2020 added Python 3 support, which was driven by Joseph Myers with help from Christof Meerwald. The changes were made using the method suggested by Peter A. Donis and Eric S. Raymond, which allowed the code base to support Python 2 (for backward compatibility) and Python 3.

The second major improvement in that release added a REST API to complement Roundup's existing XML-RPC interface. Chau Nguyen, who was mentored by Ezio Melotti, developed it as a Google Summer of Code (GSoC) project for the PSF in 2015. Schlatterbeck merged the code into the core in 2020 for the 2.0.0 release. Features to better support JavaScript web frameworks like React, Angular, and Vue were also added. These include entity tag (ETag) support, field embedding, pagination, and Post One Exactly. Extending the REST interface allows custom support for front-end applications; for example, that can be used to render markdown for the front end.

Adding JSON Web Token (JWT) support using PyJWT provides a mechanism to integrate third-party applications with a tracker. A JWT authenticates the user as well as limits their permissions. For example, it could limit a third-party time tracking application to only creating time log entries on issues. The user interface has also seen improvements. Meerwald and Cédric Krier added support for formatting issue updates using Markdown to the jinja2 template. SimpleMDE provides a WYSIWYG-like editing environment for updates. The server then converts the Markdown to HTML for display. Although it has a few rough edges, the Markdown support works well.

The current development version has a patch that makes attaching files easier. Roundup has always supported file attachments using the standard HTML file input, but the patch would allow files to be attached using drag and drop. In addition, images (e.g. screenshots in the clipboard) can be pasted into the issue. While this work was not complete in time for the 2.1.0 release, it is running on the Roundup tracker to get feedback. Hopefully, the work will be part of the bundled tracker templates in version 2.2, which will likely land in July 2022.

Roundup's development version also compresses web responses to improve performance. A front-end server can compress Roundup's responses, but setting this up is easy to overlook (in fact bugs.python.org did not have this configured). A feature request was filed; now the development code compresses dynamic and static data (CSS, JavaScript, and so on) using gzip. Alternatively, Brotli or zstd compression can be used, but those require packages from PyPI.

Getting involved

Roundup can be installed using pip. Running roundup-demo will install and start a demonstration tracker. If you want to try Roundup without installing it, you can download and unpack the tarball, then run python demo.py. The roundup-users mailing list is the primary resource to get help in deploying and using Roundup. Help is also available on the #roundup IRC channel, on irc.oftc.net.

Anybody can contribute to Roundup. You don't need to be a professional developer. Many improvements have come from Roundup users and administrators. My background is system administration, not development, but in 2002 I needed a tracker to coordinate work at my job and Roundup fit the bill. I decided to implement the multiple notation methods from Request Tracker for Roundup. While learning Python and Roundup, I was able to create this with 20 hours of effort. Since Roundup was at version 0.5 this included debugging problems in the core code.

People with writing skills can help with Roundup documentation. The documentation is extensive but grew organically; while the answer to a question is often in the documentation, finding it is another matter. Efforts to restructure, index, and edit the documentation are more than welcome. For those interested in smaller documentation tasks, look to the Roundup wiki; indexing and copy editing the wiki will improve the user's experience. Roundup currently has translations for 11 languages; additional translations and corrections to existing translations are welcome as well.

The HTML templates are being changed to work with a strict Content Security Policy (CSP). Also, the look and feel of many HTML templates are stuck in the early 2000s. Redesigning the web interface to be responsive and accessible with a modern look and feel would address a common complaint. Extending support for non-SQL databases beyond the current dbm key-value store was suggested a few years ago. Enhancing Roundup to support MongoDB, Couchbase, or LMDB would be a useful addition.

You can browse the issue tracker for starter tickets; many more opportunities of items to work on exist in the Roundup issue tracker. The Roundup development mailing list (roundup-devel) is the place to go for information on developing Roundup. Those looking for an issue tracker for their project should give Roundup a try, while those who are interested in getting involved will find plenty of opportunities within the Roundup project.

Comments (7 posted)