|
|
Log in / Subscribe / Register

Leading items

Welcome to the LWN.net Weekly Edition for April 21, 2022

This edition contains the following feature content:

This week's edition also includes these inner pages:

  • Brief items: Brief news items from throughout the community.
  • Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Fedora considers deprecating legacy BIOS

By Jake Edge
April 20, 2022

A proposal to "deprecate" support for BIOS-only systems for Fedora, by no longer supporting new installations on those systems, led to a predictably long discussion on the Fedora devel mailing list. There are, it seems, quite a few users who still have BIOS-based systems; many do not want to have to switch away from Fedora simply to keep their systems up to date. But, sometime in the future, getting rid of BIOS support seems inevitable since the burden on those maintaining the tools for installing and booting those systems is non-trivial and likely to grow over time. To head that off, a special interest group (SIG) may form to help keep BIOS support alive until it really is no longer needed.

Proposal

The proposal to "Deprecate Legacy BIOS" was, as usual, posted on behalf of its owners, Robbie Harwood, Jiří Konečný, and Brian C. Lane, by Fedora program manager Ben Cotton. It currently targets Fedora 37, which is due in October, though there is reason to believe the change will not happen quite that soon. The reasons for removing the support are described in the proposal:

UEFI is defined by a versioned standard that can be tested and certified against. By contrast, every legacy BIOS is unique. Legacy BIOS is widely considered deprecated (Intel, AMD, Microsoft, Apple) and on its way out. As it ages, maintainability has decreased, and the status quo of maintaining both stacks in perpetuity is not viable for those currently doing that work.

[...] While this will eventually reduce workload for boot/installation components (grub2 reduces surface area, syslinux goes away entirely, anaconda reduces surface area), the reduction in support burden extends much further into the stack - for instance, VESA support can be removed from the distro.

The proposal says that, eventually, BIOS support will have to be removed, but that maintaining the ability to boot existing Fedora systems with BIOS is meant to help smooth the process. Fedora already has requirements that effectively restrict it to systems made after 2006, so this change would extend those restrictions somewhat further.

Intel stopped shipping the last vestiges of BIOS support in 2020 (as have other vendors, and Apple and Microsoft), so this is clearly the way things are heading - and therefore aligns with Fedora's "First" objective.

The subject was raised back in 2020, which also led to a (predictably) long thread, though it was not a change proposal at that time. Some of the "relevant points from that thread" are listed in the Feedback section of the proposal. There are, of course, machines that are BIOS-only and any kind of hardware deprecation for the distribution is impossible "without causing some amount of friction". In addition, there is no way to migrate from a BIOS installation to a UEFI-based system, since "repartitioning effectively mandates a reinstall". In particular, a UEFI partition would need to be added to the system.

The Contingency Plan section of the proposal describes an ugly future for the status quo, but also a possible way forward:

Leave things as they are. Code continues to rot. Community assistance is required to continue the status quo. Current owners plan to orphan some packages regardless of whether the proposal is accepted.

Another fallback option could be, if a Legacy BIOS SIG organizes, to donate the relevant packages there and provide some initial mentoring. Longer term, packages that cannot be wholly donated could be split, though it is unclear whether the synchronization thereby required would reduce the work for anyone.

Affected systems

As might be guessed, multiple replies from users of affected systems were seen in the thread, starting with Neal Gompa. He said that he is sympathetic to the change, but thinks that it is "way too early to do across the board". He has a system that struggles to boot Linux with UEFI and his workarounds are beyond the abilities of most users.

David Airlie said that he recently worked on a project to rewrite the Mesa driver for a wide range of Intel GPUs. Many of the systems he used to develop and validate those drivers are pre-UEFI systems, but it was "of great benefit to me and the community" that he could use Fedora for that work. For future projects, he would have to consider moving away from Fedora, which would be a sad state of affairs.

Hans de Goede said that he had several systems at home that were BIOS-only and happily running Fedora now. Obsoleting them just contributes to the e-waste problem; "Looking specifically at fixed PCs and not laptops this proposal would (eventually) turn 2/5 PCs in my home unusable, which really is unacceptable IMHO." He also pointed to Airlie's work on Intel GPUs, saying that "it seems rather silly to drop support for this hw after just investing a significant chunk of time to breathe new [life] into their GPU support"

But it is not just desktops and laptops that are affected by a change of this sort. Fedora is also installed on cloud servers and virtual machines of various sorts, some of which do not support anything other than booting via BIOS. The proposal noted that the time of the 2020 discussion, Amazon's AWS did not support UEFI, but that has changed. Marc Pervaz Boocha pointed out that many virtual private server (VPS) providers do not support UEFI, giving Linode and Vultr as examples. Dominik "Rathann" Mierzejewski reported that OVH is also affected:

OVH is another big provider and they don't offer UEFI boot with their VPS range. I've just confirmed it with their support.

Since one of my servers is hosted by OVH, I'd have to either migrate to another hosting provider or migrate off Fedora. Which is ironic, considering my involvement in Fedora.

Stewart Smith added some "thoughts both from an EC2 [Elastic Compute Cloud] perspective, and an Amazon Linux as a downstream of Fedora perspective". Most EC2 instance types boot using BIOS by default, though many can use UEFI and new types are likely to get UEFI support, but there are "a *lot* of instance types, a whole bunch of which are less likely to support UEFI". There is no installer for cloud images, instead they use the Amazon Machine Image (AMI) format, but "AMIs that don't run on all instance types tend to cause confusion, no matter how [clearly] you document the limitations". So Amazon Linux has an interest in ensuring that BIOS booting still works well, "likely for a decent number of years to come (however much I wish this wasn't the case)".

Gompa complained that the change is not really a deprecation, but is instead a removal, because the lack of packages and tooling that support BIOS "makes several scenarios (including recovery) harder". It also puts the burden on users to determine if their hardware can boot and install Fedora, but UEFI has not yet reached a critical mass where it can be assumed to work. Alberto Abrao agreed, noting that this change would "leave behind a LOT of serviceable hardware", especially in the server space:

Ironically, Fedora is one of the distributions out there that allows me to extract the most out of older hardware. It would be a terrible loss to have to move to a different one, but it's hard to reason purchasing new hardware - especially right now, with pandemic-related supply issues still ongoing - to keep up with this change.

In that message, Gompa also said that he is a "a fan of using UEFI instead of BIOS" and that he has done work to add UEFI support to Fedora cloud images. He just does not believe that it is time, yet, to make that switch. Part of the reason is that he believes the UEFI experience on Fedora is not all that good.

A wander into secure boot

In his original reply to the proposal, Gompa also asked about Fedora support for NVIDIA drivers under UEFI secure boot. Peter Robinson said that was out of the scope of the proposal, since users can disable secure boot if they need support for drivers, like NVIDIA's, that are not signed by the Fedora kernel-module keys. But Gompa replied that it is sometimes easier to have users fall back to booting from BIOS; furthermore:

You're right that these are different problems, but I've also seen very little appetite for reducing the suffering of Fedora Linux users on UEFI Secure Boot with the *most common issue* we have: an NVIDIA driver that doesn't do anything because of the lockdown feature. If you're planning to say that UEFI is the only way to boot, then that means you need to be prepared to accept that our UEFI experience is *worse* than our BIOS one right now, and someone needs to take ownership to improve it.

Harwood, who is one of the feature owners, said that NVIDIA users can either use the open-source nouveau driver (which is signed), sign their own copy of the driver "(involves messing with certificates, so not appropriate for all users)", or disable secure boot. But several people pointed out that nouveau does not really solve the problem, especially for more recent NVIDIA hardware; it does not provide access to accelerated graphics operations for recent GPUs and tends to have quite a few bugs even for the older models. Michael Catanzaro pointed out that the options Harwood described are problematic from a user-experience perspective:

The user experience requirement is: user searches for NVIDIA in GNOME Software and clicks Install. No further action should be necessary. We didn't make the NVIDIA driver available from the graphical installer with the intention that arcane workarounds would be required to use it.

But Adam Jackson thought that the NVIDIA problem was not Fedora's to solve. There are technical means to make it work and "NVIDIA are the ones with the private source code". But Chris Murphy sees things differently, noting that Fedora is sending mixed messages:

When users have a suboptimal experience by default, it makes Fedora look bad. We can't have security concerns overriding all other concerns. But it's really pernicious to simultaneously say security is important, but we're also not going to sign proprietary drivers. This highly incentivizes the user to disable Secure Boot because that's so much easier than users signing kernel modules and enrolling keys with the firmware, and therefore makes the user *less safe*.

On the other hand, Fedora does not actually have a full secure boot implementation, Lennart Poettering said, so by disabling it "you effectively lose exactly nothing in terms of security right now". The reason is that the initrd is not signed, so it can be subverted:

What good is a trusted boot loader or kernel if it then goes on loading an initrd that is not authenticated, super easy to modify (I mean, seriously, any idiot script kiddie can unpack a cpio, add some shell script and pack it up again, replacing the original one) – and it's the component that actually reads your FDE LUKS password.

SIG

In his message linked above, De Goede volunteered to form a SIG along the lines suggested in the proposal. He noted that previous attempts to keep 32-bit x86 alive when Fedora removed support for i686 had run aground, but felt that BIOS is different:

Legacy BIOS boot support is basically only about the image-creation tools + the bootloader. As various people have mentioned in the thread BIOS support is still very much a thing in data-centers, so I expect the upstream kernel community to keep the kernel working with this for at least a couple of years. Where as both the kernel + many userspace apps were breaking on i686.

After Fedora project leader Matthew Miller started a new thread about the possibility of having a video call to discuss the BIOS issue, which was generally seen as not being something that would do much more than rehash what had already been aired, De Goede renewed his call for a Legacy BIOS SIG. He envisioned it as a lightweight organization that focused on testing Fedora on BIOS systems, particularly the next version of Fedora early in its development, in order to file and fix bugs.

The new thread largely went over the same ground as the first, at some length, naturally, though there was also some concrete planning of what a new SIG would do—and how. Harwood said that what is needed is a place to assign bugs and a way to get those problems addressed. "The overall goal of the SIG needs to be to reduce load on existing bootloader contributors." But Harwood also wondered whether Fedora should redefine its support for BIOS:

Given there is consensus that legacy BIOS is on its way out, we think Fedora release criteria in this area should be re-evaluated. Not only does support change from "fully supported" to "best effort", but we should re-evaluate what is/isn't release blocking, and probably clarify who owns what parts.

Several people disagreed with that characterization of the status of BIOS. Chris Adams said: "I don't think this statement is true, unless Fedora doesn't want to be considered for a bunch of popular VM hosts (e.g. Linode and such) that have no stated plans to support UEFI." Perhaps BIOS support for physical hardware could be phased out, though. De Goede agreed that BIOS support is still needed in some contexts:

Given what the server product folks have indicated that BIOS boot support is quite important for them I'm not sure if changing the release criteria is in order. I do agree that any blocker bugs related to legacy BIOS booting should be assigned to; and taken care of by the legacy BIOS boot SIG.

The change proposal will be discussed and decided by the Fedora engineering steering committee (FESCo), but its prospects do not look good. The FESCo ticket for the proposal has already gotten five "-1" votes from members and there are nine on the committee. Those votes are not binding, but it still looks pretty unlikely this change will go through—at least for Fedora 37.

But the change will, seemingly, happen eventually. One of the complaints seen in the ticket (and threads) is that the proposal calls it a deprecation, but that is not really what it entails; new versions of Fedora will not be able to be installed on the older hardware, which could be problematic if an in-place upgrade goes awry. In addition, Fedora versions are only supported for about a year, so at some point upgrading may not really be an option if BIOS support is removed. That means users of those systems would have to migrate to a different distribution or keep running an unsupported version as it slowly bitrots. In a blog post about the change, Cotton pointed out that while he did not think the distribution "should abandon old hardware willy-nilly", there is a balance to be struck:

I think some distros should strive to provide indefinite support for older hardware. I don't think all distros need to. In particular, Fedora does not need to. That's not what Fedora is. "First" is one of our Four Foundations for a reason. Other distros focus on long-term support and less on integrating the latest from upstreams. That's good. We want different distros to focus on different benefits.

With luck, the SIG will take over any packages needed to keep BIOS functioning, since their owners plan to orphan them regardless of the outcome of the proposal. Keeping BIOS alive on Fedora for another few years—how many is difficult to guess—would seem to have a fair number of benefits at this point, though there is obviously a maintenance burden associated with it as well. If the change does not get approved this time around, we will likely see the proposal recur for Fedora 38 (or later), but if the SIG takes off, that may be postponed for some time. When it does come up again, however, we can probably expect another lengthy discussion in Fedora-land.

Comments (87 posted)

Super Python (part 1)

By Jake Edge
April 19, 2022

A mega-thread in the python-ideas mailing list is hardly surprising, of course; we have covered quite a few of them over the years. A recent example helps shine a light into a dark—or at least dim—corner of the Python language: the super() built-in function for use by methods in class hierarchies. There are some, perhaps surprising, aspects to super() along with wrinkles in how to properly use it. But it has been part of the language for a long time, so changes to its behavior, as was suggested in the thread, are pretty unlikely.

Inheritance and super()

Perhaps a bit of an introduction to Python inheritance is in order, at least for some readers. Python is generally written in an object-oriented fashion, with classes that can incorporate behavior from other classes via various mechanisms. One of those ways is via inheritance: one class directly references a parent class and gets some of its behavior from the parent. That can look like the following: 

    class Shape: pass
    
    class Ellipse(Shape): pass

    class Circle(Ellipse): pass

The idea is that inheritance is a way to express an "is a" relationship; in the above, Circle is an Ellipse, which is a Shape, thus, transitively, Circle is also a Shape. Implicitly, Shape is an object, which is the base for all classes in Python. Behavior and attributes that are common to all shapes would be added to a parent class and those would be inherited by all of the descendant classes.

Perhaps this class hierarchy is being used in some sort of graphical display program, so all shapes might have a position and color as attributes. For behavior, shapes might need some basic operations, such as move() and draw(), but those are somewhat beyond the scope of what is needed to explain inheritance. If we modify Shape a bit, we can see this inheritance in action: 

    class Shape:
	def __init__(self, x, y, color):
	    self.x = x
	    self.y = y
	    self.color = color

    c = Circle(x = 0, y = 0, color = 'blue')
    print(c.color)  # prints "blue"

The __init__() function initializes a shape with the given attributes and Circle inherits that initialization routine from Shape (by way of Ellipse); Python automatically follows the hierarchy when it cannot find a function and uses the first that it finds. The call to Circle() creates an instance of that type and initializes it using the __init__() from Shape; the result is a Circle object with three instance variables (x, y, and color).

But what if Circle needs to be initialized with additional information specific to that kind of shape? Maybe it needs a radius; a somewhat simplistic way to add that might be the following: 

    class Circle(Ellipse):
	def __init__(self, x, y, color, r):
	    self.r = r
	    Ellipse.__init__(self, x, y, color)

Now, a Circle gets initialized with four parameters; it peels off one and calls its parent class's __init__() explicitly. Given the example so far, it could simply go straight to the __init__() method of Shape, but that is generally a bad practice; Ellipse may well have (or get) an __init__() of its own. But now, if the class hierarchy is refactored in some way and a class is inserted between Ellipse and Circle, or Circle gets some other parent, it needs to be changed in the class definition and every member function that specifically refers to Ellipse. The super() function is meant to help with that: 

    class Circle(Ellipse):
	def __init__(self, x, y, color, r):
	    self.r = r
	    super().__init__(x, y, color)

super() is effectively a proxy for the parent class, without needing to name it directly. Should the hierarchy change, the new parent class just needs to be named once in the class definition. In a straightforward single-inheritance hierarchy like this, it is clear the path that Python will take to resolve super(); it simply walks the hierarchy upward until it finds a parent that implements the member function in question.

super() and multiple inheritance

But with multiple inheritance, things are a bit different. If we have a new class, that inherits from Circle and another class, there might be some question how methods get resolved—and in what order. If we torture our example a bit further, we might have: 

    class Circle(Ellipse):
	def __init__(self, r, **kwargs):
	    self.r = r
	    super().__init__(**kwargs)

    class Shape3D(Shape):
	def __init__(self, z, **kwargs):
	    self.z = z
	    super().__init__(**kwargs)

    class Sphere(Circle, Shape3D):
	def __init__(self, **kwargs):
	    super().__init__(**kwargs)

So Sphere inherits from both Circle and the new Shape3D class, which in turn inherits from Shape. Since we are using keyword arguments to initialize the objects, using **kwargs allows the __init__() functions to just peel off the arguments of interest to them and pass the rest up the chain via super(). A Sphere could be created as follows: 

    s = Sphere(x = 0, y = 0, z = -2, r = 2, color = 'red')

In order for that all to work, both parents of Sphere need to have their initialization routines called, and that is exactly what happens. Python has a method resolution order (MRO) that was adopted for Python 2.3 back in 2003. It uses C3 linearization to determine how to order the class hierarchy in a deterministic way that preserves the relationships between the classes; if no such ordering can be found, Python will raise an error when the classes are defined. The class.mro() method can be used to examine the MRO; for Sphere it looks like this: 

    [<class '__main__.Sphere'>, <class '__main__.Circle'>,
    <class '__main__.Ellipse'>, <class '__main__.Shape3D'>,
    <class '__main__.Shape'>, <class 'object'>]

That list shows the order in which the classes will be consulted when trying to resolve a method (or class attribute) for Sphere. It also indicates the path that super() will walk as it works its way through the hierarchy.

super()-thread

It is against that backdrop that Martin Milon ("malmiteria") posted to python-ideas. He said that the MRO is effectively masking problems that the programmer should be required to resolve manually:

in case of multiple [inheritance], resolving a child method from it's parent isn't an obvious task, and mro comes as a solution to that. However, i don't understand why we don't let the programmer solve it. I think this is similar to a merge conflict, and not letting the programmer resolve the conflict feels like silencing an error. This is especially infuriating when you realise that mro doesn't solve all possible scenarios, and then, simply refuses the opportunity to solve it to the programmer. Then, super relying on mro gives off some weird behaviors, mainly, it's possible for a child definition to affect what a call to super means in it's parent. This feels like a side effect (which is the 'too implicit' thing i refer to).

He had a proposed solution to that problem, which is further described in a GitHub project repository, that he calls "explicit method resolution" (EMR). It would make no changes for single inheritance, but would force developers to use a new __as_parent__() method to explicitly choose a particular implementation of a method from among multiple possibilities in the parents, grandparents, and so on of a class. If that is not done, an exception would be raised.

But explicitly choosing the method resolution is already available in Python, David Mertz said: "You don't need `super()` to call `SomeSpecificClass.method(self, other, args)`". Milon agreed, but said that "super acts as a proxy to a parent, and is by default the feature to refer a parent from within a class method". Christopher Barker pointed out that super() is actually a proxy for the parents, not just a parent; "even if the current class inherits from only one class, that superclass may inherit from more than one class".

Milon gave a simple example, which did not use super() at all, that shows how he thinks the MRO is causing problems: two classes, A and B both implement method() and class C inherits from both (class C(A, B)):

Today, a code such as ```C().method()``` works without any problems except of course when the method you wanted to refer to was the method from B. If class A and B both come from libraries you don't own, and for some reason have each a method named the same (named run, for example) the run method of C is silently ignoring the run method of B.

The C3 mechanism uses the order in which two or more parent classes are listed in a class definition to determine the MRO. But Milon is saying that having two methods in the parent classes constitutes a "conflict" that should be resolved by the programmer. As he said in another message: "My point is that : it is not correct to assume it is meaningful to give an order to multiple [inheritance]". Stephen J. Turnbull disagreed:

[...] of course it's *meaningful* to give an order to multiple inheritance. Sometimes that order is *not useful*. Other times it may be *confusing*. Pragmatically, C3 seems to give useful results frequently, unuseful or confusing results infrequently [...]

Steven D'Aprano had a more fundamental objection to the example: "Why are you inheriting from A if you don't want to inherit from A?" He noted that A and B were not designed for multiple inheritance, which is the basis of the problem.

Multiple [inheritance] in Python is **cooperative** -- all of the classes in question have to work together. If they don't, as A and B don't, bad things happen.

You can't just inherit from arbitrary classes that don't work together. "Uncooperative multiple inheritance" is an unsolvable problem, and is best refactored using composition instead.

In that earlier message, Milon also showed an example to better explain where he thinks super() is confusing:

super is not a proxy to parents, even plural, it's a proxy to the next in mro order.

in this case : 

class Top:
    def method(self):
        print('Top')
class Left(Top):
    def method(self):
        print('Left')
        super().method()
class Right(Top):
    def method(self):
        print('Right')
        super().method()
class Bottom(Left, Right):
    def method(self):
        print('Bottom')
        super().method()
Bottom().super() would print "Bottom", "Left", "Right", "Top". super, in Left, [refers] to Right, which is not a parent of Left (and is completely absent of it's definition)

He presumably means Bottom().method() there toward the end. That result might be surprising to some—possibly even confusing—but the super() call in Left uses the MRO context of Bottom, which is where the original super() call was made. The MRO for Bottom is: Bottom, Left, Right, Top, object. In that context, Left.method() will call Right.method() via super(). So the result is what many, but probably not all, Python programmers would expect. super() allows for an automatic way to pass a method call up through a set of cooperating classes; as Barker put it: "If you didn't use super in the whole chain— it would stop when it found the method. If you do, then they are all [called], and each one only once." In order to use super(), programmers have to follow some rules; "And then it works predictably. Whether that's helpful or not depends on your use case."

Some of those "rules" are embodied in two classic blog posts, as D'Aprano pointed out. One is "a provocatively titled blog post called 'super considered harmful'" by James Knight, though the title has changed. D'Aprano said that after some discussion, Knight backed down a bit, but still said: "super is great, 'but you can't use it.'". D'Aprano disagreed: "Except of course thousands of people do use it. And it works." The other blog post is Raymond Hettinger's "Python's super() considered super!", which describes how to properly use the feature.

D'Aprano followed up with another group of links to multiple blog posts by Michele Simionato, who wrote the MRO documentation for Python 2.3 linked above. There are two series, one on super() and one on Mixins, and some other topics, including multiple inheritance in various forms. D'Aprano said:

Multiple inheritance in Python works the way it does because it is modelling cooperative MI and the MRO and super are the Right Way to handle cooperative MI.

That doesn't mean that cooperative MI doesn't have problems. Other languages forbid MI altogether because of those problems, or only allow it in a severely restricted version, or use a more primitive form of super. MI as defined by Python is perhaps the most powerful, but also the most potentially complicated, complex, convoluted and confusing.

The discussion, links, and such had given Milon some food for thought, which he appreciated. He planned to regroup and come back with a clearer proposal that incorporates elements from the discussion and tries to address a persistent call for "real life" examples, rather than the somewhat abstract examples he had been using. He posted the results on April 3, which led to the second "half" of the mega-thread. That will provide more opportunities to dig into super() and other Python inheritance topics, which is coming in part 2 of our tale.

Comments (12 posted)

Rustaceans at the border

By Jonathan Corbet
April 14, 2022
Support for developing in the Rust language is headed toward the kernel, though just when it will land in the mainline is yet to be determined. The Rust patches are progressing, though, and beginning to attract attention from beyond the kernel community. When two languages — and two different development communities — come together, the result can be a sort of cultural clash. Some early signs of that are appearing with regard to Rust in the kernel; if the resulting impedance mismatches can be worked out, the result could be a better development environment for everybody involved.

The latest round of Rust patches was posted by Miguel Ojeda on March 17. This time around, Rust support has moved forward to version 1.59.0 of the Rust language, which has stabilized a couple of important (for the kernel) features. The patches add a new module abstracting access to hardware random-number generators. A CString type has been added for C strings. The spinlock implementation has been improved. All told, the patch series, which can be found in the linux-next repository, adds over 35,000 lines of code and documentation; it is a considerable body of work.

There has been no public discussion on just when these patches might be deemed ready to go into the mainline kernel. Rust support is still considered "experimental" even by its developers; that is likely to remain the case for some time (even after this work is merged into the mainline) until the language proves itself for kernel development.

Clearly, though, some developers are beginning to play with it — and they are not all traditional kernel developers. Recently, Nándor Krácser asked on the Rust-for-Linux mailing list about the possibility of including Rust modules from the crates.io repository into the kernel build. This request, seemingly, is not just for small stuff:

Currently I'm experimenting with different crates which I would like to use in my module, serialization libraries, math libraries. etc, even complex ones, are really hard to pull in as a direct source library (copy the code to the module), and if they have a transitive dependency that complicates things even more.

Shortly thereafter, Chris Suter showed up with a similar request. Rust developers working with kernel modules, it seems, want more functionality than the current kernel crate provides to them.

This should not be entirely surprising. Like many newer languages, Rust is closely tied to a language-specific package-management system and associated central repository; in this case, the Cargo package manager and crates.io. Developers in such languages quickly become accustomed to pulling in new modules (and any dependencies they may have) with a simple command, and to having the build system make dependencies magically appear when building a new program. For these developers, the idea of working in an environment where complex libraries are not obtainable with a few keystrokes starts to have a distinct lack of appeal after a while.

The kernel does not work in this way, though. To those of us who didn't grow up with that kind of development environment, it looks like a recipe for bloat, bugs, and security problems. Depending on central repositories opens up a project to problems like the famous leftpad incident or, worse, the deliberate insertion of malicious software. A lack of attention to API compatibility leads to a thicket of version requirements and dependency-resolution problems so complex that machine-learning systems are emerging to deal with them. Plus it all just looks so undisciplined and messy.

At least some of the criticisms of this mode of development are valid, but it's also not hard to detect a bit of Stockholm syndrome as well. For many of us, for much of our careers, building a new program from source was likely to involve a lengthy cycle of "try to build, figure out which dependency it wants now, install the dependency" iterations — and recursive iterations at that when the dependencies turn out to have missing dependencies of their own. This exercise helped us to understand our systems better and must somehow have helped us to build better moral character, so we can't understand why Kids These Days just don't want to live that way.

The kernel community seems more than usually likely to have developers who are resistant to newer methods of development. The kernel has to stand alone, and its developers keep a firm grip on its dependencies. The kernel repository contains all of the code needed to build a working kernel; developers can be expected to install a limited set of tools to do the build, but the idea of installing external libraries to build into the kernel would not go over well.

So when developers see a shopping list like the one posted by Suter:

Like I said, I'm interested in futures. Why it's useful: async Rust is arguably more common and easier to use than other forms of multi-threaded processing. Other crates that I'd like: anyhow, bincode, byteorder, log, once_cell, pin-project, rand, serde, slab, static_assertions, uuid plus some more esoteric ones.

The first temptation will be to either run and hide or to respond in a way that may not be compliant with anybody's code of conduct.

There are some good reasons for this. As Greg Kroah-Hartman pointed out, code that has been written to be useful in user space almost certainly does not work within the constraints imposed on kernel code. "Async Rust" knows nothing about kernel threads or how context switching is done in the kernel, for example. Kernel code must be extremely careful in how it allocates memory, must not use floating-point arithmetic, cannot store large data structures on the stack, and cannot use unbounded recursion, among many other rules. Most user-space code, which was not written with these rules in mind, will fare poorly in this environment. For this reason, Kroah-Hartman said that any functionality desired by Rust programs must be specially written and provided in the dedicated kernel crate.

The Rust-for-Linux developers understand this situation and are not envisioning adding the ability to pull in modules with a tool like Cargo. So it is interesting that a long-time kernel developer, Kent Overstreet, was the one to argue for a different approach. "The world is changing", he said, and perhaps it is time for the kernel community to change with it as well. There are numerous situations where it can be beneficial to run code in both user and kernel space, he said, and the fact that doing so is currently painful is a problem for developers on both sides:

The solution to problems like these are to stop thinking that kernelspace and userspace _have_ to be completely different beasts - they really don't have to be! and start encouraging different thinking and tooling improvements that will make our lives easier.

It is true that the boundary between kernel and user space has become more porous over the years. Various subsystems provide hooks that allow formerly kernel-specific tasks to be carried out in user space instead, while user space can use BPF to run code inside the kernel. But the two environments are still quite different, and code meant to run on one side generally cannot run on the other. There has not been a lot of effort put into thinking about how to reduce that divide; perhaps it really is time for that to change. The Rust language might just be the environment in which this transformation could happen. As Overstreet put it:

Rust's conditional compilation & generics story is _much_ better than what we've had in the past in C and C++, meaning writing Rust code that works in both userspace and kernelspace is much saner than in the past.

If an initiative like this were to work, it could greatly reduce the barrier to entry for future kernel developers while making a lot of useful code available to the kernel community. It would be a different kernel project than the one we know now, but it might be a more fun and more productive one.

Interesting things tend to happen when immigrants show up in a new land. They can often create a backlash among those who are already there — the new people dress funny and their cooking smells weird, after all, and some of them even have a crab as their mascot. But they can also bring energy and ideas that shake up their new home and make it richer for everybody involved. It may just be that we will see something like that happen if and when a crowd of Rust developers descends upon the kernel community. The end result could be difficult to recognize — and perhaps better than anything we had before.

Comments (151 posted)

User events — but not quite yet

By Jonathan Corbet
April 18, 2022
The ftrace and perf subsystems provide visibility into the workings of the kernel; by activating existing tracepoints, interested developers can see what is happening at specific points in the code. As much as kernel developers may resist the notion, though, not all events of interest on a system happen within the kernel. Administrators will often want to look inside user-space processes as well; they would be even happier with a mechanism that allows the simultaneous tracing of events in both the kernel and user space. The user-events subsystem, developed by Beau Belgrave and added during the 5.18 merge window, promises that capability, but users will almost certainly have to wait another cycle to gain access to it.

Kernel tracepoints are hooks at specific locations in the code. They are designed to add as little overhead as possible when they are not active, which is the case most of the time. When a tracepoint is activated, it produces a stream of structured data specific to the event being monitored; user space can read that data via a number of different interfaces. By turning on just the tracepoints of interest, user space can collect the data needed to analyze a specific situation without slowing down the kernel overall.

The user-events ABI

Belgrave's user-space equivalent to kernel-space tracepoints, merged for 5.18, requires a bit more work to support, though libraries provided in the future may ease some of that burden. The first step is to open a new file added to the tracefs kernel filesystem: 

    /sys/kernel/debug/tracing/user_events_data

A program then needs to register each event that it wishes to make available to the system. That is done by filling out this structure: 

    struct user_reg {
        u32 size;
        u64 name_args;
        u32 status_index;
        u32 write_index;
    };

The first two members are input parameters, while the last two are set by the kernel. The size parameter should just be the size of the user_reg structure itself; this helps to ensure compatibility if the structure grows in future kernel releases. The event itself is described by name_args, which is a pointer to a string; it uses a special format added with this patch set. The first token is the name of the event; the rest of the line describes the data reported for that event. So an event that reports an integer named level and a 20-character string named badness could be described as: 

   my-event u32 level; char[20] badness

This structure is then registered with an ioctl() operation on the previously opened user_events_data file, using the DIAG_IOCSREG command. On successful registration, the kernel will store two index values in status_index and write_index, the use of which will be described below.

Once the event is registered, it will show up in tracefs under the user_events subsystem. That means it can be activated, and its data collected, using any of the usual user-space tools. But to get there, the application must still provide that data when the time comes.

To do that, the program should open the other new tracefs file as well: 

    /sys/kernel/debug/tracing/user_events_status

That file should then be mapped into the program's address space with an mmap() call.

Like its kernel counterpart, the user-events mechanism has been designed to minimize its overhead when nobody is interested in the events. So the program implementing the events will only want to provide the data if it has been requested. The user_events_status file that was just mapped above will contain a byte of data indicating whether the event is active or not; its index will be the value stored in the status_index field during registration. If that byte is zero, the event is not active and the program should not output any data; that is expected to be the case most of the time.

When somebody attaches to the event, the associated byte will no longer read as zero. It is, in fact, a bitmap giving information about how the event has been attached; one bit corresponds to ftrace, while another is for perf. When the program sees that non-zero byte, it should write the data associated with the event to the user_events_data file opened at the beginning. The first four bytes of the written data should be the value the kernel stored in write_index at registration time; the rest will be the data as described. Typically, a writev() call will be needed to assemble the requisite bits.

That describes the bulk of the API. More information can be found in this documentation commit and this sample program. There is also, inevitably, a way to attach BPF programs to user events, but that feature is not described in detail in the documents.

Concerns

After this code was merged, Linux Trace Toolkit (LTTng) developer Mathieu Desnoyers posted some criticisms of the new interface. The byte-based status mechanism struck him as inefficient; providing a single bit for each event would allow for a more compact representation and, thus, better cache utilization. The multiple bits of information indicating how the events had been attached to have no real value to the application being traced, which should produce the same data regardless.

He had some other concerns as well. If the page(s) containing the data to be written for an event are forced out of memory, the resulting page fault will cause writev() to fail and, absent active countermeasures, the event data will be lost. The mechanism as a whole is built around access to tracing data via the kernel; it will only add overhead when purely user-space tracers (such as LTTng) are in use. There were a number of implementation concerns as well.

Desnoyers also brought the facility to the attention of BPF maintainer Alexei Starovoitov, who had been unaware of it. He was not happy with what he saw; he called for the BPF mechanism to be removed immediately: "It's a hard Nack to add a bpf interface to user_events". He has reiterated that position in subsequent discussions.

Belgrave quickly posted a patch removing the BPF feature, as requested. But it looks like that will not be enough for this feature to be enabled in 5.18. Tracing maintainer Steven Rostedt stated his agreement with Desnoyers, saying that he is considering marking the whole mechanism as "broken" so that the issues can be resolved. It is conceivable that Belgrave could address all of the concerns in this development cycle, but it is unlikely; that sort of work is not meant to go into the mainline after the merge window closes. So, chances are, users will have to wait until 5.19 for access to the new user-events tracing mechanism.

Comments (6 posted)

KOReader: a free electronic-book reader for e-ink devices

By Jonathan Corbet
April 15, 2022
Your editor has a certain tendency to accumulate books, to the point that they crowd everything else out of the house. There is a lot to be said for books: a physical book has a user interface that has been optimized over centuries, and one can have a reasonably high degree of certainty that any given book will still work a few decades from now. Neither of those can be said for electronic books, but they do have the advantages of taking less shelf space and being more portable. So electronic books are part of the reading menu, which naturally leads to the search for a free reader for those books; KOReader turns out to be an interesting alternative.

KOReader is distributed under version 3 of the GNU Affero General Public License. It is, seemingly, a highly portable application, being available for Linux, Android, Windows, and a number of dedicated reader devices. The Android version is available via F-Droid, which makes this version a good starting place for anybody wanting to try out KOReader without performing surgery on a more specialized reader device. In this setting, it is a worthy alternative to FBReader (which is also a reasonable application) and, unlike FBReader, it doesn't try to sell proprietary plugins.

Those of us who have been playing with Linux for a long time, though, remember the special adrenaline rush that comes with trying to install free software onto a device that wasn't intended to support modification by its owner. One does not really know a piece of software until one has given it the opportunity to turn a useful device into electronic waste. So naturally your editor went quickly from the Android application to installing KOReader on his Kobo device. The Kobo, as it turns out, is a relatively open device; the installation is just a matter of mounting it as USB storage and unzipping a couple of files into it. Then comes the long, sweaty-palms pause while the Kobo meditates on the new software while presenting a blank screen.

Turning the pages

Eventually that phase passes, the device comes back to life. KOReader, at least when this installation method is used, does not replace the existing Kobo interface; instead, it presents itself as if it were just another book waiting to be read. Entering the application (by telling the device to "read" the KOReader "book") yields a list of top-level directories on the device; there does not appear to be a [KOReader user interface] way to just list all of the books on the device short of just dumping them all into the root directory. The top-level view, in other words, is essentially a file browser. One moves through the directory hierarchy in the usual way; tapping on a book will open it to the cover image.

KOReader, it seems, must go through a process of rendering the entire book before it can present the first page. Asking it to read a large text book led to a long delay (for a "several minutes" value of "long") before the cover was shown. If one changes a parameter like the font size, the entire process begins again, making tweaking such parameters a less than fully rewarding experience. Neither FBReader nor the native Kobo reader exhibit this behavior. Rendering time for more reasonably sized books is not terrible, though.

User interfaces for electronic-ink devices must be developed with that medium's constraints in mind. Flashy animations just do not translate well, for example; the application needs to put up its control interfaces with a minimum of screen rewriting. KOReader has clearly been written with this kind of screen in mind, and the interface works well on the Kobo reader. This interface will be familiar to users of other book-reading applications. Tapping on the right edge moves forward one page; the left edge moves back. Swiping up or down the left edge adjusts the screen brightness. Tapping at the top yields information about the book and a plethora of menu options. It is possible to jump in and start reading books without having to work through the manual first.

[KOReader font adjustment] That said, a reading of the manual does have its rewards. One place where KOReader clearly stands out is in its extensive customization options, not all of which are obvious without a bit of supplementary description. Almost any electronic-book reader allows the tweaking of the font size and, perhaps, margins. KOReader lets one adjust just about anything that somebody might have thought of to play with. There are controls for font hinting and kerning, for example. Spacing within and between words can be tweaked, as can the line spacing. There is a set of menus to tweak the style sheets used for rendering or do things like suppress blank pages at the end of a chapter. Most of these tweaks (and more) can be assigned to any of a large set of gestures recognized by the software. If one wants to adjust the bottom margin with a two-finger diagonal (bottom right to upper left swipe), it is possible.

One truly useful feature is that the default value for most parameters can be set with a long tap. Every book seems to come with its own wacky font size; setting the default size eliminates the process of adjusting it at the beginning of every new book.

Beyond the usual EPUB, KOReader can handle PDF files (and a long list of other formats as well). The results are variable. PDF is a page-oriented format, and full pages tend not to render legibly on small reader screens, so some accommodations must be made. KOReader has a whole set of [KOReader PDF rendering] options to try to ease the process, starting with simply eliminating the margins on the page. There is a mode that will divide each page into squares and step through them in any of eight different sequences; the primary use case for this mode appears to be comic books. There's another mode that simply tries to identify comic panels (and step through them) automatically. KOReader can be asked to extract the text from the PDF and reflow it for the screen, an option that works reasonably well for some PDFs that contain recognizable text. For the truly hard cases, KOReader can use Tesseract to perform optical character recognition and then reflow the results.

There are a number of connectivity modes available with KOReader. One can, of course, just plug in the reader as a USB-storage device and dump books onto it. Or KOReader can start up its own SSH server and accept files copied in that way. There is a plugin to obtain books from a number of cloud-storage services. KOReader can also connect to a local Calibre server and be fed books that way — though, when your editor tried this, the result was a crash and an illegible traceback on the reader screen, which is not the reading material that had been hoped for.

Many of the other expected features are there. KOReader allows the highlighting of sections and attaching notes to books. It can look up words in a dictionary. Happily for the purposes of this article, KOReader can take screen shots (just simultaneously tap on two diagonally opposite corners). And so on; the basic point that KOReader is a program with a lot of features should be clear by now.

One thing that KOReader cannot do, of course, is read books that are protected by DRM; it doesn't even make them visible in the interface if they exist on the device. There are authors and publishers who make at least some of their books available without DRM; for the rest, it's a matter of falling back to the built-in reader or employing one of the ways of removing DRM that are alleged to exist. This is not ideal, but it is a reflection of the less-than-ideal world we live in rather than on KOReader itself.

Looking at the code

The KOReader source is available on GitHub. It turns out to be a little over 140,000 lines of mostly Lua code that can be built for any of the platforms that KOReader supports.

The project seems to produce a release on a roughly monthly schedule. Since the v2021.03 release (March, 2021), 873 non-merge commits have been made to the project trunk; that is a little over two per day. That work was contributed by 47 developers, with the top two developers ("NiLuJe" and "hius07") being responsible for just under half of the total. This project, in other words, appears to be alive and well, though doubtless more contributors would be appreciated.

Will KOReader take the place of the native Kobo reader application on this device? There are a few more science-fiction novels to be read in KOReader before the decision can be made, but at least the research is entertaining. KOReader will be a good reader application for users who are willing to spend a bit of time tweaking its knobs to get the ideal experience. KOReader will also be good for anybody who might (rightly) worry that their proprietary reader device might just be spying on them. Users who want a simpler interface or who have to work with DRM-encumbered books will probably want to look for something else. Either way, KOReader provides a free-software alternative that can be used across a wide range of hardware, and that seems like a good thing.

Comments (49 posted)

Page editor: Jonathan Corbet
Next page: Brief items>>


Copyright © 2022, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds