|
|
Log in / Subscribe / Register

Leading items

Welcome to the LWN.net Weekly Edition for May 22, 2025

This edition contains the following feature content:

This week's edition also includes these inner pages:

  • Brief items: Brief news items from throughout the community.
  • Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

A kernel developer plays with Home Assistant: case studies

By Jonathan Corbet
May 16, 2025
The first article in this series provided an overview of Home Assistant, its community, and its capabilities. It was deliberately short on descriptions of interesting things that can be done with Home Assistant, though — the reasons why one might actually want to use this program. In this closing article, we'll look at how Home Assistant was used to solve some real problems.

Solar panels

Electrification is good, but it is even better in the presence of abundant electricity; thus, some years ago, an investment was made to cover my roof with solar panels. The result was indeed abundant power, about double what the home and the (non-Musky) electric vehicle actually used, which was a good thing. But the situation could have been better.

The panels came with a monitoring system from a company called SunPower; it collects data on panel performance and grid usage, reporting it all to some system in the cloud somewhere. The company produced a reasonably competent app that provided information about current and past performance, including data on each individual panel. Naturally, the app was the only point of access to that data, a fact which made me a bit nervous. Depending on a "free" cloud service from some vendor often does not go well.

Sure enough, in 2024, SunPower went bankrupt. The company that acquired SunPower's assets (among which are counted me and my data) goes by the name SunStrong. This company recently informed me that free access to my data was coming to an end; without a payment, the app would only provide basic instantaneous data. In other words, the data generated by the solar panels, which I own, is collected by the monitoring system, which I own, and sent off to a cloud system, which I definitely do not own and which will hold my own data hostage.

That, of course, is just the sort of situation I got into free software to prevent. Fortunately, my early nervousness had led me to look for ways to collect and hold that data locally; it is what drove me to install Home Assistant. As it happens, the monitoring system does not readily give up its information to the confused people who think they own it, but somebody out there on the net figured out a way using a network port intended for installation-time diagnostics, and documented it thoroughly. I was able to solve the problem in a slightly simpler way (the machine running Home Assistant has two network ports, so I was able to do without the intermediate Raspberry Pi system), and have had full access to the data from the solar system ever since.

[Panel production plot] After some of the inevitable fiddling (see the first article for some details), the result is a far better interface to solar-related information than the app ever provided. As one example, see the image to the right, plotting the output from a sub-cluster of panels, clearly showing the effects of the partial shading of two panels, even at the beginning of April. It has become easy to see how well each part of the system is performing and, should a panel or its inverter go bad someday, which panel is the problem will be immediately apparent.

This kind of data should not be held hostage by some corporation that feels empowered to alter the deal whenever it sees fit. Freedom from that situation is, in my mind, one of the most compelling reasons to use a tool like Home Assistant.

Helpers

That said, as mentioned before, Home Assistant requires some fiddling to get to a useful point. As an example, consider that the monitoring system reports the instantaneous power from both the panels and the grid. It does not report the power consumed by the house, which is often of interest. Of course, that quantity is easily enough obtained: it is simply the sum of the solar and grid power flows. At least, it is easy once one figures out the abstraction known as "helpers".

If one goes into the "Settings" screen of the Home-Assistant interface, then into "Devices & services", one will see a tab marked "Helpers". A helper, as it turns out, is a way to instantiate a virtual sensor that is defined as a function of one or more existing sensors. This screen, among many other things, enables the creation of a "home power usage" helper that is the sum of the solar and grid power sensors. Once it exists, it can be used to generate plots, as input to other helpers, or as part of an automation.

Now imagine that it might be nice to know how much power the house has consumed in any given day. The "integral" helper comes in handy here; the house power helper we just found can be fed into this new helper, which will integrate its value over time, turning kilowatts into kilowatt-hours. That will yield a constantly increasing value that is nice if you want to know the house's all-time usage, but it is less useful to tell how the day is going. Happily, there is yet another type of helper called a "utility meter" that can be configured to reset to zero at a given interval — every night at midnight being a logical choice. The output of that helper will be the day's usage.

The end result of all this is that helpers can do a lot of interesting things with the data collected by Home Assistant, but that their use does require a lot of clicking around in the interface (or YAML writing for those with that inclination). That work can add up; if there is a way to generate, say, daily production totals for each of 22 panels without manually creating a helper chain for each, I have not found it.

Another way to put this is that Home Assistant could really use a more general sort of query language for the data it manages. But, as a system that is set up once then used for years, what is there works pretty well.

One other note regarding data is worth mentioning: Home Assistant will collect high-resolution data from sources that provide it, and that data is stored — but only for a while. Unless configured otherwise, data more than ten days old will be boiled down severely, seemingly to one sample per hour. Since many people run Home Assistant on small, resource-constrained systems, taking some pains to reduce the amount of storage space required is understandable. But this discarding of data may come as a surprise to users with large disks who were hoping to retain full-resolution data indefinitely.

Heat pumps

What does one do with a surplus of electricity? Donating it back to the electric utility company (or something close to that effect) has little appeal. It is better to find a good use for that power, so the next step was to install air-source heat pumps and stop heating the house with fossil fuels. It takes a bit of a leap of faith to do this in Colorado, where the temperature dropped to -15°F (-26°C) this winter, but that technology has come a long way; the house remained warm.

The pre-purchase research yielded a clear indication that heat pumps made by Mitsubishi can work in this sort of climate. Respect for that company's hardware is universally high. It was also clear, before the purchase, that respect for Mitsubishi's software, and its "Kumo Cloud" app, was rather lower. Kumo Cloud is, as its name would suggest, a cloud-based solution that requires an Internet connection to work. The app was just as bad as the reviews had suggested. In March, with much hype, Mitsubishi replaced it with a new app called "Comfort"; it has been completely rewritten and is far worse.

[Heat-pump control] Naturally, Home Assistant has a Mitsubishi integration. It requires access to the cloud account at initialization to obtain the IP addresses for the controllers in each of the heat-pump heads; thereafter, it can interoperate with the controllers locally. This integration collects the data provided by the hardware (operating mode, temperature, set point, humidity), and can make it available in the usual sorts of ways. The app (in either version) is unable to plot the temperature of the room throughout the day, for example, but Home Assistant can do that.

It can also, of course, control the hardware through a simple thermostat-like widget that can be placed on a dashboard. It is easy to see the climate-control configuration of the entire house on a single screen. If remote access has been set up, it is also possible to change that configuration while away, perhaps turning the heat up while on the way home.

While I have not done this, it is possible to do a lot more by setting up automations. One can set up a schedule to vary the set temperature over the course of the day. A more complex setup might use motion or Bluetooth sensors to determine which rooms are occupied and adjust the climate-control settings accordingly. But even as a way to just control what the heat pumps are set to at any given time, Home Assistant has replaced the use of the vendor app. At some point, I may cut the controllers off from the Internet entirely.

Power monitoring

The heat pumps have done a fine job of keeping the house warm in the winter, but they also had the effect of causing that surplus of electricity to disappear. That naturally led to a higher degree of interest in how power is consumed in the house; the simple data provided by the SunPower unit was not sufficient. Wouldn't it be nice to know what specific rooms or appliances are consuming? It turns out that there are a number of companies selling power monitors that can report usage on a number of circuits in the house; it was just a matter of picking one.

That question turned out to be easy to answer in the end. There is one unit, made by Refoss, that is affordable and able to monitor sixteen independent circuits in the house. (Note: that product page may not be available in all regions. The main site is available here.) Readers will undoubtedly be surprised to learn that this device is designed to report data to a cloud server, which will then make it available via an app. There is a crucial difference here, though: the Home-Assistant integration for Refoss devices, which interfaces directly with the monitor and needs no cloud connectivity, is written and provided by Refoss itself. Home-Assistant compatibility is the first bullet item on the above-linked product page.

There are not many companies that see fit to support Home-Assistant integration for their products; there are probably more that are actively hostile to it. So when a company supports this integration, it moves directly to the top of my list of candidates; that is what happened here. After a few hours of nervous work inside the main service panel for the house and more time spent tweaking the names of all the sensors, the monitor was up and reporting into Home Assistant as designed.

[Heat-pump power usage] The data returned by the monitor turned out to be immediately useful. The plot to the right shows the power consumption of the two installed exterior heat-pump units, one for the downstairs, one for the up. The blue trace (downstairs) shows how these systems are meant to operate; they should cycle rarely, and run steadily for some time once they start. The yellow trace (upstairs), instead, demonstrates a problem called "short cycling"; it results in increased power usage and increased wear on the unit. Short cycling can be caused by a number of problems, including configuration mistakes or hardware problems.

In this case, the cause would appear to be hardware, but it has taken some time to reach that diagnosis. It was, however, easy to demonstrate the problem to the repair technicians by showing them a plot like this one. Without detailed power-usage information, the conversation with the technicians would likely have been a far longer one. There is value in having visibility into the operation of crucial systems — like one's house.

Having power information has helped in other ways. Seeing how much current is drawn by my monitors in their "power save" mode has led to them being turned off at night. It has become easier to say what is using power during the high-cost times of day and shift that use to other times. It is amusing to learn that a full kernel build costs about 20Wh. In general, visibility into how the house is working is both interesting and useful.

To close

It is also sobering to realize how much that information can reveal about what is going on in the house — how many people are present, which rooms they are in, what they are doing in those rooms, etc. The thought that nobody is out there trying to gain access to (and exploit) that data seems naive at best. Yet we happily allow that data to be exported and stored on somebody else's server with no real idea of how long we'll have access to it, how well that server is protected, or even where it is. That is the world we have built for ourselves, but it is not the one many of us envisioned.

Tools like Home Assistant offer a different vision of how the world could be — how we should have built it in the first place. Free software is not just for operating systems; it is for all of the systems that touch our lives. It is software that actually serves us, rather than the people who sold it to us.

Home Assistant is a wildly successful project, with vast numbers of users, but most people still have never heard of it. Many others, undoubtedly, have been scared away by the fiddly nature of the system and the challenges of integrating devices that were never designed to work together. Continued success will require a lot more effort to make things Just Work. There is almost certainly a business to be had in this area, just waiting for somebody with the requisite energy and skills to make it real. Meanwhile, a versatile and useful tool is available for those who are willing to put the effort into it.

Here ends this look at Home Assistant. It is a partial look at best; as mentioned before, I have not yet had the time to play much with features like automations and scenes. There is voice control under development, and Music Assistant for media streaming. My house is also devoid of motion sensors, remote-control light bulbs, video doorbells, automatic blinds, Internet-connected cookware, app-driven pet feeders, "smart speakers", and other sorts of IoT stuff, all of which can be connected to Home Assistant if it is present. So I have been unable to test those integrations. In the end, every deployment will, like every home, be different — but all will be free and under the owner's control.

Comments (45 posted)

Recent disruptive changes from Setuptools

May 21, 2025

This article was contributed by Karl Knechtel

In late March, version 78.0.1 of Setuptools — an important Python packaging tool — was released. It was scarcely half an hour before the first bug report came in, and it quickly became clear that the change was far more disruptive than anticipated. Within only about five hours 78.0.2 was published to roll back the change, and multiple discussions were started about how to limit the damage caused by future breaking changes. Nevertheless, many users still felt the response was inadequate. Some previous Setuptools releases have also caused problems on a smaller but still notable scale, and hopefully the developers will be more cautious going forward. But there are also lessons here for the developers of Python package installers, ordinary Python developers and end users, and even Linux distribution maintainers.

Python packaging

Python code is commonly distributed using a Python-native packaging system, rather than as a Linux package or standalone executable. The basic ideas have been described by LWN before.

Packages are available in two standard formats: the "wheel" format, and the "sdist" (or source distribution) format. Both are compressed archives, but a wheel has been pre-built and can be installed without much work besides copying files. An sdist, on the other hand, roughly represents the developer's source tree (although it might omit tests, documentation, etc.), and must be built on the end user's machine. This is generally only useful for packages that contain non-Python code, but it allows for additional setup prior to installation. An sdist also includes instructions (perhaps including Python code that will be automatically run) to compile non-Python code, arrange files, and ultimately "build" a corresponding wheel.

Creating packages — in either format — can be quite complex. Usually, therefore, most of the work is delegated to a build backend, which provides logic to create an sdist or wheel directly from a source repository, as well as to create a wheel from an sdist. To install an sdist, an installer such as pip will check the sdist's metadata to determine the right build backend, ensure that backend is installed, and invoke it via a standard API. The resulting wheel is then installed as if it had been downloaded directly.

By default, this process uses a kind of build isolation: the backend is installed in an isolated, temporary virtual environment. This ensures that builds don't interfere with each other, allows different packages to use different versions of Setuptools, and means that Setuptools doesn't need to be installed ahead of time in pip's own environment.

Historically — since back when end users were expected to resolve and install dependencies themselves — Setuptools was the standard tool for a variety of development tasks. Nowadays, much of its functionality has been deprecated, but it still acts as a build backend — and for legacy support reasons, packages are assumed to use it by default.

What went wrong

Projects built with Setuptools may describe some of their metadata in a configuration file named setup.cfg (which uses a simple INI-like format, with key-value pairs organized into sections). The unreleased Setuptools 78.0.0 added stricter validation for the contents of such files. Historically, hyphens and underscores were treated as equivalent in the names of keys appearing in the file — behavior originally implemented by the distutils standard library module. In 2018, a bug report claimed that this normalization caused problems with tox (a Python tool for automating development tasks such as testing). Setuptools developer Jason R. Coombs agreed that there was a problem; after some changes to the Setuptools configuration-parsing code, a a deprecation warning for hyphens in key names was added in 2021.

Setuptools version 78 sought to turn that warning into an error. Developers creating new sdists would be able to fix the problem trivially, but the change also immediately broke the automated wheel-building process for many sdists already available on the Python Package Index (PyPI).

In fact, this change was already known to be incompatible with the popular Requests library — its setup.cfg includes provides-extra and requires-dist keys, where the standard names are provides_extra and requires_dist respectively. But, since Requests uses only Python code, and already publishes a wheel, it would only cause problems for those (such as Linux distribution maintainers) who insist on building directly from source.

The 78.0.1 patch simply removed Requests from the Setuptools integration tests. Setuptools developer Anderson Bravalheri also submitted a pull request to Requests to fix the problem with setup.cfg (though as it turns out, the next minor version of Requests is expected to modernize its packaging setup completely, so as to avoid any future issues). However, packages with now-invalid setup.cfg files turned out to be much more common than expected — about 12,000 packages are estimated to have been affected.

Reporting and updating

When a resulting installation failure was first reported, Bravalheri's initial response was fairly dismissive:

Hi @andy-maier, please contact the package developers and inform about the problem highlighted in the error message: [...] This [key name in setup.cfg] has been deprecated in 2021.

He followed that up with:

We bumped the major version of setuptools from 77.0.3 to 78.0.0 to indicate an intentional breaking change, namely the removal for the deprecated handling of --separated options in setup.cfg. They are described in this section: https://setuptools.pypa.io/en/latest/history.html#deprecations-and-removals.

At some level, this response makes sense. The project correctly followed its semantic versioning policy, and the deprecation had occurred years ago. In principle, users could avoid problems by simply not upgrading to the new Setuptools version, and developers could patch their setup.cfg files to meet the new expectation. However, things aren't really that simple. Updating the released packages is a lot of work for everyone — it's clear that many maintainers also resent this kind of churn. Besides, it doesn't prevent attempts to install the previous versions of the package with now-invalid setup.cfg files.

On the user side, meanwhile, build isolation makes it difficult to keep Setuptools downgraded — because of all the temporary copies installed automatically. Historically, there was no good way for packages to specify what version of Setuptools they used. That's possible now, but projects using Setuptools are under no pressure to change anything. And the standards-mandated default is to use the latest available version. The net result is that every sdist following any kind of deprecated practice becomes a ticking time bomb.

Worse yet, every package that breaks has the potential to cause major ripple effects, in today's world of "software ecosystems" where projects often have large graphs of transitive dependencies. Some of the projects broken this time around turned out to be abandoned or poorly maintained, but used by many others. For example, the stringcase module broke due to the change, but it is seemingly abandoned; when I learned of this, I decided to republish the library following modern packaging standards.

Similar events

This is not the first time that a breaking change in a major Setuptools version was disruptive. In recent memory, arguably the most similar case is that of Setuptools version 72, which attempted to remove the setuptools.command.test module. This, too, was following up on a long-standing deprecation of using setup.py directly to run tests. This isn't even relevant to installing packages from PyPI — but because setup.py is Python code, removing the module causes an ImportError at build time. Notably, the Requests project was also impacted by this, even though its developers were no longer even using this workflow.

Less notably, Setuptools version 77 was intended to enforce standards for how license files are referred to in project metadata, while also implementing the new standard enabling the use of SPDX license expressions in package metadata. However, the implementation caused problems for some large multi-language projects (notably for Apache projects) where the source tree has relevant license files in a parent directory of the actual Python project. Setuptools version 71 changed its vendoring system to favor separately-installed versions of its dependencies (as a stepping stone toward de-vendoring); this caused problems for users when those dependencies removed functionality that was still present in the vendored version. An internal reorganization in version 69 broke Astropy's continuous-integration (CI) system, and another in version 70 broke PyTorch. Not every new version causes widespread problems, but it's certainly possible to continue this list.

Criticism

Aside from the GitHub issue, users critiqued handling of the incident on Hacker News and Reddit. In particular, many questioned why the use of hyphens should have been deprecated in the first place. Indeed, one might infer that Setuptools can't really ever safely remove support for anything — as much as the developers might like to clean up the code, Hyrum's Law reigns supreme. Setuptools is seen as critical infrastructure for Python, and it's clear that the Python world will be stuck with old packages following deprecated packaging practices for quite some time.

The problematic version 78.0.1 was also never deleted nor "yanked" (to prevent pip from installing it unless explicitly selected by an exact version number) from PyPI, causing further objections.

One might argue that the simple fact that Setuptools is on version 78 points to a problem in itself — that Setuptools development is moving far too quickly. This is not entirely fair: about half of Setuptools major versions date to before the implementation of the new pyproject.toml-based system, with version 39 being released in March 2018. However, we still now see about five new major versions of Setuptools per year. Progress on updating legacy projects to modern standards, needless to say, has been far slower. While I'd prefer not to see Setuptools development artificially slowed, it would be nice to have breaking changes bundled together so that they occur less often.

Discussion

More importantly, though, there needs to be better awareness of such changes in Setuptools, and better processes for addressing them. On the official Python discussion forums, pip developer Damian Shaw started a thread about how build backends could improve their approach to deprecating and removing functionality, summarizing the overall problem for the Python ecosystem:

In this ecosystem it has meant that any build backend that is heavily relied on that makes a backwards incompatible change can break user workflows on a very large scale, and the tooling available for users to recover isn't well developed.

The ensuing discussion revealed that the documentation for some other build backends recommends setting an upper bound for the backend version in the project metadata. However, doing so can potentially break packages in the future — for example, if a newer version of the backend is required to work under a new version of Python.

Another approach is to have the installer use different logic to choose a build backend version. Paul Moore, a pip developer, initially resisted such a change:

I don't think it's inevitable, and the UX [user experience] design is one of the things I'm concerned about. IMO, pip shouldn't be taking on complexity here - I don't see any evidence that this is a major issue. We've never had any problems like this with any other build backend. And while I don't want to blame setuptools, I also don't want pip's feature set to be dictated by mishaps in setuptools' release management.

However, the discussion on the initial Setuptools bug report revealed that pip already provides some useful functionality here: a hack that allows end users to constrain the version of the build backend used for isolated builds. The pip team then discussed improving the surrounding user interface and making this an officially blessed approach to the problem. An analogous approach is explicitly supported by uv; Poetry users installing dependencies in a development environment may have to wait for the resolution of a GitHub issue. Pip users can also just disable build isolation, which might fix the problem in some cases.

Regarding front-end tools failing to expose warnings, Bravalheri proposed that installers should show build warnings to end users by default. This is not the first time the problem of deprecations has been raised for the language. Python developer Steve Dower noted that end users can't really do anything about the warnings. However, if a project only publishes an sdist, the maintainers won't see the output from trying to build a wheel unless they test installation locally. Even if end users can't fix the problem, they could issue bug reports about the warnings if they are shown.

A final problem with communication between installers and build backends is that the existing API doesn't seem to provide a good way to separate warnings from the rest of the build output. Moore suggested building a separate system for this:

So why don't we ask the setuptools maintainers what they would like from build frontends? Not right now - emotions are too high at the moment - but once things have cooled down, we should work with setuptools to understand how we can help things go more smoothly in future.

He had a list with several suggested paths for working with Setuptools developers, including looking into ways to get deprecation warnings in front of users:

There was no way for setuptools to get the deprecation warning in front of their users? Then frontends could add a way for backends to issue "priority warnings", which will be displayed even when normal output is suppressed.

Hopefully, something like this will be implemented in the future.

In summary, the Python packaging system is amazingly complex — to the point that a quibble over hyphens versus underscores can break thousands of packages and lead to hundreds of discussion posts. But with this complexity comes opportunity for many parties to improve the experience.

Comments (101 posted)

Debian AI General Resolution withdrawn

By Joe Brockmeier
May 20, 2025

Despite careful planning and months of warning, Debian developer Mo Zhou has acknowledged that the project needs more time to grapple with the questions around AI models and the Debian Free Software Guidelines (DFSG). For now, he has withdrawn his proposed General Resolution (GR) that would have required the original training data for AI models to be released in order to be considered DFSG-compliant—though the debates on the topic continue.

Zhou has been working toward the GR for some time. In February, he posted an early draft to the Debian-project mailing list to ask for help and to give other developers time to provide input or develop their own counter-proposals. On April 19, he sent his revised proposal—with detailed reasoning for his stance, comments on possible implications of the resolution, and several appendices of resources—to the debian-vote mailing list, which we covered at the end of April.

The text for Debian members to consider and vote for (or against) was short and simple:

Proposal A: "AI models released under open source license without original training data or program" are not seen as DFSG-compliant.

If a model (or other artifact) is not DFSG-compliant then it cannot be included in the Debian main repository that comprises the Debian distribution; only those packages in the main repository are considered part of the Debian distribution, as described in the Debian Policy Manual. Being outside main does not entirely prohibit Debian from distributing an artifact, though. The project also has the contrib, non-free, and non-free-firmware repositories for software that, directly or via dependencies, does not comply with the DFSG. However, packages outside main are not considered part of the distribution. These details are spelled out more fully in the Debian wiki SourcesList page.

Zhou's proposal is in contrast to the Open Source Initiative's (OSI) Open Source AI Definition (OSAID) that was announced in October last year, which does not require the training data to be released. The OSAID requires that an AI system be released in a way that will grant the freedom to use, study, modify, and share an AI system or elements of the system. OSI has determined that it is sufficient to provide model parameters ("such as weights or other configuration settings") and detailed information about the training data that would allow "a skilled person can build a substantially equivalent system". LWN covered the OSAID in October 2024.

Initially, it looked like smooth sailing for Zhou's proposal. Early discussion was largely positive—though Zhou's proposal did attract a few counter-proposals. Thorsten Glaser's lengthy proposal would have raised the bar even higher for AI models to enter Debian main. For instance, it would have required model training to happen during package build or that the model be built "in a sufficiently reproducible way that a separate rebuilding effort from the same source will result in the same trained model". That would be in addition to requiring training data, of course. It would also have a dramatic impact on Debian's infrastructure in terms of requiring the hardware to actually perform training.

Sam Hartman put forward a proposal that would allow an application to define the preferred form of modification, which might or might not include training data. Bill Allombert pointed out that, without training data, Debian has no way to know what's inside the model. "The model could generate backdoors and non-free copyrighted material or even more harmful content." Hartman countered that the project had accepted x86 machine code as a preferred form of modification. Inspectability, he said, has never been at the core of the DFSG. He also predicted that there would, eventually, be "black box inspection tools" that would improve the ability to inspect models over time.

What would be simpler, Aigars Mahinovs said, would be to vote on the Debian project endorsing the OSAID. He later submitted a proposal to clarify that training data is not source code for the purposes of the DFSG. It would have, instead, required "training data information" as defined in the OSAID. The actual training data would be considered merely "an intermediate build artifact". Wouter Verhelst objected to this and said that the project would have to drop its reproducibility goals if AI models were accepted in main. LWN covered Debian's progress toward reproducibility in August last year. Stefano Zacchiroli suggested that there might be room to merge the proposals from Mahinovs and Hartman since they seemed to go in the same direction. Ultimately, however, none of the counter-proposals had received enough sponsors to be added to the ballot if Zhou's GR had gone to a vote.

Spam classifiers

A few Debian developers wondered about the impact on software that preceded the current AI craze and rampant rapacious data scraping that accompanies creating many of today's AI models. Applications that one would not usually lump in with AI, such as games, spam filters, optical-character recognition (OCR) tools, and text-to-speech software, also depend on trained models that are missing training data. Depending on one's reading, a fair amount of software already in Debian could be seen as non-free if Zhou's proposal were adopted.

For example, Ansgar Burchardt pointed out that it would not be possible to package spam or phishing emails as part of a training data set for Bayesian classifiers because those emails are unlikely to be under a free license. Russ Allbery said that he did not think that a classifier trained on such data would be DFSG-free, and did not think it should be included in Debian main.

That doesn't mean I think it's bad or immoral or anything like that. I have a database like that myself. :) It's simply not free software, and is outside the scope of what Debian is for. Not even all of Debian's own data is free software. For example, I would not consider the [Debian bug tracking system] database or the mailing list archives to be free software because the licensing status is not sufficiently clear.

The idea of blocking spam filtering software that uses trained Bayesian filters, which has been available in Debian main for ages, troubled some Debian developers. Hartman said that software freedom is supposed to be an achievable set of standards that empowers users. It may require users to forego convenient commercial software, but it should not be about sacrificing potential:

Users might want a Bayesian classifier--I do enough that I've trained one. Software in main like a mail reader or a mail system might well want to include a classifier. Saying that even if someone is as dedicated to freedom as they can be, they can never live up to our standards and include that reasonable functionality in Debian main makes me think we have lost sight of our users.

Zacchiroli predicted that, if Zhou's proposal won, Debian packagers who included some form of AI model without DFSG-free training data would be forced to patch their software to download data on first use or "just give up on maintaining those packages". He said he failed to understand how this served Debian's users, and would not really protect them from "evil OSAID-but-not-DFSG-free stuff" anyway.

Withdrawal

After much discussion, Zhou withdrew his proposal on May 8. He said that it had become clear that the community was unprepared to vote on the proposal. Initially, he wanted to simply address the "conceptual interpretation" of the DFSG with regard to AI models, but the real implications had given Debian members pause. He asked for suggestions on tools that might help him scan the Debian archive to figure out which packages might be affected by the GR.

Zhou also added that many people seemed to assume that pre-trained models were trustworthy. He said he would create a demonstration to illustrate how a backdoor could be planted in a neural network. This would allow those who consider models the preferred form of modification to demonstrate how they could fix the backdoor. He indicated that he would need a few months before he could return to working on the GR.

Russ Allbery thanked Zhou for his work and said that this happens a lot. People often wait until the GR is proposed before speaking up, and the discussion often brings opinions to the surface that had not been expressed before. He added that he thought delaying the GR was the right decision:

I also hope it doesn't discourage you from continuing to work on this. I don't think anyone is saying that we shouldn't have this conversation and a vote, only that we (myself very much included) are realizing that we hadn't actually thought this through as thoroughly as we had thought.

Hartman and Mahinovs followed suit and formally withdrew their proposals even though they did not have sufficient sponsors, just to be clear that they were not to be voted on in the absence of Zhou's GR. To date, Glaser has not formally withdrawn his proposal.

More complicated than first thought

It seemed that there was plenty of support at the beginning of the discussion for requiring training data with AI models to consider them DFSG-free. However, coming up with a definition of AI models that does not overlap with other, less controversial, data is clearly going to be difficult. It will be interesting to see how the discussion goes when Zhou returns to the topic down the road, and whether Debian can adopt a policy without unintended consequences.

Comments (49 posted)

A new DMA-mapping API

By Jake Edge
May 15, 2025
LSFMM+BPF

Leon Romanovsky began his session at the 2025 Linux Storage, Filesystem, Memory Management, and BPF Summit (LSFMM+BPF) by explaining that the improved DMA-mapping API that he has been working on is a group effort. He, Chaitanya Kulkarni, Christoph Hellwig, Jason Gunthorpe, and others are proposing to modernize the API and to "make it more suitable for current kernels". He told the assembled storage and filesystem developers that the progress on the proposal has stalled, but that it was the basis for further work in various areas, so he hoped to find a way to move forward with it.

The existing DMA API is based on struct page, which is fine, but using DMA requires scatter-gather (SG) lists, so there is a lot of conversion between the two formats. In addition, many DMA users have their own formats for the data being transferred, leading to more conversions (between SG and native formats).

A struct scatterlist has two fields that are supposed to contain a CPU address (page_link) and a DMA address (dma_address) but various (ab)uses of the scatterlist have changed the meanings of those fields. For peer-to-peer DMA, page_link is a synthetic CPU address used to get information from the page structure, he said. For dma-buf usage, page_link is null and dma_address is synthetic in order to access the device-private memory. Hellwig pointed out that the dma-buf usage is explicitly invalid according to the documentation, but "the dma-buf people did it anyway and refused to fix it".

[Leon Romanovsky]

So the proposal (which was v7 during the summit, but is now on v11) is "very simple", Romanovsky said. There is a new set of APIs that is based on a careful refactoring of the DMA-mapping API; it is meant to be bug-for-bug compatible with the existing API. Instead of having to use SG lists, DMA users will be able to directly manage their own I/O virtual address (IOVA) space, which is provided by the I/O memory-management unit (IOMMU). The new API will optimize for DMA using the IOMMU path; systems without an IOMMU can fall back to using the existing API.

He briefly described the API, which is described more fully in an LWN article from November. It starts with a call to dma_iova_try_alloc() to allocate the IOVA range, then calls to dma_iova_link() to map memory into the range. Once all of the memory of interest has been mapped, dma_iova_sync() can be called; rather than synchronize on each mapping, that is an optimization to only do the expensive synchronization operation once for the whole range.

He and Hellwig converted three or four subsystems ("depending on how you count") to use the new API as part of the patch set, Romanovsky said. The easiest of those to follow is the virtual function I/O (VFIO) live-migration code. He showed the before and after code; currently there are three separate loops over lengthy SG lists representing large areas of memory, which is "a huge amount of work just to make sure we are getting back DMA addresses" that can be used to program the hardware. Using the proposed API, that will be done in a single loop, if the optimized path can be used; if not, it will fall back to the existing mechanism.

This work is part of a large roadmap. The idea is to eventually remove the use of SG lists for dma-buf and then for other DMA users. The roadmap is just a set of ideas at this point, no concrete work has been done while awaiting the new API.

But he showed a response from DMA maintainer Robin Murphy that left Romanovsky "unclear how to proceed now". Earlier in the thread, Murphy explicitly rejected the patches, but in Romanovsky's opinion, there are "no technical objections" that have not been addressed. He highlighted the sentence ending Murphy's final email in the thread, which does not seem to lead in a direction toward resolution:

And there is no obligation for maintainers to accept code with obvious significant issues just because they don't have the time or inclination to personally engage in trying to fix said issues.

Dan Williams suggested that a way to help break the logjam would be to point out which features that developers want are being blocked. For example, some confidential computing features are stuck waiting for the new DMA API. Simply listing additional plans to switch away from SG lists does not demonstrate the use cases that need those changes; Williams suggested making those use cases visible in order to make it "extremely clear" that the new API is needed.

David Howells said that he was looking forward to removing SG-list support from the crypto layer. There is a lot of SG-list creation for no real gain when encrypting and decrypting network traffic using DMA; he doesn't think the crypto layer actually needs SG lists for what it does. James Bottomley said that the SG handling was there to support crypto accelerators, which "almost nobody" has; the crypto developers think that having support for SG lists is a mistake, since there are so few accelerators. Hellwig said that there are a few accelerator drivers that do use SG lists, but that many crypto developers would like to remove that functionality; unfortunately, he said, the crypto maintainer wants to replace it with an even more complex interface.

The new API "looks really good and I can't wait to actually use it", Chuck Lever said. He noted that he is the maintainer of a consumer of the "RDMA rw interface" and asked Hellwig whether that would be changed to use the new API. Hellwig said that it definitely should be, but he did not have the time or the hardware to do so; he was willing to help anyone who wanted to work on it and thought it should just be a few days worth of effort.

Luis Chamberlain asked if Romanovsky could describe "to the best of your ability" what Murphy's objection to the patch set is. After a long pause, Romanovsky said: "Honestly, I can't"; he said that he had tried to find "any rationale, any action item that we can take" from the responses, but was unable to. Hellwig said there were "a lot of minor nitpicks" that he was not sure were true; if they are, they need to be described better. Murphy has an overall objection that "it pushes too much low-level knowledge into the consumers of the DMA API", Hellwig said. But that is the point of the new API, Gunthorpe said; Hellwig concurred, noting that Murphy's objection was "at least semi-valid" though he did not agree with it.

So what help were the developers asking for, Williams asked; "is it just 'flood the zone with use cases', is it 'provide a wrapper API'?". Hellwig said that the work on adding support to various users has been adding wrappers to make it easier to use and to hide the low-level details from the drivers; this work is meant to be a building block to allow subsystems to use their specific data structures for DMA, not to make the drivers individually handle those details.

Gunthorpe agreed that having people post their use cases and their interest in using the feature would be useful, both for Murphy and for Marek Szyprowski, who is another DMA maintainer. No one is suggesting that Murphy's complaints should be ignored, he said, but there is a strong push for the changes from various users of the DMA interface. Maybe seeing additional use cases will encourage Murphy to spend more time on it. Williams thought that was a good path and recommended that "everybody chime in, but be nice about it".

As of early May, Szyprowski had merged the patches into his dma-mapping-next branch, presumably with an eye toward getting the new API into the 6.16 kernel.

Comments (5 posted)

An update on continuous testing of BPF kernel patches

By Daroc Alden
May 20, 2025
LSFMM+BPF

Ihor Solodrai has been working on the BPF subsystem's continuous-integration (CI) testing for the last six months. At the 2025 Linux Storage, Filesystem, Memory-Management, and BPF Summit, he remotely shared an update on his work, and solicited feedback on how the tests could be further improved. Much of the work he's done has been specific to the BPF subsystem, but some is more generic and could potentially be of use to other subsystems. He also shared some general lessons learned from working on the BPF CI tests.

Before getting to his main update, he first showed a visualizer he has been working on for understanding verifier failures. If one uploads a log file produced by the verifier, the tool shows "something like a debugger view" that lets one step through the verifier's process. Solodrai was optimistic that it could help with debugging, especially for people working on extending the verifier. He said that the tool is currently a prototype, and he would appreciate feedback on it.

The main task Solodrai has been working on since the 2024 Linux Plumbers Conference has been expanding the coverage of the BPF CI tests. In the course of that work, he has fixed a number of other things. Still, his main focus has been on adding GCC to the test suite alongside Clang, and adding testing for sched_ext. At the time of the summit, GCC's BPF implementation was not passing the BPF self-tests, and therefore not included in the CI testing. Since then, that appears to have changed, because recent test runs have included testing for GCC 14.

The sched_ext tests are in a similar position; at the time of the summit, Solodrai was running into problems with divergence between the kernel Git repository used by the sched_ext developers and the one used by the BPF developers. Trying to run the sched_ext tests against bpf-next was turning up a lot of problems due to missing commits. Now, it also appears that these have been enabled in recent test runs.

Expanding the number of tests being run has also meant expanding the number of test runners. The BPF CI uses GitHub Actions to run tests on AWS-hosted virtual machines. When Solodrai first started looking at the project, it was using a fixed number of test runners. That meant that a spike in the number of patches to test could cause a backlog of jobs that would take some time to clear. Worse, many of the kernels being tested were cross-compiled to other architectures from x86_64, which left the limited number of x86_64 runners frequently overwhelmed. In order to fix that, Solodrai explained that he had changed the tests to do the build step using a separate service that automatically scales up and down as required. This lets the dedicated test runners focus just on running the tests.

Ideally, the test runners would scale automatically as well. The problem is that testing a kernel build essentially requires an actual virtual machine. User-space software can frequently get away with running tests in a container, but that doesn't suffice for testing the kernel. It should still be possible to make automatic scaling work in theory, Solodrai said, but the existing software that other projects use to manage testing in GitHub actions won't suffice.

Without scaling for the test runners, however, Solodrai hasn't quite accomplished what he wanted when he first started working to improve the usability of the tests. Before, building the kernels was the bottleneck. Now, the actual testing is — but it's still a bottleneck.

As an example, he went into some detail about the test runners for the s390x architecture. "I have a difficult relationship with them," Solodrai said. They are hard to maintain for a variety of reasons, but primarily because s390x is not in common use. For example, the control software that GitHub Actions connects to in order to request that a test start doesn't have an official s390x build. IBM provides a modified test runner for s390x, but it doesn't quite work for this use case, Solodrai explained.

So he ended up needing to reverse-engineer the build process of IBM's modified test runner and apply that to the official GitHub Actions code in order to produce a build that can be run on his s390x virtual machines. Every time GitHub Actions releases an update, there's a 30-day window for clients to update before things start breaking, "and on s390x, it's a roll of the dice."

One of Solodrai's questions for the attendees was whether s390x support was really worth it. On the one hand, s390x is one of only a handful of big-endian architectures, which makes testing on it a great way to find endianness bugs in the BPF code. On the other hand, it's not in common use, available CPUs don't have great performance compared to CPUs for other architectures, and it's taking a lot of his time that could otherwise be directed to improving the test suite in other ways.

The participants didn't have a solid answer for him — and the s390x tests identified an endianness problem less than a week after the summit. So for the moment, the tests are still running and proving useful. In case anyone else wanted to step up and help maintain or expand the BPF tests, Solodrai went through a number of hard-earned lessons on how to do that.

His main advice was to use as little YAML as possible. "You really don't want to use YAML. You have to use YAML, but you'd much rather write a bash script." He also recommended setting things up to use sparse checkouts, environment variables rather than GitHub Actions inputs, and being cautious about using the built-in caching support since it has confusing rules. Reusable workflows are also only somewhat useful; they can help deduplicate code, but the security rules are tricky here too.

Solodrai warned people to remember that there are many ways to run code on different architectures, but most of them are actually QEMU under the hood. At one point, he had "an adventure" with three seemingly unrelated problems all cropping up at once; it turned out that he needed to manually install a newer QEMU that had fixed the bug.

Finally, Solodrai made sure that all of the attendees knew how to make sure the CI tests ran for their patch set: make a fork of the kernel-patches/bpf repository on GitHub, put the patch set on a branch, and create a PR. The automation will see the PR and kick off the tests.

At that point, the session opened up into a group discussion of the current coverage of the tests and what, if anything, they should be extended to cover. Solodrai suggested testing multiple compiler releases, long-term-support kernels, and 32-bit architectures. There was a lot of general enthusiasm about testing long-term support-kernels, and an acknowledgment that the other things would be useful as well if possible. Alexei Starovoitov specifically wanted to test LLVM 19, which has support for BPF arenas, to obtain better coverage in that area.

Solodrai was also interested to know which features of the BPF CI tests people were using. For example, he asked whether putting more effort into speeding up the tests would make a real difference for anyone's work. That question prompted extensive discussion, but the ultimate conclusion was that it would not — the tests are always going to be long enough to interrupt someone's work, at which point the difference between 45 minutes and two hours is not massively important.

Since the summit, Solodrai seems to have made good progress on putting the things discussed there into practice. The kernel's testing infrastructure is somewhat patchwork, with many subsystems and independent testing efforts each working on their own areas of interest. The BPF CI tests, however, seem to be increasingly useful and reliable.

Comments (none posted)

Reports from OSPM 2025, day one

By Jonathan Corbet
May 19, 2025
OSPM

The seventh edition of the Power Management and Scheduling in the Linux Kernel (known as "OSPM") Summit took place on March 18-20, 2025. It was organized by Juri Lelli, Frauke Jäger, Tommaso Cucinotta, and Lorenzo Pieralisi, and was hosted by Linutronix at Alte Fabrik, Uhldingen-Mühlhofen, Germany. The event was sponsored by Linutronix, Arm, and the Scuola Superiore Sant'Anna in Pisa.

The following contains summaries of the sessions; each summary is written by the session presenter. A recording of the entire summit is available as a playlist on the RetisLab YouTube channel. Photos of the event can be found on this Google Photos page or as a 965MB zip archive. The full set of slides from the sessions is also available.

Scheduling for core asymmetry and shared resources

Speaker: Morten Rasmussen (video)

The Linux scheduler currently represents CPU throughput, by capacity and efficiency, through an energy model (EM), which was introduced with energy-aware scheduling (EAS). The EAS EM has been around for years and is quite simple relative to the complexity of modern systems. The EM, therefore, often fails to capture platform-specific constraints, leading to suboptimal performance and efficiency. The aim of the talk was to discuss potential ways forward to close the gap between the scheduler and power-management frameworks in the mainline kernel and the increasingly complex solutions developed by vendors out of tree.

Previous studies have shown that power consumption and efficiency vary with the workload, which is not surprising, but the variation is significant and might be worth optimizing for. In mobile systems running Android, this and other aspects of optimization have been addressed by vendors with kernel modifications loaded in modules that attach functions to Android-specific vendor hooks. The number of vendor hooks in Android has grown to a level where most key scheduler mechanisms can be overridden by vendor modules.

A look at a common dynamic voltage and frequency scaling (DVFS) implementation for mobile systems and expected upcoming implementations of DVFS with shared compute resources illustrates how the current CPU-frequency and scheduler abstractions fall short. In the first example, a subset of CPUs shares a clock frequency with the interconnect and L3 cache, which means that L3 cache DVFS scaling impacts CPU efficiency and performance directly. In the second example, DVFS scaling is further complicated by having a shared compute unit used transparently by all CPUs in the system. Modeling the clock-frequency dependencies of both examples and taking these into account for task placement is nontrivial. Often, simple platform-specific heuristics can help, while coming up with a generic, future-proof EM and EAS policy is virtually impossible.

Two potential paths forward to improve mainline Linux were discussed:

  • Improve EAS and EM to model the platform in more detail. This was considered mostly unrealistic.
  • Work on expanding the architecture-specific callbacks from the scheduler to give architectures more influence on task-placement decisions. The scheduler already has a few callbacks to prioritize CPUs for certain systems. This could be extended to provide platform-specific preferred CPUs.

In the discussion, the gap between the mainline, Android, and sched_ext communities was acknowledged. Adding architecture-specific callbacks seemed the most promising approach, although concrete proposals need to be shared for discussion.

CAS/EAS on Intel client platforms

Speaker: Rafael J. Wysocki (video)

Wysocki took the stage to discuss advances in implementing energy-aware scheduling on Intel hybrid chips. He started with a recap of two OSPM-summit 2024 presentations, one from himself and one from Ricardo Neri, that focused on the problems this effort was facing. Namely, the implementation of full-scale invariance on Intel hybrid processors was problematic, but has been overcome since last year's summit. There were also problems with creating energy models for those chips. The latter is still an issue but, fortunately, it can be addressed by using artificial energy models containing abstract cost values rather than any remotely realistic power numbers. He has been working on this problem recently.

Next, Wysocki described scale invariance on the processors in question. There are two parts of it, capacity invariance and frequency invariance, each of which is implemented with the help of a function returning values between zero and SCHED_CAPACITY_SCALE, inclusive. One of them returns the CPU capacity relative to the capacity of the most performant CPU in the system and the other one returns the CPU frequency relative to the one at which the full capacity is reached.

Generally, on x86 processors, the frequency-invariance part is derived from the values of the APERF and MPERF performance counters; this has been the case for several years. MPERF counts at a certain reference frequency, while APERF counts at the current frequency of the CPU. Snapshot them at two points in time, compute the deltas, divide the APERF delta by the MPERF delta, and you get the average CPU frequency relative to the MPERF counting frequency. However, this frequency needs to be relative to the frequency at which the full CPU capacity is reached.

On hybrid Intel processors, the HWP (hardware P-state) interface is used and the frequency in question is proportional to the value of the HIGHEST_PERF field in the HWP capabilities (HWP_CAP) register, but that value is not in frequency units. It is an HWP performance level; the frequency corresponding to it can be obtained with the help of the performance-to-frequency scaling factor used by the intel_pstate driver for converting HWP performance levels to frequency values required by the CPU-frequency sysfs interface. This scaling factor is hard-coded in the driver for a few platforms but, in general, it can be derived from the information supplied by the ACPI CPPC interface. Even though this requires the platform firmware to get things right, intel_pstate is going to rely on it in the future.

Since intel_pstate uses the performance-to-frequency scaling factor already, it can readily compute the frequency corresponding to HIGHEST_PERF and divide it by the MPERF counting frequency, which can be read from the MSR_PLATFORM_INFO register. The result of this computation is what is needed to turn the ratio of the APERF and MPERF deltas into the number required for frequency invariance.

Once the frequency invariance has been taken care of, the capacity invariance is easy: multiply HIGHEST_PERF of the given CPU by SCHED_CAPACITY_SCALE and divide the resulting number by HIGHEST_PERF of the most performant CPU in the system.

Scale invariance implemented in accordance with the above description has been used for enabling capacity-aware scheduling (CAS) on Intel hybrid processors and, by itself, it has led to some nice reductions in energy usage. It causes high-utilization tasks to be migrated to high-capacity CPUs when there is no room for them on the low-capacity CPUs, and it does not require assigning high priorities to high-capacity CPUs for this purpose. Thus low-utilization tasks can stay on low-capacity CPUs even when the higher-capacity CPUs are not fully utilized and energy can be saved.

Of course, CAS is also required for supporting EAS, but the value added by the latter on top of the former turns out to be harder to demonstrate in practice than the value of CAS itself. Generally speaking, on low-power systems based on Intel hybrid processors, the difference made by EAS with respect to CAS appears to be in the noise.

Nevertheless, according to Wysocki, work on enabling EAS on Intel hybrid processors has been in progress and it all boils down to finding a suitable artificial energy model that will cause the scheduler to prefer low-capacity CPUs to high-capacity ones until CAS starts to move tasks in the opposite direction. Basically, such an energy model needs to assign higher cost values to high-capacity CPUs and lower cost values to low-capacity ones regardless of their current frequency (or, generally, performance level).

In the first energy model considered, one performance domain (PD) with only one performance level was created for each CPU type, so on a system with two types of CPUs there would be two PDs and all CPUs of the same type would belong to the same PD. The cost value associated with a given PD through its single performance level could be regarded as an inverse priority of the CPUs of the corresponding type.

That energy model was as simple as it could be on the scheduler side, and it allowed some wakeup-path computations to be avoided, but more complexity was involved in creating it on the driver side. First of all, the driver needed to know the type of each CPU to add it to the right PD; it would also need to track the connections between CPUs and PDs in order to create CPU masks needed for PD registration. Next, the PDs might need to be expanded after they had been registered; that required some changes in the energy-model management code, which was not prepared for dealing with this use case. Finally, the energy-model management code expected all of the CPUs in one PD to be of the same capacity, which did not play very well with systems where the so-called favored cores were present (as those cores are just like the other cores of the same type, but their HIGHEST_PERF is higher, so additional HWP performance levels are available on them).

Those difficulties did not prevent that energy model from being implemented in the first EAS enablement for Intel chips patch series posted near the end of 2024, and from being tested. The results, however, were not conclusive. Some improvements were observed, but mainly on high-power processors designed for desktop systems; on low-power systems, where energy usage improvements would be arguably more welcome, the difference made by EAS with respect to CAS alone was in the noise. Moreover, it was observed that EAS caused tasks to be migrated much more often between CPUs of the same type, which was attributable to using the highest spare capacity for CPU selection within one PD.

That caused Wysocki to consider an alternative energy model in which PDs were created per CPU and the dominant cost component that still depended on the CPU type was amended by a small contribution proportional to the current performance level of the CPU. Four performance levels, corresponding to 100%, 80%, 60%, and 40% of the full capacity, were added to the energy model for each CPU, so that tasks would not be migrated to another CPU of the same type until the utilization of the CPU they were running on reached 40% of its capacity. In turn, if all of the CPUs of the same type were 40% utilized, tasks would start to be migrated between them if one of them became 60% utilized, and so on. The migration of tasks between CPUs of different types would still only happen by means of CAS.

The alternative energy model was implemented in the second EAS enabling for Intel chips patch series posted right before the summit, so it was new at the time of the conference and the evaluation of it was still in progress.

As clearly demonstrated by Wysocki's work, EAS can be enabled on hybrid Intel hardware. However, it is somewhat heavyweight, so if the practical difference made by it continues to be hard to demonstrate convincingly, a different option may look more appealing. Specifically, what seems to be really needed is a way to move low-utilization tasks to low-capacity CPUs when there is spare capacity on them, to prevent the distribution of tasks in the system from remaining suboptimal for too long. It can be done by means of EAS, which is relatively expensive due to the EAS wakeup path overhead, but it can also be done with the help of a scheduler feature called "asym packing".

Asym packing was originally introduced as a way of supporting favored cores. It is based on assigning priorities to CPUs and moving tasks from lower-priority CPUs to higher-priority CPUs during load balancing. It is used on multi-threaded Intel hybrid processors, where using CAS is not practical; in that case, CPU priorities are simply based on maximum capacity. The obvious drawback of doing so is that low-capacity CPUs are then only used when the high-capacity CPUs are fully utilized, so the energy usage tends to be higher than it needs to be. Still, in principle, asym packing can also be used for moving tasks in the reverse direction, but for this purpose it needs to be made capacity-aware and the CPU priorities need to be based on energy efficiency.

Wysocki said that all of this could be done, so maybe asym packing would turn out to be a viable alternative to EAS, but the audience was skeptical about that prospect. The overall concern was that asym packing would not prevent tasks from being moved to CPUs that they should not be running on and taking them away from those CPUs later might not be sufficient to really improve the energy usage.

Scheduler interface evolution

Speaker: Vincent Guittot (video)

The talk was about scheduler interface evolution, specifically focusing on the sched_setattr() system call and how the parameters of run time, deadline, and period can be utilized to more effectively describe the behavior of different types of tasks. Guittot pointed out that these parameters are already used outside their original scope, the deadline scheduler, as the run time is used to set a custom slice duration for EEVDF. These three parameters can be used to describe several categories of tasks, and the parameters can be used by the scheduler and other frameworks to tune the behavior and the estimated utilization of the CPU.

Guittot started describing a matrix for these task types:

  • Fully periodic tasks are the most straightforward. They provide all information about their execution, but we don't often have such well-defined tasks.
  • Sporadic, latency-sensitive tasks are characterized by having deadlines that are critical for responsiveness, but their arrival or the interval between their executions are not predictable. This is probably the most interesting type of task that enables setting a CPU bandwidth without providing a period.
  • Sporadic running tasks only know their average running time but the period between two wakeups can fluctuate and its estimated utilization of the CPU as well. The running time can be translated into a utilization that will not vary with period.
  • Random, latency-sensitive tasks might only have a critical deadline by which they must finish their execution. The scheduler might have little or no information about their expected run time or how frequently they will need to run, but it can use such information to set the maximum slice, as an example.

A member of the audience pointed out that the scheduler often knows the deadline of a group of tasks but not always for individual tasks. The current proposal focuses on per-task attributes, since it doesn't seem easy to describe the attributes of a group of tasks if we can't even describe a single task. Graph algorithms should be evaluated when describing the groups of tasks.

In addition to setting the slice, the scheduler could compute peak and/or average bandwidth for a task or better estimate its utilization of the CPU. It can even take into account a deadline and use these metrics when selecting the CPU. Those metrics are not that far from those used by the deadline scheduler; an audience member said that the first version of the deadline scheduler would fall back to the completely fair scheduler policy when tasks exhausted their bandwidth

There were some concerns that user space can bias the values, but this is already the case with uclamp.

The presentation ended with some open issues related to the sum of estimated utilization which is often far higher than the maximum utilization of the CPU.

Scheduler Governors

Speaker: Steven Rostedt (video)

Rostedt led a session about scheduler governors, starting with a history of the Linux kernel scheduler that began with the original, single-run-queue scheduler. In 2002, Ingo Molnar created the O(1) scheduler, which had per-CPU run queues and some bitmask-based logic to find the next task to run in O(1) time. But it required heuristics to differentiate interactive and non-interactive tasks, so it could try to prioritize interactive ones. In 2004, Con Kolivas introduced "plugsched", a way to add different schedulers into the Linux kernel. Molnar was not pleased with this and said:

I believe that by compartmenting in the wrong way we kill the natural integration effects. We'd end up with 5 (or 20) bad generic schedulers that happen to work in one precise workload only, but there would not be enough push to build one good generic scheduler, because the people who are now forced to care about the Linux scheduler would be content about their specialized schedulers.

Later that year, Kolivas came back with a new scheduler called the "Staircase Scheduler" that replaced the heuristics with multiple run queues per CPU, where different queues had different priorities. As a task continued to run, it would move to lower-priority run queues and be preempted more often. Tasks that did not run on the CPU very long would stay in the higher-priority run queues and have lower latency.

In 2007, Kolivas improved his scheduler into the Rotating Staircase Deadline (RSDL) scheduler, which added a quota for the entire level of a run queue. Tasks that slept through a quota-change period would not be affected by it, giving it better latency. Several people said at the time that Kolivas's scheduler gave a better user experience on a desktop environment and there was a push to get it upstream.

During the debate about this scheduler, Molnar disappeared for three days and came back with a new scheduler called the Completely Fair Scheduler (CFS). This scheduler did away with run queues and used red-black trees instead. Tasks were added by their "wait time", and the scheduler code was broken up into separate files, making the interface a little more modularized.

Kolivas left kernel development for a while, but then, in 2009, came back with the Brain F*** Scheduler (BFS). It used a single run queue for the entire machine (Kolivas said it would work fine up to 16 CPUs) and created a new, non-privileged scheduling class (SCHED_ISO) that ran between SCHED_OTHER and SCHED_RT, but was limited to how many tasks in that class could run. BFS implemented an earliest eligible virtual deadline first (EEVDF) algorithm, but did not support control groups.

In 2011, Kolivas came back with a v2 of BFS that also introduced "skip lists". It had a O(1) algorithm for looking up the next task, a log(n) insertion, and made the scheduler a bit more topology aware. In 2016, Kolivas further improved the BFS scheduler, allowing it to scale better with more run queues; he called this scheduler the Multiple Queue Skiplist Scheduler (MuQSS). It implemented the BFS scheduler with a run queue on each CPU and was able to access the next task on other CPUs locklessly. It implemented an earliest effective virtual deadline first scheduler (loosely based on EEVDF).

In 2022, Tejun Heo introduced ext_sched_ext (or sched_ext for short). This was a way to have pluggable schedulers using eBPF programs with the scheduling class of SCHED_EXT, which ran below SCHED_OTHER. To use it, one had to set all tasks to this scheduling policy.

In 2023, Peter Zijlstra introduced a full-featured Earliest Eligible Virtual Deadline First (EEVDF). It replaced CFS, providing tasks with smaller deadlines with better latency. Eligibility (or the lack thereof) could still cause latency due to a task having run more than it should have.

Rostedt then talked about scheduler constraints. A scheduler must be fair, and give equal CPU time to each task of the same priority. It is responsible for quickly scheduling tasks that have interactive requirements. The scheduler must respect a task's affinity for which CPU it may run on. It also affects the power efficiency of the system, especially if there are multiple types of cores to choose from.

Rostedt then showed the graphs that were presented at the 2023 OSPM summit, which showed that, when comparing all the schedulers for a Chromebook work load, the MuQSS scheduler performed the best. Unfortunately, since the person who ran these benchmarks left Google and nobody has taken up the work recently, Rostedt didn't have any updated benchmarks for the latest EEVDF scheduler.

Rostedt talked about scheduling being hard and complex. But can it be simplified if the scheduler could be focused on a specific type of system. Rostedt broke up the types of systems that a scheduler could be used for:

  • Server: A system that deals with many users.
  • Desktop: A system that deals with a single user interacting with many applications.
  • Phone: A system that deals with a single user interacting with a single application.
  • Embedded: A system with no users but just deals with event stimuli.

Here is where Rostedt proposed a concept called "scheduler governors", making a point not to use the term "pluggable schedulers", as that usually implies changing the scheduler via a module or other out-of-tree source. He took the term from the power-management governors. The idea is that the governor would describe the system as a whole. A "server governor" would focus the scheduler to deal with multiple users, while a "desktop governor" would focus on scheduling tasks for a single user with many applications. If this structure had been in place when EEVDF came about, then one could have been able to easily benchmark EEVDF and CFS as they could have been implemented for different governors. Rostedt pointed out that sched_ext could still be used to tweak any of the governors.

During the discussion, Rostedt described the difference between governors and pluggable schedulers like sched_ext. Sched_ext may enable the scenario that Molnar was afraid of, where everyone develops their own scheduler, and ideas would be more fragmented. A scheduler governor would be in the kernel proper, where developers focusing on the same environment could easily collaborate on the type of scheduler they needed. Server folks would focus on improving the server governor, where those who care about a phone could focus on the phone governor. It was brought up that there may only need to be two governors, as the desktop, phone, and embedded may not have such a different environment that different governors would be useful. But there was some agreement that a scheduler for a server has different requirements than a user device.

It was mentioned that sched_ext could also be a governor, where the governor would give full control to a sched_ext eBPF program.

Thomas Gleixner was still not convinced. He said that Apple has a quality-of-service interface that works fine and that Linux should do something similar. Rostedt agreed, but also mentioned that Apple doesn't deal with servers. There is an effort to create a library that would let tasks state their scheduling requirements and the library would translate that to the kernel. But making this work with all applications is difficult.

Rostedt finished by saying that, at the bare minimum, he plans on working on cleaning up the scheduler code and make it more modular, where it could eventually have scheduler governors (at least out of tree perhaps). This work could also benefit sched_ext in the long run.

Further uclamp sum aggregation

Speaker: Hongyan Xia (video)

Xia proposed some improvements to the uclamp mechanism in a pre-recorded session viewed at the event.

The current uclamp implementation, max aggregation, has several drawbacks that are being addressed in multiple proposed patch series. Hongyan's work, uclamp sum aggregation, attempts to address these problems by proposing a different implementation that may simplify the design while improving the effectiveness for uclamp as a performance hint. He presented the concept and design of sum aggregation in last year's OSPM.

This year, with his experience on Android Dynamic Performance Framework (ADPF) and the interaction with applications, Hongyan has gathered further evidence and obtained new insights as to why sum aggregation makes sense in real-world scenarios like Chrome web browsing and virtual-machine CPU-frequency hints. To prove his claims, Hongyan presented benchmark results and energy numbers to show that the new implementation is more effective and energy-friendly. He also made a detailed decomposition to illustrate what part or parts of sum aggregation exactly contribute to the improved effectiveness. His findings add on top of last year's presentation to show that sum aggregation could be a nice alternative to the existing uclamp implementation as performance hints.

Extending EAS with push callback

Speaker: Vincent Guittot (video)

The talk addressed limitations in the existing EAS implementation and proposed additional changes on top of what is already under review on the mailing list. This presentation is a follow up on what has been presented at the Linux Plumbers Conference and the previous OSPM summit.

The current EAS makes scheduling decisions only when a task wakes up, which can lead to suboptimal CPU selection as thermal conditions and system load change over time. The proposed solution introduces a push mechanism to re-evaluate task placement and possibly migrate tasks to more suitable CPUs. This mechanism is not used by the fair-scheduling class, with the exception of active migration of the running task, as mentioned by the audience. But the periodic load balance implemented by CFS can't scale for EAS because it implies checking all runnable tasks on all CPUs. Instead, the push mechanism looks for opportunities to migrate a task when it is put back in the enqueued list after being preempted.

The current proposal is quite close to the mainline algorithm except that it adds a new level of decision for CPUs with the same cost level in a performance domain, which includes the maximum spare CPU capacity, the only parameter used currently. A member of the audience asked: why is the maximum spare capacity not enough? Utilization includes blocked load, which can make a CPU appear more loaded without any running tasks, as an example.

Then, Guittot explained what he considered to be a "stuck task": one that doesn't have a wakeup event anymore, or which wakes infrequently because of uclamp, thermal constraints, or simple scheduling delay when sharing CPU time. The proposed condition is simple: it compares the util_avg, util_ext and runnable_avg of the task with the CPU capacity or its uclamp constraint. A participant asked why push is not used for other cases than EAS. Guittot has not considered other cases, like idle load balance (ILB), because there were no strong requests raised by the community so far. Zijlstra noted that ILB has always been an issue because it can be expensive on large systems; the push mechanism could be a solution for migrating tasks between low-level cache domains.

In addition to the current proposal on the mailing list, Guittot proposed to add more events to trigger the push of other types of tasks, such as a short-slice task that is being being preempted, a task with a changing profile, or just because it has run enough on the same CPU that the system conditions have changed. He mentioned 32ms, which is the half-life period, as a suitable value for "enough".

Another proposal was to add more hints in the least-loaded-CPU selection, like the slice duration of a task in order to minimize the scheduling delay. Finally, the last change proposal is about biasing the energy decision; the energy delta between two performance domains is sometimes small, but can prevent migrating a task onto a CPU with a shorter scheduling delay. The problem remains to find a way to bias this decision without adding a new knob.

At the end of his presentation, Guittot described a few open issues with the overutilized state, which is far too aggressive, and the run-to-parity feature, which can add significant scheduling delay when a short-slice task exhausts its slice. Then the discussion moved on discussing uclamp max versus sum and ended on this topic

Comments (none posted)

Page editor: Joe Brockmeier
Next page: Brief items>>


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds