LWN.net Weekly Edition for October 21, 2021

Welcome to the LWN.net Weekly Edition for October 21, 2021

This edition contains the following feature content:

• Empowering users of GPL software: a different type of GPL-enforcement lawsuit from the Software Freedom Conservancy.
• A viable solution for Python concurrency: there may finally be a way to get rid of the Python global interpreter lock.
• Possible changes to Debian's decision-making processes: tweaks to make Debian's democracy work more smoothly.
• A disagreement over get_mm_exe_file(): enforcing code quality by removing exports.
• Moving toward Qubes OS 4.1: what's coming in the next release of this "reasonably secure operating system".

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Empowering users of GPL software

A new style of GPL-enforcement lawsuit was filed on October 19 by Software Freedom Conservancy (SFC) against television maker Vizio. Unlike previous GPL-enforcement suits, which have been pursued on behalf of the developers and copyright holders of GPL-licensed code, this suit has been filed on behalf of owners of the TVs in question. The idea that owners of devices that contain code under the GPL have the right to access that code seems clearly embodied in the license, but it remains to be seen if the courts will decide that those owners have the legal standing to sue for relief.

Novel

There are some other ways this lawsuit stands out from previous efforts. For one thing, it comes with an extensive press kit to help media outlets (and others) understand the suit and its ramifications. The press kit fills in some of the details from the complaint, but also provides a "Q&A" section, biographies of SFC spokespeople, quotes from various industry experts, a glossary, and more. It is an excellent summary of the background for those who are not well-versed in our community and its licenses, but there is quite a bit of interest in that document even for LWN readers and others who are generally knowledgeable about such things.

The press kit makes clear that an additional goal of the lawsuit is to educate users about their rights under the GPL and why those rights should matter to them. The lawsuit could be seen as something of an attention-grabbing effort to try to help ensure that the benefits of licenses like the GPL actually end up reaching the users who are supposed to be the beneficiaries of the source-code disclosure required. Doing so is in keeping with the public-benefit role of charities like SFC, but is typically not part of the strategy embodied in lawsuits, which are clearly targeted at the legal system—and the defendant, of course.

Unlike many high-profile lawsuits, SFC's suit is not asking for monetary damages from Vizio; instead it is asking the court to order Vizio to comply with the terms of the GPL and LGPL that cover code in its TVs. Beyond that, it is asking the court to declare that the terms and conditions of the GPL licenses require source-code disclosure, effectively determining that the GPL operates the way that the free-software community believes that it does—and that Vizio is in breach of the license. The only monetary requests are for reimbursements of the costs and attorney fees needed to pursue the lawsuit.

The 27-page complaint is, as might be guessed, a detailed description of the problems that SFC sees in the actions of Vizio. The next step will presumably be a response to the complaint, which will be public as well. This stands in contrast to the confidential nature of many of the filings in the SFC's most recent GPL-enforcement suit, Christoph Hellwig v. VMware, which was filed in Germany and eventually dismissed on procedural grounds. That lack of transparency was one reason to file in the US, as SFC executive director Karen Sandler said in the press kit:

Given that so many GPL violators are based here in the U.S. (including Vizio) and because of the transparency of the U.S. legal system, it makes sense as a venue for this litigation. In particular, we listened to individuals in the FOSS legal community who complained that the Hellwig vs. VMWare case in Germany—which we partially funded—was not transparent. We worked very hard to compel more transparency in the German legal system, but we were simply unable to do so. We heard those complaints and listened to the FOSS legal community and its suggestion that transparent litigation about the GPL was the right way to go.

Details

As with many (most?) "smart" TV makers today, Vizio uses Linux and a variety of other GPL/LGPL-covered code in its SmartCast platform that runs on its devices. According to the complaint, Vizio TVs that SFC purchased did not come with offers to provide the source code for those components, nor with the source code itself (on a CD, say), either of which is required in order to comply with the licenses.

SFC became aware of the problem and purchased one of the TVs in question in March 2018. It found that the Linux kernel was present on the device, but there was no offer or source code. SFC contacted Vizio in August 2018 about that problem, which started a lengthy process where Vizio tried to provide the source code for the kernel.

Throughout 2019, Vizio provided six separate versions of the source code to SFC, but none of them would actually compile, according to the complaint. Each time, SFC provided a detailed report to Vizio "showing what was provided and what it believes, after its analysis, to be missing". Toward the end of 2019, the organization participated in a conference call with Vizio and its chip supplier to discuss the problems with the source code provided. In January 2020, Vizio expressed hope that the supplier would come up with the missing pieces soon, but that was the last communication SFC had with the company—even after six follow-ups in the following five months.

In mid-2021, SFC purchased three different models of Vizio TVs. In looking at the firmware on the devices, SFC found 13 more GPL-covered components (including BusyBox, Bash, and Awk) and 11 LGPL-covered components (including the GNU C library, FFmpeg, and systemd). Those three models are specifically at issue in the suit, though it is clear that the hope is that any ruling will cover all models that Vizio makes or will make that contain GPL code.

The question of legal standing would seem pivotal in this case. Suits can only proceed if the plaintiff can show "sufficient connection to and harm from the law or action challenged"; a determination of a lack of standing for SFC (as a placeholder for all users) could derail the suit without the court ever looking deeper into the issue. The procedural issue that caused the dismissal in Germany was, effectively, that Hellwig did not show enough copyright in the kernel to bring his suit. Here, instead of a copyright holder suing over the misuse of their copyrighted code, we have users who are not parties in the license directly asking for relief from the courts.

The complaint lays out the case for granting standing starting at clause 107 on page 20. A few separate clauses get to the heart of the argument SFC is making:

110. Therefore, a motivating purpose—indeed, the sine qua non—of the GPLv2 and LGPLv2.1 is to provide the Source Code to downstream recipients of computer programs covered by those license agreements.

[...] 112. It is consistent with the objectives and express language of the GPLv2 and LGPLv2.1 to permit recipients of executable software covered by those license agreements, such as purchasers of Vizio smart TVs, to seek court assistance to enforce their right to the corresponding Source Code.

[...] 116. Plaintiff is not a contracting party to the GPLv2 or the LGPLv2.1 as that agreement relates to the Vizio smart TVs, in that it is not asserting that it is the licensor of the software used in Vizio smart TV s or any works based on the software or derivative thereof.

117. Plaintiff is a member of a class of persons for whose benefit the GPLv2 and LGPL v2 .1 were created and intended.

[...] 121. As a third-party beneficiary under the GPLv2 and LGPLv2.1, Plaintiff has a right to the Source Code corresponding to the executables of the SmartCast Programs at Issue and Linking Library Programs that reside on the Vizio smart TVs purchased by it.

As can be seen above, the complaint is somewhat repetitive and verbose, which is normal for a filing of this sort. It does make for interesting reading for those who want a deeper look into what SFC alleges and what it thinks a judge should do if they agree.

Often, in the case of a lawsuit against a device maker like Vizio, it is filed as a class action, where multiple consumers harmed by the actions of the manufacturer join together in the suit. Typically, those kinds of suits target a monetary penalty that gets shared among those who join into the suit—after the lawyers take their, usually hefty, fees off the top. But, as the Q&A notes, the penalties requested in this case obviate the need for multiple plaintiffs:

Furthermore, pursuant to "The Principles of Community Oriented GPL Enforcement," the lawsuit does not prioritize financial remedy over compliance. This lawsuit seeks the most important remedy for the public good: release of the Complete, Corresponding Source (CCS) for all GPL'd components on Vizio TVs. Once that is achieved, the benefit is immediately available to not only all who purchased a Vizio TV, but also to the entire FOSS [Free and Open-Source Software] community.

Looking ahead

This is an interesting case for a number of different reasons, though it could well play out over quite a few years before we really know its outcome. One need only look at how long Google v. Oracle—or the seemingly never-ending SCO v. IBM case—took to resolve to realize that this case could stretch out for a decade or more. Assuming the facts are being reported accurately in the complaint—and they seem rather cut and dried—it is difficult to see what argument Vizio could make to short-circuit its GPL obligations, except, of course, for SFC's standing.

Vizio could also presumably cut the lawsuit off at the knees by simply complying with the licenses. Perhaps that is hard to do, however, as it would seem that the chip supplier is unable to produce the complete corresponding source—at least so far. From a business perspective, though, one might guess device makers would be leery of seeing an adverse judgment in this case. If any buyer of a device can successfully win a lawsuit of this nature, it greatly expands the number of potential plaintiffs out there. On the other hand, it also means that the arguments made about copyright assignment being necessary for GPL enforcement will have been weakened—or rendered moot.

But, of course, if users are found not to have standing, there is no shortage of potential targets for a GPL-enforcement suit by copyright holders of Linux and other FOSS components. The right answer, as always, is for companies to simply voluntarily comply with the licenses on the plethora of high-quality code that our communities make available to them. SFC is slowly ratcheting up the pressure to hopefully someday make that happen as a matter of course.

Comments (75 posted)

A viable solution for Python concurrency

Concerns over the performance of programs written in Python are often overstated — for some use cases, at least. But there is no getting around the problem imposed by the infamous global interpreter lock (GIL), which severely limits the concurrency of multi-threaded Python code. Various efforts to remove the GIL have been made over the years, but none have come anywhere near the point where they would be considered for inclusion into the CPython interpreter. Now, though, Sam Gross has entered the arena with a proof-of-concept implementation that may solve the problem for real.

The concurrency restrictions in the CPython interpreter are driven by its garbage-collection approach, which uses reference counts on objects to determine when they are no longer in use. These counts are busy; many types of access to a Python object require a reference-count increment and (eventually) decrement. In a multi-threaded program, reference-count operations must be performed in a thread-safe manner; the alternative is to risk corrupted counts on objects. Given the frequency of these operations, corruption in multi-threaded programs would be just a matter of time, and perhaps not much time at that. To avoid such problems, the GIL only allows one thread to be running in the interpreter (i.e. to actually be running Python code) at a time; that takes away almost all of the advantage of using threads in any sort of compute-intensive code.

The reference-count problem can be trivially solved (for a relatively advanced value of "trivially") by using atomic operations to increment and decrement the counts. There is just one little problem with this solution, as outlined in this design document posted by Gross:

The simplest change would be to replace non-atomic reference count operations with their atomic equivalents. However, atomic instructions are more expensive than their non-atomic counterparts. Replacing Py_INCREF and Py_DECREF with atomic variants would result in a 60% average slowdown on the pyperformance benchmark suite.

Given that the vast majority of Python programs are single-threaded (and likely to remain so), it is not surprising that there has never been much appetite for solutions that impose this sort of cost.

Biases, immortals, and deferrals

Gross has taken three different approaches to the CPython reference-count problem, the first of which is called "biased reference counts" and is described in this paper by Jiho Choi et al. With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast, non-atomic instructions. Any other thread accessing the object will use atomic operations on the shared reference count.

Whenever the owning thread drops a reference to an object, it checks both reference counts against zero. If both the local and the shared count are zero, the object can be freed, since no other references exist. If the local count is zero but the shared count is not, a special bit is set to indicate that the owning thread has dropped the object; any subsequent decrements of the shared count will then free the object if that count goes to zero.

This algorithm improves reference-count performance because, of all the objects that any thread will create, few will be shared with other threads. So, most of the time, the shared reference count will be unused and the cost of using atomic operations to manipulate that count will be avoided.

There are, naturally, some subtleties in how the reference counts are handled. One of those is that, for reasons to be described next, the two least-significant bits of the local reference count are reserved. An increment to the local reference count, thus, adds four to that count. These details are hidden in the Py_INCREF() and Py_DECREF() macros, so most code need not be aware of them.

Some objects are heavily shared between threads, though; these include singletons like True, False, and None, as well as small integer values, some type objects, and more. These objects will also never go away during the execution of the program — they are "immortal" objects for which reference counting is a waste. Gross's CPython interpreter marks these objects by setting the lowest significant bit in the local reference count. If that bit is set, the interpreter doesn't bother tracking references for the relevant object at all. That avoids contention (and cache-line bouncing) for the reference counts in these heavily-used objects. This "optimization" actually slows single-threaded accesses down slightly, according to the design document, but that penalty becomes worthwhile once multi-threaded execution becomes possible.

Other objects in Python programs may not be immortal, but they are still long-lived; functions and modules fall into this category. Here, too, it can make sense to avoid the cost of reference counting. The idea makes even more sense when one realizes that many function and module objects, by virtue of appearing in the globals dictionary, essentially form reference-count cycles anyway and their counts will never go to zero. For these objects, a technique called "deferred reference counting" is used; the second-least-significant bit in the local reference count is set, and (most) reference counting is skipped. Instead, a garbage-collection pass is used to find and free unused objects.

"Most" reference counting is skipped because the CPython interpreter does not, on its own, have a complete picture of whether an object using deferred reference counting is truly unused. Specifically, extension code written in C could be holding references that the interpreter cannot see. For this reason, reference counting is only skipped within the interpreter itself; any other code will manipulate the reference counts as usual.

Other changes and results

The reference-counting changes are a key part of Gross's work, but not all of it. The interpreter's memory allocator has been replaced with mimalloc, which is thread-safe, fast, and is able to easily support garbage-collection operations. The garbage collector itself has been modified to take advantage of mimalloc, but is still "a single-threaded, stop-the-world implementation". A lot of work has gone into the list and dict implementations to make them thread-safe. And so on.

Gross has also put some significant work into improving the performance of the CPython interpreter in general. This was done to address the concern that has blocked GIL-removal work in the past: the performance impact on single-threaded code. The end result is that the new interpreter is 10% faster than CPython 3.9 for single-threaded programs. That will certainly sweeten the pot when it comes to acceptance of this work, though, as Guido van Rossum noted, the Python developers could always just take the performance improvements without the concurrency work and be even faster yet.

That seems like an unlikely outcome, though, if this work stands up to closer scrutiny. When pointed to the "ccbench" benchmark, Gross reported speedups of 18-20x when running with 20 threads. That is the kind of concurrency speedup that Python developers have been wanting for a long time, so it is unsurprising that this work has seen an enthusiastic reception. As an added bonus, almost all Python programs will run on the modified interpreter without changes. The biggest source of problems might be multi-threaded programs with concurrency-related bugs that have been masked by the GIL until now. Extensions written in C will need to be recompiled, but most of them will not need to be changed unless they (as some evidently do) access reference counts directly rather than using the provided macros.

The end result thus appears to be a GIL-removal effort that has a rather better-than-average chance of making it into the CPython interpreter. That would be cause for a lot of rejoicing among Python developers. That said, a change this fundamental is unlikely to be rushed into the CPython mainline; it will take a lot of testing to convince the community that it is ready for production use. Interested developers may be able to hasten that process by testing this work with their programs and reporting the results.

Comments (35 posted)

Possible changes to Debian's decision-making processes

The name Debian brings to mind a Linux distribution, but the Debian project is far more than that; it is an ongoing experiment in democratic project governance. Debian's processes can result in a lot of public squabbling; one should not lose track, though, of the fact that those processes have enabled a large community to maintain and grow a complex distribution for decades without the benefit of an overseeing corporate overlord. Processes can be improved, though; a recent proposal from Russ Allbery gives an interesting picture of where the pain points are and what can be made better.

Decision-making in Debian

To a great extent, Debian leaves decisions in the hands of its individual developers. A developer's package is their castle, and they can generally manage it as they see fit. That freedom is somewhat constrained by the Debian constitution and the extensive Debian policy manual, both of which are designed to ensure that both developers and the packages they create all get along. Most of the time, this process just works, the project generates (mostly) regular releases, and users are happy.

Occasionally, though, some sort of intervention is required; two of the mechanisms provided by the project for such cases are the Technical Committee and general resolutions. The Technical Committee is empowered to make decisions on technical policy and may, in extreme cases, override Debian developers if their actions are seen as sufficiently damaging to the distribution. General resolutions can, by way of a vote of the project membership, change or override decisions made by the Technical Committee (or others), set new policies, or amend the constitution.

Voting is a key part of decision-making at levels above the individual developer. This is not particularly unusual in the free-software community; many projects make decisions by a vote of either the general membership or some sort of elected (via a vote, usually) representatives. Debian is nearly unique, though, in the way it decides what its members will vote on. Rather than simply being presented with a list of choices, Debian developers create those choices themselves, often in great number, and often with a lot of associated discussion. The creation of the ballot is the important part of a Debian resolution; the vote at the end is just calculating the final score.

This process is designed to create outcomes that reflect, as well as possible, the will of the project as a whole. Debian's voting scheme allows a ballot to contain numerous options with small differences without fear of splitting the vote in a way that causes a relatively unpopular option to ultimately prevail. At its best, it creates ballots where developers can vote for the options they want rather than just voting against the worst case.

At times, though, this process has not worked as well as many would have liked. The long and painful discussion leading up to the adoption of systemd involved an extended and contentious discussion in the Technical Committee over the creation of the ballot. That discussion only came to an end when the committee chair, Bdale Garbee, cut it short by calling for an immediate vote on a simplified ballot — a vote leading to a split decision that was resolved by Garbee's casting vote. The decision was made, but nobody was particularly happy with how it came about.

A similar problem was seen in 2019 with yet another systemd debate, this one using the general-resolution process. Then Debian project leader Sam Hartman upset some participants by calling for a vote on the resolution as soon as he possibly could, before some developers felt that their ballot options were ready. An unsuccessful attempt to override this call was made, and the vote on the seven-choice ballot went forward. Once again, some developers felt that the process had been abused to the detriment of the project as a whole.

Toward an improved process

Allbery's proposal would change the process in ways intended to avoid this kind of outcome. For the Technical Committee, the new process would allow any member to call for a vote on an open resolution, with the vote beginning immediately. If, however, any other member objects to the vote within 24 hours, the vote will be canceled. Had this rule been in place during the systemd deliberations, the vote that brought things to an end would not have happened in the way it did. Another change, though, says that voting begins two weeks after a resolution is first proposed; voting starts automatically if nobody has successfully called for a vote by that point. So, in the systemd case, the ballot-creation process would not have dragged out for so long.

For votes on general resolutions, the length of the discussion period prior to the actual vote is determined by the project leader. That is still true with the proposed changes, but the new rules would set the length of that period to be between two and three weeks. Any substantive changes to the ballot (the addition of new option, for example) will force the discussion period to stay open for at least another week, except that this period still cannot exceed the maximum. Those changes cannot be made if the discussion period cannot be extended for at least 24 hours. The project leader has the power to either shorten or lengthen the discussion period by a week, but any shortening of the discussion period must still leave it open for at least 48 hours.

The purpose of these changes in both cases is apparently to give a relatively deterministic answer to the question of when a resolution will be voted on. In both cases, an upper bound is placed on the discussion time to ensure that things come to a timely close, but there are also mechanisms to prevent the calling of a vote before the parties involved have a chance to prepare their ballot options. If these changes are adopted, and if they work as intended, contentious issues should be resolved relatively quickly with fewer participants feeling that they have been outmaneuvered by procedural tricks.

It is possible to see these rule tweaks as a sort of attempt to apply a technical solution to the social problem of Debian's sometimes-messy decision-making processes. Allbery described his motivation this way:

I would encourage everyone to not discount the merits of having a clear process, even if it feels nit-picky and constraining to specify one in advance. My experience in multiple heated debates in Debian, and in similar problems in other governance debates and on-line communities, is that having good, clear, and previously-agreed process is exactly what creates the space for people to be gracious and collaborative even when they strongly disagree with the opinions of others.

Additionally, he said, it is easier to accept a decision that one doesn't agree with if one doesn't feel outmaneuvered during the process.

An earlier version of these changes generated a fair amount of discussion but little in the way of opposition or substantive changes. Perhaps the biggest point of discussion was a suggestion that a tie vote in the Technical Committee should automatically trigger a general resolution to make the decision rather than relying on the chair's casting vote. In the end, though, the consensus seems to be that such a provision would add more complexity for a small benefit, so it seems unlikely to happen.

In response to the second posting, Timo Röhling suggested fixing the length of the discussion period with almost no provision for changes; that, Röhling said, would minimize attempts to game the rules and would make things more predictable for developers. Allbery expressed some sympathy for that point of view, but also said a bit of flexibility could help the project deal with urgent issues in a timely manner. Unless others express views on the length of the discussion period, the proposal seems unlikely to change.

There seems to be a fair amount of support for Allbery's proposal but, in this case, "a fair amount" may not be enough. This is a constitutional change, meaning that a general resolution vote (under the old rules) will be required, and the resolution must pass with a 3:1 supermajority. Allbery does not want to actually propose a general resolution until he feels that it has a good chance of attaining that level of support, so he has asked for anybody who would not support it to explain why. There has been little response to the second posting, which is perhaps a sign that the project is, indeed, ready to make these changes.

Comments (10 posted)

A disagreement over get_mm_exe_file()

Differences of opinion over which kernel symbols should be exported to loadable modules have been anything but uncommon over the years. Often, these disagreements relate to which kernel capabilities should be available to proprietary modules. Sometimes, though, it hinges on the disagreements over the best way to solve a problem. The recent discussion around the removal of an export for a core kernel function is a case in point.

Loadable modules, of course, are chunks of kernel code that are loaded into the core kernel after the system boots. Most modules are device drivers, but a surprising amount of kernel functionality can be built in modular form. While code that is built into the kernel can use any symbol that is accessible via the usual C scoping rules, loadable modules are rather more constrained; they can only use symbols that have been explicitly exported to them. In theory, the exported-symbol interface is tightly regulated; in practice, tens of thousands of symbols have been exported over the years without a lot of oversight. That said, the community still sees occasional disagreements when a module developer wants to use a symbol that core-kernel developers do not wish to export.

The AMD GPU developers have recently run into a problem. While much user-space graphics code has long since moved over to using the atomic mode-setting API, it would seem that Chrome OS has only partially made that transition. By using a mix of old and new(ish) APIs, Chrome OS can get itself into a position where the hardware will no longer cooperate with it; to prevent that from happening, the amdgpu driver restricts the use of overlay planes for all clients. This message from Simon Ser describes the situation in more detail and, undoubtedly, more correctly.

This restriction makes Chrome OS work, but it also unnecessarily constrains other clients that do not mix the APIs in this way. In an attempt to get around this problem, Ser added a patch that calls get_mm_exe_file() to get the name of the program that user space is running. If (and only if) that name is chrome, the restriction on overlay planes is enforced. This allows Chrome OS to continue working in its accustomed manner while giving more freedom to other clients.

What Ser didn't know was that, back in September, Christoph Hellwig had removed the export for that function, making it unavailable to loadable modules. Once Ser's patch hit linux-next, its reference to a symbol that was no longer exported caused the build to break; linux-next maintainer Stephen Rothwell then duly reported the break back to the relevant maintainers. At that point, Ser became aware that his change was not going to work in the mainline kernel.

Ser's first response was to ask whether get_mm_exe_file() could be exported once again, since the amdgpu driver now uses it. Hellwig made it clear that he was not interested in doing that. In that message and those that followed, Hellwig was rather less than polite in his explanations, even after Ser requested a change of tone. There was, however, a real point to be made despite the way in which it was conveyed.

Why should the kernel not export get_mm_exe_file()? Hellwig's point is that kernel code should never vary its behavior based on the name of the program that is running in user space. That name is entirely under the control of user space and can change at any time, either for malicious reasons or just in the course of normal development. Beyond that, Chrome OS will, one assumes, eventually be fixed to no longer mix APIs in this way, at which point the check in the amdgpu driver will no longer be necessary, and may indeed be harmful. There will be no way to know, though, when that check can be taken out. So this little quirk, based on the name of the running program, is likely to stay around for a long time, long past when it is actually useful.

The problem remains, though, that changing the behavior of the amdgpu driver will break existing Chrome OS systems. The correct solution, according to Hellwig, is to require user space to explicitly opt into the less-constrained behavior. Chrome OS will not take that option and will continue to work; more modern applications can request that the restriction be removed. Ser didn't like this solution: "No, we can't have a 'I_AM_NOT_BROKEN' ioctl for each and every uAPI mis-use. User-space detection has been determined to be the best course of action." Hellwig stood firm in his insistence that, if an API change breaks user space, the only correct solution is a new call to enable that new behavior.

On October 14, Ser reverted the patch in question in the amdgpu tree. It does not appear that the separate API call is forthcoming, though; instead, it looks like the original patch will just be carried in the Chrome OS kernel instead. That, too, is a reasonable solution to this problem; the Chrome OS developers will know when they can drop that patch and, meanwhile, it will not clutter up the mainline kernel.

This little episode shows how control over the exporting of symbols to modules is used to constrain what those modules can do in the hope of improving the quality of the kernel's code base as a whole. The end result in this case looks to be headed in the right direction, even if the path to that solution was more unpleasant than it needed to be. The kernel project has gotten better over the years at upholding technical standards while treating developers in a respectful way, but there is clearly still progress to be made on both fronts.

Comments (40 posted)

Moving toward Qubes OS 4.1

On October 11, the first release candidate for Qubes OS version 4.1 was announced. Qubes OS is a security-oriented desktop operating system that uses multiple virtual machines (VMs or "qubes") to isolate various types of functionality. The idea is to compartmentalize different applications and operating-system subsystems to protect them from each other and to limit access to the user's data if an application is compromised. Version 4.1 will bring several important enhancements to help Qubes OS continue to live up to its motto: "A reasonably secure operating system".

It has been nearly five years since we looked at Qubes OS 3.2, though we have checked in on it a few times since we first wrote about it back in 2010. As with much in the security world, there are tradeoffs to be made when using Qubes OS, but it provides a level of security that is hard to find elsewhere. In addition, it does so using Linux and other open-source tools, so that users can inspect and modify the system as needed.

Moving the GUI out

Qubes OS uses the Xen hypervisor and an ever-growing set of qubes to isolate various pieces. The Xen Dom0 top-level domain, which runs the host operating system, is the most privileged part of the system after the hypervisor. It provides the administrative interface for the system as a whole, but also acts as the display manager for all of the qubes. For one thing, being in control of the display manager allows Qubes OS to enforce colors on window borders to indicate which qube they belong to. One of the headline features of Qubes OS 4.1 is splitting out the display handling into a separate GUI domain.

There are multiple reasons for separating out the GUI, but reducing the functionality in Dom0 is one of the primary motivations:

One of the biggest security concerns at the moment for Qubes is how much power is in dom0. Once a person has access to it, they can do anything: and while we separate it quite effectively from what is running inside application qubes, dom0 is still a big, bloated and complex domain that performs many disparate functions. It handles managing other domains, display and graphical interfaces, multiple devices (including audio devices), memory and disk management, and so on.

Qubes OS already has a GUI protocol that is used to compartmentalize the graphics handling: "Applications in VMs can't talk to GUI toolkits in dom0 other than through a very limited Qubes-GUI protocol, and GUI toolkits in application VMs can't talk directly to dom0's X server." But that mechanism requires Dom0 to directly map the memory of the qubes for window buffers, which is worrisome. Duplicating that ability in a separate domain would effectively result in having two Dom0s, thus increase the attack surface.

In version 4.1, the protocol has been changed so that the GUI handling does not require the ability to reach into arbitrary application qube memory. It uses "an abstract mechanism in which the qube has to explicitly allow access to a particular memory page". There is a concrete implementation for Xen (using the grant table mechanism), but other mechanisms could be used for other environments (e.g. KVM).

Beyond the memory handling, the existing graphics handling relied on the ability of Dom0 to control any of the qubes; a less-privileged GUI domain still needs to be able to do that, but in restricted manner.

But if we want to implement a more contained and less-privileged GUI domain, it would defeat part of its purpose to just permit it to run any sort of qvm-run do-what-I-want in any of the managed qubes. Qubes 4.0 introduces a special qubes.StartApp qrexec service that runs only applications defined inside the target qube (currently defined via .desktop files in Linux and .lnk files in Windows, placed in predetermined directories). This mechanism allows a qube to explicitly define "what the GUI domain can execute inside me," and not just hand over all the power to the managing domain.

The new GUI domain is an experimental feature for Qubes OS 4.1. As is noted in the future plans section of the lengthy blog post describing the feature, it is taking the system further in the direction laid out in the original Qubes architecture specification—from 2010.

Qrexec

Another large piece of new functionality for Qubes OS 4.1 is improvements to the qrexec facility and its policy handling. Qrexec provides a remote procedure call (RPC) mechanism for inter-qube requests, but, obviously, there need to be limits on what can be requested:

Of course, allowing everything all the time would be extremely dangerous. Thus, a part of qrexec called "qrexec policy" is also used to enforce who can do what and where. Furthermore, we want to be able to audit what was done, and this is provided with the logging capabilities of qrexec. The how is controlled by qrexec services, which are executed by qrexec and also must be designed in a secure and resilient way.

That detailed blog post describes several areas where qrexec has been upgraded for 4.1, starting with changes to the policy format itself. The Qubes OS 4.0 policy format, which will continue to be supported, split the policies up by the service they applied to, leading to dozens of separate files.

Under the new format, the policy for a single qube is much easier to find, parse, and edit. [...] For example, it's trivial now to forbid all calls for a given qube or forbid all with a list of exceptions. This allows for a more qube-centric approach to policies, and while the results are theoretically the same, there's a huge difference between a couple of rules, all in one [place], and dozens of rules spread across dozens of files.

In addition, the policy system is designed to "fail closed"; any syntax errors or other problems in the policy files will result in no permissions being granted by the qube. This is similar to the behavior of sudo and other security-focused tools.

The performance of qrexec has also been improved substantially for this release. Data is now transferred in much larger chunks (64KB rather than 4KB), which helps speed up large data transfers. In addition, there is a new policy daemon, rather than starting and loading a policy-handling process for each qrexec request. That cuts down the initialization time, which is important for repeated calls.

Reproducibility

In order to be sure that a particular binary has been created from the source code that it is purported to come from, there needs to be a way to reliably reproduce a binary from the source. That is what the Reproducible Builds project is all about. The Qubes project has been working on incorporating reproducible builds into its continuous integration (CI) infrastructure.

In particular, the current goal is to be able to build the Qubes OS Debian templates solely from packages that can be built reproducibly. Templates in Qubes OS are VM images that can be used to start an application qube quickly based on the template. The qube will have read-only access to the root filesystem of the template, so that the same root filesystem can be shared with multiple application qubes. There are official templates for several variants of both Fedora and Debian, as well as community maintained templates for several other distributions.

Debian has been at the forefront of reproducible builds for some time now, so it is no surprise that is where Qubes OS is focusing its current reproducible-building efforts. But Qubes ran into some problems retrieving older versions of Debian packages from the snapshot.debian.org service, which is known to have limitations that make it less that fully reliable for this purpose. That led the project to build its own snapshot service and rebuild Debian packages using it.

As of early October, 80% of the Debian unstable repository has been successfully rebuilt. The process was restarted for the Debian 11 ("bullseye") repository, which still has "a couple thousand package rebuilds remaining". Once that is done, the rebuilding of unstable will be completed. This is all being done on several servers hosted at home by Frédéric Pierret.

The next goal is to have the Debian templates reject the installation of any packages that cannot be verified as having come from the source code claimed. There is also work being done to add Fedora into the mix. It is notable that the Reproducible Builds project is using the results being reported for Qubes OS as "an example of what it has been advocating for years".

And more

There are other features coming in the release, of course, as well as lots more details on the pieces we looked at above. The release notes have a longer list of new features, which links to the closed issues for the release at the project's GitHub repository. As might be guessed, there is a long list of bug fixes made for 4.1 as well.

The last major version of Qubes OS, 4.0, was released in 2018. A half-year later, project founder Joanna Rutkowska stepped down from her leadership role in order to work on other projects. That kind of change can be highly disruptive, but the project seems to have come out of it just fine. There have been several minor releases over the intervening time—the most recent was Qubes OS 4.0.4 back in March—as well as a continuing stream of updates to address Qubes security bulletins.

According to the announcement, the second release candidate will come in "a few weeks to a few months" depending on the number and severity of the bugs that are found. If we use the 4.0 release as a guide, we can perhaps expect five release candidates over eight months before a final release of 4.1. More testing and fixing would likely speed the whole process up, of course, but a late (northern hemisphere) spring or early summer final release seems like a reasonable guess at this point.

Comments (3 posted)