Smaller Fedora quality team proposes cuts

Fedora's quality team is looking to reduce the scope of test coverage and change the project's release criteria to drop some features from the list of release blockers. This is, in part, an exercise in getting rid of criteria, such as booting from optical media, that are less relevant. It is also a necessity, since the Red Hat team focusing on Fedora quality assurance (QA) is only half the size it was a year ago.

The team is responsible for a host of activities which include testing of software, running test days, maintaining tools for test automation, and coordinating the Fedora release process with the release engineering team. The quality team is composed of Red Hat employees and Fedora community contributors, but it is fair to say that the bulk of the team's work is done by those employed to by Red Hat.

Unfortunately, according to an announcement by Kamil Páral, a member of the team, there is a somewhat urgent need to reduce its workload. Six out of ten Red Hat employees who had been working on the team have chosen to move to other teams within Red Hat over the past nine months, or have left Red Hat altogether. Only one new person has joined the team. Páral pointed out in the announcement that this was not the result of a layoff or intentional reduction of the quality team; he said that the moves were "truly decisions of our colleagues", some of whom opted to move to AI-focused roles or other jobs within Red Hat.

Red Hat is hiring for at least one more quality engineer, but with only five people currently doing the work of ten, the team is looking to shed some tasks with "a poor price/benefit ratio". Páral said that the team is looking to make permanent changes to the way it operates, rather than making temporary adjustments and waiting for Red Hat to hire enough people to keep things as-is:

We decided to look at this change as an opportunity to re-examine what's really important to Fedora right now, and aim to use our resources as wisely as possible for the future. We're not sure that spending a lot of time on manual testing of relatively less-widely-used functionality is the best thing the RH team could be doing, even if we had more people to continue doing it. We will be hiring in future [...] but we want to be able to consider the best possible way to spend all of the paid team's time going forward, and perhaps spend it on other things that will be more valuable in the end, like automation or supporting individual teams.

There is also the fact that Fedora has continued to add more and more to the testing pile over time; it's rarer for things to be removed. Páral said that some of the changes being proposed could have been done a long time ago, "we just lacked the right impetus".

Proposals

The announcement lists nine changes to Fedora's release criteria that the quality team would like to make, but only five have full proposals so far. These will need approval by the Fedora Engineering Steering Committee (FESCo) or the Fedora Council; the quality team cannot simply change the release criteria by itself. However, Páral points out that since the team does not have enough people to cover everything, if a proposal is rejected by FESCo or the council "an alternative solution might be needed".

Most of the proposals involve removing criteria from the list of release blockers; that does not mean that a feature or deliverable will no longer be included, it simply changes the priority of any bugs related to the features or deliverables to be non-blocking. If there is a bug found in one of the de-prioritized areas during testing for a new Fedora release, it will not delay the release.

One of the full proposals so far is dropping optical-media boot as a release-blocking feature. This should not be too controversial; most laptops and desktops do not even include a CD-ROM or DVD-ROM drive of any sort these days. Even if a system has optical media, it should also be capable of booting from a USB drive.

The team also wants to drop dual-booting from Intel-based Apple hardware as a release blocker. Again, this seems like a feature that is of limited relevance now that Apple has phased out Intel systems in favor of its own Arm-based silicon and is dropping support for those systems with newer releases of macOS. At any rate, the testing team lacks the hardware to test Intel dual-boot; in October last year, Páral had to put out a call for someone with an Intel-based Mac to test dual-booting Fedora with macOS because the quality team had no such hardware to test with. Igor Jagec answered the call then, but it is easy to see why the quality team is unwilling to block releases for hardware it has to go scrounging for.

Another proposal seeks to pare down the list of applications that are release-blocking for the Workstation edition. Right now the criteria says that, for Workstation on x86_64, "all applications installed by default which can be launched from the Activities menu" must have basic functionality or it's a release-blocking bug. Páral makes the case that this criteria has "multiple long-standing issues" such as a lack of agreement on what "basic functionality" actually means and that leads to problems during blocker-review meetings. That is compounded by the fact that bugs in these applications tend to be found very close to the final release, and that automating testing of those applications is particularly difficult and the time spent could be better used on other work.

The quality team would also like to limit the release-blocking status of BIOS systems to specific and simple-to-test scenarios. The team's justification for this is that it considers BIOS-only systems to be a rarity at this point, and available UEFI hardware that has a compatibility-support module (CSM) for BIOS is getting harder to find.

That does not mean Fedora users with older BIOS-based hardware would be entirely out in the cold. For example, it would still be a release blocker if Fedora will not install on BIOS-based systems "which use the default automatic partitioning layout to a single empty SATA or NVMe drive". But bugs that affect more complex partitioning layouts or alternative storage types would no longer be considered release blocking. Bugs that impact upgrading Fedora systems with a BIOS would also be considered blockers, and Fedora's cloud images will require BIOS support because Xen on AWS EC2 still uses BIOS boot.

Right now, the criteria for Fedora 43 include "all applications installed by default which can be launched from the Activities menu" for Fedora Workstation on x86_64. The testing team has proposed dropping that criteria and limiting release-blocking applications to a list of 11 basic applications, such as the default web browser, file manager, image viewer, and system settings. Culling the list, Páral said, may reduce the quality of less-critical applications, but it would "reduce the likelihood of long blocker arguments, pre-release crunches and stress".

In the interest of having fewer arguments about blocker bugs, the team also wants to be "stricter about changes between Beta and Final" unless the changes were agreed on ahead of time. That seems likely to be approved by FESCo, depending on the details in the final proposal.

Just one Arm

There is also a proposal to focus on only one popular 64-bit Arm device, such as the Raspberry Pi 4, as a release-blocker. Currently, there are three separate lists of allegedly supported Arm hardware for Fedora that have developed over time; a supported platforms page that has not been updated since 2020, a second supported platforms page with different hardware, and the reference platforms for the Fedora IoT edition. In the Arm proposal, Páral notes that slimming down the list to one device would not only reduce the burden of testing, "it would also clarify that confusion and remove the burden of reconciling and updating all these lists". Being able to run Fedora in a virtual machine on Arm would still be a requirement for release, however, which should cover cloud use cases.

There are two other Arm-related items that may be more controversial when the full proposals are published: dropping desktops on 64-bit Arm as release-blocking criteria and asking the Fedora Council whether Fedora IoT "still makes sense" as an edition. Presumably, the question is whether IoT should be demoted to a Fedora spin, which would not be release-blocking by default. It became an edition in 2020, with the Fedora 33 release; LWN took a look at Fedora IoT at the time.

That is a fair question; the IoT mailing list and discussion category on Fedora's Discourse forum show few signs of life. There are no posts under the IoT category from June 24 through July 24, and only four emails sent to the mailing list during the same period. Three of the emails are automated meeting reminders, and the other is a notification from Páral of the proposal to limit release-blocking Arm hardware.

I emailed Peter Robinson, who drove the original effort to promote Fedora IoT to edition status while he was employed by Red Hat, to get his thoughts on its possible demotion. He wondered why the quality team would be driving an effort to change IoT's status; it is the upstream of some of Red Hat's "edge" products, and he said that one might think that other Red Hat employees would be more involved with the project. Ultimately, though, he said it might be good for IoT to be demoted because it might allow "a reset of expectations" and experimentation with different technologies. For example, he said he did not think that OSTree or bootc were the right choices for IoT. (LWN covered bootc last June.)

He also questioned the decision to reduce focus on aarch64 hardware, given that the number of Arm laptops and single-board computers (SBCs) is on the rise. That is true, but it is also something of a chicken-and-egg situation; Fedora does not seem to have a large user base on Arm, but to get there it needs better Arm support.

The Fedora Workstation and KDE Plasma Desktop editions have installation images for Arm systems; one might expect that if KDE or GNOME are failing tests on Arm then a Fedora release is not ready to be shipped. It is unclear, however, how many people actually use Fedora as a desktop operating system on Arm hardware. In the the quality team meeting on July 21 (meeting log), Páral said that testing desktops on Arm was one of the slowest things to do in QA, and there were too few users to justify it "according to some data estimates" from former Fedora Project Leader (FPL) Matthew Miller.

Adam Williamson, who leads Fedora's quality team, said during the meeting that he thought there was some discussion that "Fedora is just so slow for desktop purposes on SBCs that most people wind up using the distros with out-of-tree patches". Neal Gompa said that KDE on Raspberry Pi 4 was "reasonably performant", but Fedora Workstation "chugs" with the larger problem being that "GTK4 just flat out crashes on RPi systems since moving to Vulkan".

Kashyap Chamarthy asked Gompa if there were "a non-trivial portion of users who care about aarch64 desktops". Gompa said that Fedora KDE on Arm use was growing "in large part because we're actively promoting it that way", but the challenge was that the current release-blocking criteria for Arm hardware seemed like a mess right now.

Sorting out that mess, as well as other work that has traditionally fallen to the quality team, will require more hands than the team has available at the moment. The team is asking for more involvement from other teams—such as the Workstation working group, KDE special-interest group, and cloud working group—when it comes to manual testing, debugging issues, and validating fixes.

It is also looking for new maintainers for the Fedora Packager Dashboard and its data parser, oraculum. Currently the team is trying to find maintainers within Red Hat to take these over but will send out an announcement to the larger community if that is unsuccessful. If both of those efforts fail, the dashboard may go away entirely.

Reactions

So far, the quality scope-reduction announcement has not generated as much response as one might expect. This is probably, at least in part, because it's peak vacation season in much of the Northern Hemisphere; many of the usual participants in Fedora discussions may be enjoying time away from their computers. It may also be because of the venue chosen for the discussion; Páral asked that feedback be restricted to the Discourse forum when he sent the announcement to the fedora-devel mailing list. Fedora is a project with a long history, and many of its participants still prefer to discuss things via email rather than web forums.

Most of the responses that have come in so far are largely in favor of the proposals. Pat Kelly thought the team was taking a good approach to the situation, and that it would serve the community well in the long run.

"P G" agreed that the proposals seemed reasonable, but wondered about the impact of the "resourcing situation" on the initiative from Fedora's strategy 2028 plan to add accessibility features in its editions to the release-blocking criteria. Accessibility features are part of test criteria now, but they do not block a release.

Sumantro Mukherjee was supposed to drive an effort to help prepare Fedora's editions so that the project could make accessibility tests must-pass for release. Unfortunately, he is one of the people who have recently left the team to work on AI-related things. It is unclear who might fill that gap. Williamson replied to P G that he hoped that it would not impact the initiative, but it could do so if the team is unable to find ways to automate the needed testing.

Fedora, in its early days, was not known for good quality control. It has done much to remedy that over the years, and it generally enjoys a good reputation as a solid, user-friendly distribution today. One hopes that the project can find ways to maintain its quality despite having fewer people being paid to focus on the task.