Distributions

Fedora adds Proven Testers program to QA process

In an effort to beef up the quality assurance (QA) process for Fedora, the project has launched a new QA-focused team called Proven Testers. The Proven Testers are a select group of QA volunteers who are responsible for stress-testing and approving updates to what Fedora calls its "critical path" packages. The project hopes this new approach will increase the quality of Fedora releases, but also hopes it will attract more core developers and packagers to the QA process itself.

The Fedora QA process is already a systematic process, consisting of organized teams for bug triage, organized test plans and test days, and drafting release criteria. Prior to the Proven Testers subproject, however, testing release milestones and updates was not a strictly-organized affair. The Fedora test list acts as a coordination point for individual volunteers, who test both packages hitting the development repository Rawhide and packages hitting the updates-testing repository for maintained releases.

Following the critical path

Adam Miller proposed the Proven Testers program in March, with the goal of providing increased attention to packages that affect the critical path — essentially, the core system functionality (installation, boot, mounting filesystems, graphics, login, network, fetching package updates, etc) without which the system is unusable.

Proven Testers are asked to perform a full system update from updates-testing at least once per day, updating individual packages more frequently when there is an urgent need. They then test for basic stability, and provide feedback against the update. Major bugs reports are to be reported in the project's Bugzilla, and positive or negative "karma" votes are given using Bodhi. Updates must receive positive votes from members of the Proven Testers group in order to be promoted through the system for release.

The initial set of Proven Testers numbered just twelve, drawn from the existing QA group members, to attempt a trial run. Subsequently, the QA project has opened up the Proven Testers group to others, although Miller says it is intended for "members who have a 'proven' track record of having good testing habits, file meaningful feedback (bugs or karma to Bodhi), being familiar with the over all Fedora QA processes/guidelines, etc."

This does not exclude newcomers, he explained; rather, new testers who wish to join are paired with a more experienced mentor to guide them through the process. Given the critical nature of critical path updates, he added, there is an informal process that could be used to remove a Proven Tester in the case that one consistently gives positive feedback to broken updates, but it has never been used, and it is hoped never to be needed.

As of now, there are 33 Proven Testers, responsible for testing 579 critical path packages. Following this past spring's trial run, Fedora 13 (released in May) is the first to enjoy the support of the Proven Testers program since day one.

Testing testing processes

Testing processes among the other community-driven Linux distributions vary considerably in terms of formality. Ubuntu's Testing Team performs organized daily smoke tests and pre-release ISO testing, sponsors testing days aimed at particular features and applications, and maintains distribution- and application-test cases. It, too, maintains a repository for proposed updates to stable releases, and has a public stable release update (SRU) verification team. The SRU verification team, however, does not have to sign off on updates for packages to be approved for release, and there is not a formal membership application-and-approval-process.

OpenSUSE has a volunteer Core Testing Team responsible for ensuring basic functionality in development releases, and maintains separate sub-teams for specific core areas such as KDE, GNOME, installation, wireless, and LAMP servers. OpenSUSE has recently excised and relaunched its public wiki, which makes finding current documentation of the team's processes a challenge, but according to the mailing list the emphasis is placed on ISO testing as a part of the regular release process. A separate Maintenance Team also exists for packaging updates for maintained releases, although it does not appear to encompass testing. A similar function, though, seems to be provided by Novell's QA team for the company's SUSE Linux Enterprise products.

Debian's testing process, of course, is different entirely, as is its release process. Individual package updates progress through the experimental and then unstable distributions based on the amount of time each has been available, the build status for all of the supported architectures, and the number of release-critical bugs being fewer or equal to the number for the prior update. Historically, stable releases of the distribution are made at the discretion of the release manager, and updates are generally limited to security fixes.

Fedora's use of voting through Bodhi already distinguishes it from the other distributions, where bug reports are the determining factor in an updates acceptance. Bodhi solves the problem that the mere absence of a negative (a bug) does not prove an update is ready. However, the positive (a vote in Bodhi) clearly makes "false positives" a potential pitfall in addition to the "false negative" of an undiscovered bug.

The Proven Testers project is an effort to correct for this, at least for the most critical packages. But in addition, Miller hopes it will attract more individual developers and packagers to participate in Fedora's QA team. By and large, the historical QA and testing communities have seen more participation from non-developers. Hopefully, by bringing developers, packagers, bug reporters, and testers into a closer release process overall, the stability of the distribution will be improved, and the community can engage in better overall communication.

The future of distribution testing

Miller is pleased with the Proven Testers project thus far, although he notes it is not perfect. "A perfect example of that is how PackageKit worked like a champ if you were in Gnome or XFCE but had an issue where it would no longer alert on updates if you were running KDE. So again, not a perfect process but we do try." In the long run, though, he is more excited about other developments in the Fedora QA process, such as the AutoQA automated test system.

Several of the other distributions are developing automated test tools. While certainly helpful, they will never supplant human testers as the last line of defense. All large free software projects are concerned about QA and testing. No individual distribution can hope to amass the sheer volume of testers attracted by the Linux kernel itself, so the more systematic approach being taken by Fedora is a welcome development — perhaps attracting additional participants, but more practically, allowing them to focus their energies towards a measurable test process. If it works, it may be a valuable case study for other large projects for whom release stability is a major concern, from the various desktop environments, to development frameworks, to X.org.

Comments (2 posted)

Ubuntu 10.04.1 LTS released

The Ubuntu team has announced the release of Ubuntu 10.04.1 LTS, the first maintenance update to Ubuntu's 10.04 LTS release. "This release includes updated server, desktop, and alternate installation CDs for the i386 and amd64 architectures."

Full Story (comments: none)

Distribution quotes of the week

-- Jono Bacon

-- Bill Nottingham (on Fedora release schedules)

Comments (none posted)

One week slip of Fedora 14 schedule

New Fedora project leader Jared Smith has announced that the release of Fedora 14 has been pushed back a week, from October 26 to November 2. "Today we held our readiness meeting for the Alpha release of Fedora 14. As you may know, this is a meeting with representatives from the Development, Release Engineering, and Quality Assurance teams. In these meetings, we evaluate the list of blocker bugs and give a "go" or "no go" signal on the state of the Fedora release. [...] You can read the minutes of the meeting here, but in short the decision was made that the release has not passed its release criteria." Click below for the full announcement.

Full Story (comments: 4)

Duffy: Fedora Board Meeting 13 August 2010

Máirín Duffy has a summary of the August 13 meeting of the Fedora Board. This was a public meeting on IRC. Topics include the schedule, a vision statement for Fedora, code maturity for inclusion, meeting protocol, Fedora Board composition, updates, and other board business.

Comments (none posted)

Multitouch support for Ubuntu 10.10

Canonical has announced the release of uTouch 1.0, a multitouch/gesture stack which will be shipped with the upcoming 10.10 release. "With Ubuntu 10.10 (the Maverick Meerkat), users and developers will have an end-to-end touch-screen framework — from the kernel all the way through to applications. Our multi-touch team has worked closely with the Linux kernel and X.org communities to improve drivers, add support for missing features, and participate in the touch advances being made in open source world. To complete the stack, we’ve created an open source gesture recognition engine and defined a gesture API that provides a means for applications to obtain and use gesture events from the uTouch gesture engine."

Comments (29 posted)

Shuttleworth: N-imal?

Mark Shuttleworth introduces the mascot for Ubuntu 11.04, "Natty Narwhal". "The Narwhal, as an Arctic (and somewhat endangered) animal, is a fitting reminder of the fact that we have only one spaceship that can host all of humanity (trust me, a Soyuz won't do for the long haul to Alpha Centauri). And Ubuntu is all about bringing the generosity of all contributors in this functional commons of code to the widest possible audience, it's about treating one another with respect, and it's about being aware of the complexity and diversity of the ecosystems which feed us, clothe us and keep us healthy. Being a natty narwhal, of course, means we have some obligation to put our best foot forward. First impressions count, lasting impressions count more, so let's make both and make them favourable."

Comments (15 posted)

Distribution newsletters

• Debian Project News (August 11)
• DistroWatch Weekly, Issue 367 (August 16)
• Fedora Weekly News Issue 238 (August 11)
• openSUSE Weekly News, Issue 136 (August 14)
• Ubuntu Weekly Newsletter, Issue 206 (August 14)

Comments (none posted)

D'Amore: The Hand May Be Forced

Illumos founder Garrett D'Amore looks toward the future of that project given the changes at Oracle. "So, by their actions here, Oracle may be forcing Illumos to 'fork', which was always a prospect, even if not one I cherished. But with the backing of the innovators I know who are with us, I think we have a chance to actually be the premiere foundation for SunOS derived technology. Oracle may be investing more into Solaris, but if the best and brightest have left for greener pastures and are contributing to Illumos, then I think we'll have the 'best' investments in the base. Following Oracle's lead when the brightest minds have already left looks less and less desirable by the moment."

Comments (1 posted)

Lumsden: OpenSolaris canceled

Alasdair Lumsden has posted a document claimed to be a leaked Oracle engineering memo describing how Solaris will be handled within the company. It suggests that OpenSolaris as a community project (such as it was) is no more. "We will distribute updates to approved CDDL or other open source-licensed code following full releases of our enterprise Solaris operating system. In this manner, new technology innovations will show up in our releases before anywhere else. We will no longer distribute source code for the entirety of the Solaris operating system in real-time while it is developed, on a nightly basis."

Comments (86 posted)

Manterola: Debian Appreciation Day

Margarita Manterola notes that today, August 16, is Debian's birthday. You can show your appreciation for Debian at thank.debian.net. (Thanks to Paul Wise)

Comments (2 posted)

Where do Debian Developers Come From? (Linux Journal)

Linux Journal covers a study by Christian Perrier on where Debian developers come from. "The land that gave the world Linus [Torvalds] also gives the world the most Debian developers per million population. Ranked number one last year as well, Finland is home to 3.92 active developers per one million souls. In second place is Switzerland with 2.83 per million. New Zealand holds a very respectable third place with 2.51 per million. The United Kingdom beats out the United States with their 1.03 developers per million to .53. In last place is the Ukraine, China, and India. Making their first showing this year is Ecuador with one new developer or .07 developers per million people. Sweden, who ranked third last year, fell to sixth this year. Ireland has gained three new developers bringing their total to nine which allows them to hold ninth place, up from 13."

Comments (1 posted)