In an effort to beef up the quality assurance (QA) process for Fedora, the project has launched a new QA-focused team called Proven Testers. The Proven Testers are a select group of QA volunteers who are responsible for stress-testing and approving updates to what Fedora calls its "critical path" packages. The project hopes this new approach will increase the quality of Fedora releases, but also hopes it will attract more core developers and packagers to the QA process itself.
The Fedora QA process is already a systematic process, consisting of organized teams for bug triage, organized test plans and test days, and drafting release criteria. Prior to the Proven Testers subproject, however, testing release milestones and updates was not a strictly-organized affair. The Fedora test list acts as a coordination point for individual volunteers, who test both packages hitting the development repository Rawhide and packages hitting the updates-testing repository for maintained releases.
Following the critical path
Adam Miller proposed the Proven Testers program in March, with the goal of providing increased attention to packages that affect the critical path — essentially, the core system functionality (installation, boot, mounting filesystems, graphics, login, network, fetching package updates, etc) without which the system is unusable.
Proven Testers are asked to perform a full system update from
updates-testing at least once per day, updating individual packages more
frequently when there is an urgent need. They then test for basic
stability, and provide feedback against the update. Major bugs reports are
to be reported in the project's Bugzilla, and positive or negative "karma"
votes are given using Bodhi. Updates must receive positive votes from members of the Proven Testers group in order to be promoted through the system for release.
The initial set of Proven Testers numbered just twelve, drawn from the existing QA group members, to attempt a trial run. Subsequently, the QA project has opened up the Proven Testers group to others, although Miller says it is intended for "members who have a 'proven' track record of having good testing habits, file meaningful feedback (bugs or karma to Bodhi), being familiar with the over all Fedora QA processes/guidelines, etc."
This does not exclude newcomers, he explained; rather, new testers who wish to join are paired with a more experienced mentor to guide them through the process. Given the critical nature of critical path updates, he added, there is an informal process that could be used to remove a Proven Tester in the case that one consistently gives positive feedback to broken updates, but it has never been used, and it is hoped never to be needed.
As of now, there are 33 Proven Testers, responsible for testing 579 critical path packages. Following this past spring's trial run, Fedora 13 (released in May) is the first to enjoy the support of the Proven Testers program since day one.
Testing testing processes
Testing processes among the other community-driven Linux distributions
vary considerably in terms of formality. Ubuntu's Testing Team performs organized
daily smoke tests and pre-release ISO testing, sponsors testing days aimed at particular features and applications, and maintains distribution- and application-test cases. It, too, maintains a repository for proposed updates to stable releases, and has a public stable release update (SRU) verification team. The SRU verification team, however, does not have to sign off on updates for packages to be approved for release, and there is not a formal membership application-and-approval-process.
OpenSUSE has a volunteer Core Testing Team responsible for ensuring basic functionality in development releases, and maintains separate sub-teams for specific core areas such as KDE, GNOME, installation, wireless, and LAMP servers. OpenSUSE has recently excised and relaunched its public wiki, which makes finding current documentation of the team's processes a challenge, but according to the mailing list the emphasis is placed on ISO testing as a part of the regular release process. A separate Maintenance Team also exists for packaging updates for maintained releases, although it does not appear to encompass testing. A similar function, though, seems to be provided by Novell's QA team for the company's SUSE Linux Enterprise products.
Debian's testing process, of course, is different entirely, as is its
release process. Individual package updates progress through the
experimental and then unstable distributions based on the amount of time
each has been available, the build status for all of the supported
architectures, and the number of release-critical bugs
being fewer or equal to the number for the prior update. Historically,
stable releases of the distribution are
made at the discretion of the release manager, and updates are generally limited to security fixes.
Fedora's use of voting through Bodhi already distinguishes it from the
other distributions, where bug reports are the determining factor in an updates acceptance. Bodhi solves the problem that the mere absence of a negative (a bug) does not prove an update is ready. However, the positive (a vote in Bodhi) clearly makes "false positives" a potential pitfall in addition to the "false negative" of an undiscovered bug.
The Proven Testers project is an effort to correct for this, at least for the most critical packages. But in addition, Miller hopes it will attract more individual developers and packagers to participate in Fedora's QA team. By and large, the historical QA and testing communities have seen more participation from non-developers. Hopefully, by bringing developers, packagers, bug reporters, and testers into a closer release process overall, the stability of the distribution will be improved, and the community can engage in better overall communication.
The future of distribution testing
Miller is pleased with the Proven Testers project thus far, although he notes it is not perfect. "A perfect example of that is how PackageKit worked like a champ if you were in Gnome or XFCE but had an issue where it would no longer alert on updates if you were running KDE. So again, not a perfect process but we do try." In the long run, though, he is more excited about other developments in the Fedora QA process, such as the AutoQA automated test system.
Several of the other distributions are developing automated test tools. While certainly helpful, they will never supplant human testers as the last line of defense. All large free software projects are concerned about QA and testing. No individual distribution can hope to amass the sheer volume of testers attracted by the Linux kernel itself, so the more systematic approach being taken by Fedora is a welcome development — perhaps attracting additional participants, but more practically, allowing them to focus their energies towards a measurable test process. If it works, it may be a valuable case study for other large projects for whom release stability is a major concern, from the various desktop environments, to development frameworks, to X.org.
Comments (2 posted)
The Ubuntu team has announced the release of Ubuntu 10.04.1 LTS, the first
maintenance update to Ubuntu's 10.04 LTS release. "This release
includes updated server, desktop, and alternate installation CDs for the
i386 and amd64 architectures.
Full Story (comments: none)
Debian is awesome. You know that, I know that, we all know that, and 17 years ago Debian entered into our lives. Before I joined the Ubuntu project I was an avid Debian user, and still am. Ubuntu owes a huge amount of thanks to Debian and it's global family of contributors for all of their incredible work.
-- Jono Bacon
I can't help but note that the slips have become more frequent as we
started to actually *have* release criteria to test against. We
didn't slip nearly as much when we weren't testing it. (Whether that's
a good or bad thing is left as an exercise for the reader.)
-- Bill Nottingham
(on Fedora release
Comments (none posted)
New Fedora project leader Jared Smith has announced that the release of Fedora 14 has been pushed back a week, from October 26 to November 2.
"Today we held our readiness meeting for the Alpha release of Fedora
14. As you may know, this is a meeting with representatives from the
Development, Release Engineering, and Quality Assurance teams. In
these meetings, we evaluate the list of blocker bugs and give a "go"
or "no go" signal on the state of the Fedora release.
You can read the minutes of the meeting here, but in short the
decision was made that the release has not passed its release
" Click below for the full announcement.
Full Story (comments: 4)
Máirín Duffy has a summary
of the August 13 meeting of the Fedora Board. This was a public meeting on
IRC. Topics include the schedule, a vision statement for Fedora, code
maturity for inclusion, meeting protocol, Fedora Board composition,
updates, and other board business.
Comments (none posted)
Canonical has announced
the release of uTouch 1.0, a multitouch/gesture stack which will be shipped with the upcoming 10.10 release. "With Ubuntu 10.10 (the Maverick Meerkat), users and developers will have an end-to-end touch-screen framework from the kernel all the way through to applications. Our multi-touch team has worked closely with the Linux kernel and X.org communities to improve drivers, add support for missing features, and participate in the touch advances being made in open source world. To complete the stack, weve created an open source gesture recognition engine and defined a gesture API that provides a means for applications to obtain and use gesture events from the uTouch gesture engine.
Comments (29 posted)
Mark Shuttleworth introduces
mascot for Ubuntu 11.04, "Natty Narwhal". "The Narwhal, as an Arctic (and somewhat endangered) animal, is a fitting reminder of the fact that we have only one spaceship that can host all of humanity (trust me, a Soyuz won't do for the long haul to Alpha Centauri). And Ubuntu is all about bringing the generosity of all contributors in this functional commons of code to the widest possible audience, it's about treating one another with respect, and it's about being aware of the complexity and diversity of the ecosystems which feed us, clothe us and keep us healthy. Being a natty narwhal, of course, means we have some obligation to put our best foot forward. First impressions count, lasting impressions count more, so let's make both and make them favourable.
Comments (15 posted)
Newsletters and articles of interest
Comments (none posted)
Illumos founder Garrett D'Amore looks toward the future of that project
given the changes at Oracle. "So, by their actions here, Oracle may be forcing Illumos to 'fork', which was always a prospect, even if not one I cherished. But with the backing of the innovators I know who are with us, I think we have a chance to actually be the premiere foundation for SunOS derived technology. Oracle may be investing more into Solaris, but if the best and brightest have left for greener pastures and are contributing to Illumos, then I think we'll have the 'best' investments in the base. Following Oracle's lead when the brightest minds have already left looks less and less desirable by the moment.
Comments (1 posted)
Alasdair Lumsden has posted a
document claimed to be a leaked Oracle engineering memo
Solaris will be handled within the company. It suggests that OpenSolaris
as a community project (such as it was) is no more. "We will
distribute updates to approved CDDL or other open source-licensed code
following full releases of our enterprise Solaris
operating system. In this manner, new technology innovations will
show up in our releases before anywhere else. We will no longer
distribute source code for the entirety of the Solaris operating
system in real-time while it is developed, on a nightly basis.
Comments (86 posted)
Margarita Manterola notes
that today, August 16, is Debian's birthday. You can show your
appreciation for Debian at thank.debian.net
. (Thanks to Paul Wise)
Comments (2 posted)
Linux Journal covers
a study by Christian Perrier on where Debian developers come from. "The land that gave the world Linus [Torvalds] also gives the world the most Debian developers per million population. Ranked number one last year as well, Finland is home to 3.92 active developers per one million souls. In second place is Switzerland with 2.83 per million. New Zealand holds a very respectable third place with 2.51 per million. The United Kingdom beats out the United States with their 1.03 developers per million to .53. In last place is the Ukraine, China, and India. Making their first showing this year is Ecuador with one new developer or .07 developers per million people. Sweden, who ranked third last year, fell to sixth this year. Ireland has gained three new developers bringing their total to nine which allows them to hold ninth place, up from 13.
Comments (1 posted)
Page editor: Rebecca Sobol
Next page: Development>>