Can Fedora Ship on Time?

If a Fedora release has been on schedule throughout the entire development process, it hasn't been in recent memory. Fedora Project Leader Jared Smith's announcement that Fedora 14 would suffer a one week slip came as no surprise, but sparked a long discussion on the fedora-devel list about the project's long history of schedule slippage.

Mike McGrath compiled an extensive list of Fedora slips, but it's by no means complete. A quick trip to Google shows that every release since Fedora Core 4 has suffered a slip of at least a week. Reasons for the slippage vary by release, but the blocker for Fedora 14 Alpha was the inability to choose the basic vesa driver when booting and after installation.

The specific cause may not be that important, however. John Poelstra points out that every release has different blockers:

One example may be large changes that derail other system components. Bruno Wolff III cites Python and systemd as major causes for delays, and suggests that "big changes" land a month earlier for easier testing.

Some distributions have different schedules for different components. openSUSE, for example, has different freeze dates for the toolchain, base system, and remainder of the system. In theory this means that the pieces with the greatest impact on the rest of the system are stabilized first and allow for testing of other components without breakage going on "underneath" them. In practice, Adam Williamson says that it's rare for the blocker bugs to reside in the core components in Fedora.

But Fedora does have critical path packages that could be frozen earlier in the schedule. This would include components like Anaconda, GNU Bash, GCC, Grub, the kernel, X.org, RPM, Yum, and many others.

Smith offered three suggestions starting with documenting the reasons for failure, as Poelstra has already been doing. Poelstra has a detailed list of things that have worked well and those that have not with the Fedora 14 release so far. Some of the problems Poelstra has identified include the Koji outage on July 10 and 11, and a need for redundancy on the Release Engineering team. Smith called for more work to document failures (and presumably successes) to try to address them in future releases.

Secondly, Smith called on the Fedora Engineering Steering Committee (FESCo) to "take a more active role in tracking the major changes that land in the distribution and judging the impact that they might have before freezes." Smith noted that Python and systemd weren't the blockers for F14 Alpha, but "had an impact on our ability to build test composes, and also our ability to thoroughly test the RCs" before evaluating the release.

Lastly, Smith asked for contributors to help the Quality Assurance (QA) team with automated testing, as too much is being done manually — and Fedora does not have the resources for extensive manual testing. Fedora has been boosting its testing efforts lately with the "proven testers," who are specifically focusing on the critical path packages. Fedora 13 was the first cycle that had the proven testers at the helm, and F13 slipped as well. However, one of the goals for the team is to recruit more people for testing, which may help in the long run.

Other posters suggested extending the release cycle, some by just a month, others suggesting a nine to 12 month cycle, but this was not met with a great deal of enthusiasm. Will Woods argued against it, saying that no matter how long the cycle was, it wouldn't "stop people trying to cram fixes/changes in at the last minute." As with Smith and others, Woods suggested "better tools and processes" rather than longer development cycles.

How long are the freezes? Users outside the development process may be surprised at how short they are. Peter Jones noted that it depends a great deal on where one starts counting. From development start to feature freeze, the freeze was 43 days for F12, 53 days for F13, and 63 days for F14 — including holidays that fell within the development cycle. From development start to alpha freeze, F13 had 84 days, F12 only 76 days, and F14 just 70 days. In the end, Jones says that "from computing these numbers I think the best lesson is that our schedules have been so completely volatile that it's very difficult to claim they support any reasonable conclusions." Given the frequent changes to the development process for Fedora, there's something to be said for that conclusion.

In the end, are the slips really a major problem for Fedora? Williamson points out that the releases are still "reasonably fixed" with less than a two weeks slippage on the last three releases of Fedora. Others have chimed in to say the process isn't broken, and that slipping to fix issues "is a feature not a bug."

While that might sound glib, there's some validity to it as well. Fedora long suffered from a bad reputation for pushing out releases with serious, sometimes show-stopping bugs. The latest releases have been, if not perfect, of much higher quality. What of the arguments that Fedora is less dependable for "a project manager looking at using a Linux OS in my project" or that it's losing users due to slips? Fedora's relatively short lifecycle after being released makes it an unlikely candidate for many projects even if it were on time. At any rate, Fedora's release cycle is fairly dependable — it can be expected that the release will slip by a few weeks past the scheduled date, but not much beyond that. The suggestion that Fedora is "losing users" over a few weeks of slippage is also unsupported.

Nothing final has been put into place yet, but it does look like the Fedora developers are taking the slips seriously. Over the years, the Fedora community has been aggressive about improving its development processes and addressing perceived issues with the Fedora distribution. If the community remains concerned and focused on avoiding slips, it just might be that Fedora 15 arrives as scheduled. But if it's a choice of quality vs. meeting the schedule, the slips aren't so bad.