Slipping or skipping Fedora 18

Fedora release dates have slipped in the past; the QA and release teams meet regularly for a Go/No-Go vote as the schedule draws to a close to assess whether outstanding blockers mandate additional development or QA time. But the Fedora 18 release cycle has seen more delays than usual — in large part because it incorporates an overhaul of the Anaconda installation tool, which is behind schedule. The inherent riskiness of rewriting such a critical component has some project members asking whether the feature-approval process itself needs refactoring, and others asking whether Fedora 18 should be pushed back significantly to ensure that the changes arrive with stability.

Anaconda runs loose

At the heart of the dilemma is the new UI feature for Anaconda. Although the name of the feature suggests an interface revamp, the work actually encompasses a host of changes, some of which touch on other low-level parts of the system, such as the removal of logical volume manager (LVM) storage as the default, and separating system-updating functionality into a separate tool.

Regardless of what one thinks about the specific changes, the refactored Anaconda is quite a bit behind schedule — and, just as importantly, it is still undergoing major development at a time in the development cycle when features should be implemented, and bugfixes should be the focus. As a result, the release schedule has been delayed five times; originally set for October 2, the beta release is now expected November 13. Anaconda issues dominate the list of blockers, though of course there are other issues.

Tom Lane expressed his frustration over the issue on the fedora-devel list on October 30:

Lane's concerns were echoed by others, including the notion that the Anaconda development process was being allowed to run roughshod over rules and deadlines that were enforced on other packages. But the crux of the matter remained whether or not the new Anaconda could reasonably be expected to be finished. If not, Lane and the others argued, then surely shipping the Fedora 17 version of Anaconda was better than delaying the new release by still more months.

Adam Williamson argued that reverting to the old Anaconda would be more time consuming, since many of the new Anaconda's problems are fallout from changes to the Dracut initramfs infrastructure. "Oldui wasn't fixed for that, so if [we] were to try and switch back to oldui at this point, we'd have to go through the whole process of adjusting the code for the changes to dracut again, quite apart from any other issues."

Freeze or cut bait

For some, the one-week-delay announcements arriving weekly from the Go/No-Go meetings are as much a part of the problem as the readiness of the code. Fedora would be better to acknowledge the scope of the work remaining in Anaconda by pushing back the release schedule by one or two months, the argument goes. After all, the six month development cycle is not set in stone, and introducing major changes would be an acceptable reason to move it. Meanwhile, the weekly delay decisions consume time on their own, holding up other development teams — as well as generating many more "Fedora 18 delayed" news items. Lane contended that the project "can slip a month (or two) honestly, or you can fritter it away a week at a time, and ensure that as much of that time is unproductive as possible. There is not a third option."

David Airlie went even further, asking if the project should skip the Fedora 18 release entirely. That idea gained little traction, but there was support from many for pushing the release schedule back substantially. Tim Lauridsen called that idea preferable to reverting to the Fedora 17 Anaconda, since reverting would mean revisiting the new Anaconda integration problems during the Fedora 19 cycle.

But the length of the release cycle itself could also be contributing to the new Anaconda's continuing problems. Anaconda developer Vratislav Podzimek argued that the six month period on the calendar is not enough for a major rewrite, given how early the feature freeze takes place.

To that point, Toshio Kuratomi replied that, theoretically, each release cycle includes nine months of development time, because it begins when Rawhide is branched for the N+2 release, several months before N is final. However, he said, "there's a mismatch between this practice and the theory of how we develop Fedora." In practice, most developers do set up their development schedules around the not-quite-six-month release cycle. Fedora, he concluded, can either decide to bless this approach and work to make it more effective, or adopt a longer cycle.

There are some who feel that a longer cycle would benefit the project, but the idea has its downsides. For one, Fedora depends on several large upstream projects (e.g., GNOME) that use a six-month release cycle. Changing it would come at the cost of synchronization. For another, the reality is that most feature changes fit into the existing six-month schedule, and events like the Anaconda rewrite are the exception. Lengthening the release cycle for everyone would likely encourage other teams to work slower, and it would not prevent other large-scale features from running late — they would simply be later when they do run late. As Scott Schmit put it, if Fedora moves to a nine-month release cycle, "then people just get more ambitious with their features and then what? Slow the release cycle down more?"

Features and contingencies

But Schmit also contended that the root cause of the Anaconda problem was not that the Anaconda rewrite was big and disruptive, but that the feature came with no contingency plan, and the distribution did not adequately prepare for the risks in advance. There have been other "critical features" introduced in previous releases (GNOME 3 and systemd, for example), he said, about which Fedora leadership knew from the outset that a problem would force the entire release schedule to slip. In those instances, the teams worked hard to ensure that the new feature was ready and tested. In contrast, Anaconda's new UI "seems to have slipped through the cracks."

Others concurred; Williamson listed several high-impact effects of the new UI work that were not discussed when the feature was approved for inclusion:

Some, like Ralf Corsepius, suggested that the Fedora Engineering Steering Committee (FESCo) could alleviate similar woes in the future by making fallback strategies and contingencies part of the feature approval process. Williamson suggested extending Fedora's existing "critical path packages" concept to include "critical path features" as one such approach.

But not everyone sees a systemic problem in this case. To some, the Anaconda team simply should have started work on the rewrite a full release cycle before it was scheduled to be merged in — either working on it in parallel during the Fedora 17 cycle, or proposing it as a Fedora 19 feature. Jóhann B. Guðmundsson pointed out a conference presentation in which the Anaconda team speculated that the work would take two release cycles to complete. Considering that estimate from the team, it could hardly be a surprise that fitting the work into a single cycle proved to be a strain.

Of course, given limitless funds and an inexhaustible pool of developers, all sorts of distribution-building headaches would disappear. But the reality is that the Anaconda team has plenty on its plate already; certainly no one in the discussion accused team members of sitting idly by. They are as strapped for resources as anyone else. In any case, what-might-have-been is a moot point, or at least secondary in importance to preventing a recurrence in the future. The general consensus seems to be in favor of strengthening the feature approval process, particularly where high-impact changes are concerned. The list discussed the possibility of moving to a rolling-release model as well, but that suggestion never got beyond the speculative stage — for every proposed benefit, there was a downside, and the majority of both were hypothetical.

At least it is clear how working harder at assessing new critical features' viability and risks will improve things in future releases; whether that amounts to a formal process, or if the wake-up call of the new Anaconda experience will suffice, remains to be decided. As for what to do this cycle, so far no change in the process has been announced. The release schedule was pushed back again at the November 1 Go/No-Go meeting. The next meeting is scheduled for Thursday, November 8. As the schedule stands now, Fedora 18 is expected to be released December 11 — or about one month from now.