openSUSE rethinks its release process

After a release cycle dogged by problems and delays, the openSUSE distribution announced the inevitable: its upcoming 12.2 release would be delayed. However, what might have been less expected was that release manager Stephan "Coolo" Kulow used the occasion to call for a rethinking of openSUSE's release process. The result was an ongoing discussion that, although short on definite conclusions, is something of a case study of the challenges faced by large and growing projects.

Kulow made his statement against a background of steady growth for openSUSE project. According to Kulow, in the last year, the number of unique developers has increased from 182 to 222, or 22%, growth that is welcome, but strains long-established procedures and practices. As community manager Jos Poortvliet noted, in the last five months of the release schedule, "Pretty much every milestone of openSUSE 12.2 has been delayed or even canceled." The first two milestones were over a week late, and the first beta two weeks late. Even worse, the fourth milestone and first release candidate were canceled altogether. A revised release date has yet to be set, but Kulow is now aiming for mid-September, almost two months later than the original date.

Both Kulow and Poortvliet point to package integration, especially of GCC 4.7 and Automake, as the major problem. In other words, the packages work well enough themselves, but not enough work is being done to ensure that they interact with other packages properly. Talking about Factory, openSUSE's testing repository, Kulow stated that:

The result, Poortvliet said, is a "broken-window problem" in which existing problems create an atmosphere that encourages further neglect, creating a "downward spiral."

In announcing the delay in the release, Kulow suggested three improvements were needed: more attention to integration work, more work in staging projects (repositories for early development work on specialized subsystems) rather than Factory, and a change in release schedules — either abandoning them, or releasing only once a year. Kulow also declared his intention to have "a no-tolerance strategy about packages breaking other packages" and suggested: "It's time we realize delaying milestones is not a solution. Instead, let's use the delay of 12.2 as a reason to challenge our current development model and look at new ways."

Technologies, policies, and people

Kulow's announcement generated a long thread on the Factory mailing list, with many potential problems raised, along with suggestions for possible solutions. Much of the discussion followed his lead and approached the state of openSUSE as a technical or organizational problem.

For example, Brian K. White asked the basic question: "How does anyone find out that a change in their package somehow breaks some other package?" The answer that emerged was that Factory needed to be rebuilt after uploading packages to it.

However, since constant rebuilds would strain the project's hardware resources, Adrian Schröter suggested only having staging projects for packages that were a key part of the distribution's structure. Another way to ease the demands on the servers was proposed by Guido Berhoerster:

Alternatively, todd rme suggested that Tumbleweed, the rolling release repository for openSUSE, could become the source of official releases rather than Factory — moving openSUSE to rolling releases entirely. Or perhaps, as M. Edward Borasky speculated, a combination of methods might be developed, with core packages using cascading repositories and less vital packages using a repository set up for rolling releases.

As the discussion expanded to include these possibilities, Greg Kroah-Hartman pointed out that openSUSE's build system is so slow as to be impractical, and that packages were sometimes requiring unnecessary rebuilds. "We need a way to be able to mark a checkin as needing to cause this rebuild . . . or for a way for the build system to determine this on its own," Kroah-Hartman said. In response, Borasky mentioned Gentoo's USE flags, which can be toggled to include or exclude packages from a build, as a model for a solution.

Other sub-threads focused on policy and personnel. "Maybe we need more public reminders of out of date and out of sync packages and people from [the] community would step up," Alin M. Elena suggested, adding that this system had worked in the preparation of GCC 4.7. Similarly, Marcus Meissner pointed out that the technical suggestions would still require people to do the integration work.

More generally, Sascha Peillicke pinpointed the problem as the lack of a "welcoming and open-minded spirit of doing community work" as well as a lack of direct contact among contributors. Robert Schweikert went even further, critiquing openSUSE's package maintenance model. According to Schweikert, so many people are listed as maintainers for some development projects that "in the end no one really feels responsible."

This discussion is still continuing, with more sub-threads than mentioned here and with any conclusions still to be made. But, if still short of the "soul searching" that openSUSE community manager Jos Poortvliet mentioned in a media alert about Kulow's announcement, the discussion provides a detailed illustration of some of the difficulties faced by a growing project. It also highlights a number of plausible partial solutions to openSUSE's specific problems.

What's next

Asked to summarize the situation, Kulow replied:

On his blog, Jos Poortvliet was more specific. In trying to summarize the discussion to date, he suggested that either a longer release cycle, or simply a determination to release only when ready, was probable. He also foresees an increased role for the Tumbleweed repository and the creation of staging projects in one form or the other. He also urged fewer maintainers, each with greater responsibility for their packages as well as changes in procedure and a need for new hardware to implement any improved development model.

Whatever reforms are finally undertaken, openSUSE may receive criticism for disorganization, or for not seeing the problems before the release schedule collapsed. Still, it takes some courage and honesty to admit that the present system is no longer working and to abandon the release schedule. But, in the long run, the implementation of reforms in the development model seems likely to be of more benefit to openSUSE than keeping to its original schedule.