After a release cycle dogged by problems and delays, the openSUSE
distribution announced the inevitable: its upcoming 12.2 release would be
delayed. However, what might have been less expected was that release
manager Stephan "Coolo" Kulow used the occasion to call
for a rethinking of openSUSE's release process. The result was an
ongoing discussion that, although short on definite conclusions, is
something of a case study of the challenges faced by large and growing projects.
Kulow made his statement against a background of steady growth for openSUSE project. According to Kulow, in the last year, the number of unique developers has increased from 182 to 222, or 22%, growth that is welcome, but strains long-established procedures and practices. As community manager Jos Poortvliet noted, in the last five months of the release schedule, "Pretty much every milestone of openSUSE 12.2 has been delayed or even canceled." The first two milestones were over a week late, and the first beta two weeks late. Even worse, the fourth milestone and first release candidate were canceled altogether. A revised release date has yet to be set, but Kulow is now aiming for mid-September, almost two months later than the original date.
Both Kulow and Poortvliet point to package integration, especially of GCC 4.7 and Automake, as the major problem. In other words, the packages work well enough themselves, but not enough work is being done to ensure that they interact with other packages properly. Talking about Factory, openSUSE's testing repository, Kulow stated that:
I don't remember a time during 12.2 development when we had less than 100 'red' [problem] packages. And we have packages that fail for almost five months without anyone picking up a fix. Or packages that have unsubmitted changes in their devel project for six months without anyone caring to submit it.
The result, Poortvliet said, is a "broken-window problem" in which existing problems create an atmosphere that encourages further neglect, creating a "downward spiral."
In announcing the delay in the release, Kulow suggested three improvements
were needed: more attention to integration work, more work in staging
projects (repositories for early development work on specialized
subsystems) rather than Factory, and a change in release schedules —
either abandoning them, or releasing only once a year. Kulow also declared
his intention to have "a no-tolerance strategy about packages
breaking other packages" and suggested: "It's time we realize delaying milestones is not a solution. Instead, let's use the delay of 12.2 as a reason to challenge our current development model and look at new ways."
Technologies, policies, and people
Kulow's announcement generated a long thread on the Factory mailing list,
with many potential problems raised, along with suggestions for possible solutions. Much of the discussion followed his lead and approached the state of openSUSE as a technical or organizational problem.
For example, Brian K. White asked
the basic question: "How does anyone find out that a change in their
package somehow breaks some other package?" The answer that emerged
was that Factory needed to be rebuilt after uploading packages to it.
However, since constant rebuilds would strain the project's hardware
resources, Adrian Schröter suggested
only having staging projects for packages that were a key part of the
distribution's structure. Another way to ease the demands on the servers
was proposed by Guido Berhoerster:
How about a multi-tiered Factory to address this and have all changes go into an openSUSE:Factory:Staging project and only migrate packages to openSUSE:Factory after a certain delay when they do not break others or introduce serious regressions. Factory would then ideally always build and thus make it easier to cut releases at fixed dates.
Alternatively, todd rme suggested
that Tumbleweed, the rolling release repository for openSUSE, could become
the source of official releases rather than Factory — moving openSUSE to
rolling releases entirely.
Or perhaps, as M. Edward Borasky speculated,
a combination of methods might be developed, with core packages using
cascading repositories and less vital packages using a repository set up for rolling releases.
As the discussion expanded to include these possibilities, Greg Kroah-Hartman pointed out that openSUSE's build system is so slow as to be impractical, and that packages were sometimes requiring unnecessary rebuilds. "We need a way to be able to mark a checkin as needing to cause this rebuild . . . or for a way for the build system to determine this on its own," Kroah-Hartman said. In response, Borasky mentioned Gentoo's USE flags, which can be toggled to include or exclude packages from a build, as a model for a solution.
Other sub-threads focused on policy and personnel. "Maybe we need
more public reminders of out of date and out of sync packages and people
from [the] community would step up," Alin M. Elena suggested,
adding that this system had worked in the preparation of GCC
4.7. Similarly, Marcus
Meissner pointed out that the technical suggestions would still require
people to do the integration work.
More generally, Sascha Peillicke pinpointed the problem as the lack of a "welcoming and open-minded spirit of doing community work" as well as a lack of direct contact among contributors. Robert Schweikert went even further, critiquing openSUSE's package maintenance model. According to Schweikert, so many people are listed as maintainers for some development projects that "in the end no one really feels responsible."
This discussion is still continuing, with more sub-threads than mentioned
here and with any conclusions still to be made. But, if still short of the
"soul searching" that openSUSE community manager Jos
Poortvliet mentioned in a media alert about Kulow's announcement, the
discussion provides a detailed illustration of some of the difficulties
faced by a growing project. It also highlights a number of plausible partial solutions to openSUSE's specific problems.
Asked to summarize the situation, Kulow replied:
It's not as dramatic as it may appear, but we do have problems - and we do need to realize that we do have core contributors we can't easily replace in case they have no time. So possibly we need to come up with a more flexible development process or we stick to the development process, but relax the release schedule. This is something to be discussed in the project - and so far the discussion is constructive.
On his blog, Jos Poortvliet was more specific. In trying to summarize the discussion to date, he suggested that either a
longer release cycle, or simply a determination to release only when ready, was
probable. He also foresees an increased role for the Tumbleweed repository and
the creation of staging projects in one form or the other. He also urged fewer
maintainers, each with greater responsibility for their packages as well as
changes in procedure and a need for new hardware to implement any improved
Whatever reforms are finally undertaken, openSUSE may receive criticism for
disorganization, or for not seeing the problems before the release schedule
collapsed. Still, it takes some courage and honesty to admit that the
present system is no longer working and to abandon the release
schedule. But, in the long run, the implementation of reforms in the
development model seems likely to be of more benefit to openSUSE than
keeping to its original schedule.
Comments (5 posted)
So you come back from two weeks of vacation, and you find that there are
about 2000 non-spam mails in your inbox. And you hate yourself because you
have to work through them. But then you notice that 20% of those are a
single fedora-devel flamewar you don't care for, and you happily mark them
all and delete them, and you are happy and relieved, because you worked
through 400 mails within a second. So, should you be sad that fedora-devel
exists and spams you? Or be happy about the fact that it provides you with
a reliable source for such a feeling of quick accomplishment?
-- Lennart Poettering
We apologize for any inconvenience while we’re fixing the community :-) .
-- Charles-H. Schulz
on the formation of the Mandriva Foundation
Comments (none posted)
The Fedora ARM team has announced the availability of the Fedora 17 GA
release for ARM. "The GA release includes prebuilt images for
Versatile Express (QEMU), Trimslice, Beagleboard xM, Pandaboard, Kirkwood
Plugs, Highbank and iMX based hardware platforms.
Full Story (comments: 7)
openSUSE release manager Stephan "Coolo" Kulow has put out a call for discussion of a new development model for the distribution at least partly in reaction to the delay in various milestones for 12.2.
"openSUSE has grown. We have many interdependent packages in Factory. The
problems are usually not in the packages touched, so the package updates
work. What's often missing though is the work to fix the other packages
that rely on the updated package. We need to do a better job making sure
bugs caused by updates of "random packages" generate a working system.
Very fortunately we have an increasing number of contributors that
update versions or fix bugs in packages, but lately, the end result has
been getting worse, not better. And IMO it's because we can’t keep up in
the current model.
" Jos Poortvliet has added his thoughts
Full Story (comments: 63)
The runoff election for the remaining Fedora Board seat has ended.
Nick Bebout is the lucky winner.
Full Story (comments: none)
Newsletters and articles of interest
Comments (none posted)
The H examines
the "Rascal Micro," a tiny ARM board running OpenEmbedded Linux that is designed to be hardware compatible with Arduino shields, while offering a more complete computing environment. "The Rascal Micro offers on-board programming in Python via a built-in, web-based editor. The Pytronics Python library serves for implementing the actual sensor queries and for communicating with the shields. Data is visualised on the web interface via jQuery and jQplot.
" The list of diminutive, hobbyist-style Linux PCs continues to grow.
Comments (7 posted)
Page editor: Rebecca Sobol
Next page: Development>>