User: Password:
Subscribe / Log in / New account


FreeBSD and release engineering

February 1, 2012

This article was contributed by Nathan Willis

On January 16, John Kozubik posted to the FreeBSD-hackers list and expressed his disappointment in some of the recent trends in the project. Namely, an increasingly-slow release cycle, too many overlapping "production" releases, and an estrangement between the core developers and end users when it comes to support issues like bug fixes. The list has since debated Kozubik's assessment of the situation in a heroically long thread, but while the majority agree that FreeBSD would benefit from refocusing its energies and polishing its processes, it has not yet developed a concrete plan of action.

Too many series, not enough releases

Kozubik himself is not a FreeBSD developer; he manages enterprise production environments (including that run almost 1,000 FreeBSD machines. As he explained in his initial message, he was disappointed to hear that the next point release of the OS, 8.3, has been pushed back to March 2012 — more than a year after 8.2. Such a long stretch between minor releases makes maintaining a stable server farm difficult, but paradoxically FreeBSD has also complicated the lives of its end users by simultaneously pushing out multiple "production releases" with different major numbers — at present including versions 7.4, 8.2, and 9.0.

Once a new "major" version is in development, he said, important patches to the previous major version start to languish, as developers lose interest in donating their time to the old code and to less-than-exciting maintenance tasks. As a result, enterprise users face a difficult choice: either go without new features and fixes in the "old" production release, or risk the instability of the "new" production release. He worries that FreeBSD may be "becoming an operating system by, and for, FreeBSD developers" — to the ultimate alienation of users.

Kozubik then outlined five underlying causes that contribute to the alienation problem, and proposes three fixes. First, there is a "widening gap of understanding" between the developers and end users, caused by developers frequently jumping between bleeding-edge snapshots of the code, rather than running the releases tagged as stable for production use. The disconnect makes discussing maintenance issues difficult. Second, maintaining multiple "production releases" simultaneously dilutes developer focus, which keeps the releases from ever truly maturing. Third, the simultaneous production releases drive away potential paid investment from enterprise customers, because they view each FreeBSD release with uncertainty.

Fourth, the slow pace of minor releases means that important fixes often sit unreleased long after they have been verified by maintainers. That not only hurts end users (by depriving them of regression fixes and security updates), but downstream projects as well. Finally, when the slow pace of minor releases is coupled with the multiple-major-releases problem, new code and fixes increasingly get bumped from the "old production" release to the "new production release," solely because the new release is what developers are interested in working on. This traps enterprise users in "the same bad choice again: make major investments in testing and people and processes every two years, or just limp along with old, buggy drivers and no support." Kozubik called this "the culture of 'that's fixed in CURRENT' or 'we built those changes into (insert next major release)'."

He suggested that the project take three steps to ameliorate the trouble. First, intentionally consider the processes and costs incurred by large FreeBSD deployments. "Think about the logistics of major upgrades and the difficulty of running snapshot releases, etc. Remember - if it's not fixed in the production release, it's NOT FIXED. Serious (large) end users have very little use for CURRENT." Second, the project should focus on just one production release at a time, and commit to a definite support schedule (Kozubik suggests five years as a production release, followed by five years as a "legacy" release). Third, in concert with the predictable major release schedule, the project should commit to doing smaller, more frequent minor updates, around three times per year.

Dissecting the problem

A majority of the people who joined in the discussion agreed that something like what Kozubik proposed would be beneficial. Opinion varied, however, on what the underlying causes are, and whether or not they can be fixed in the short term. Rich Healey observed that there are very few paid FreeBSD developers, and while volunteers have little motivation to undertake the unexciting maintenance tasks, the paid developers that exist are often contracted to implement specific new features.

Warner Losh said that no corporate sponsor has been pushing the project to keep the release process on schedule since the demise of Walnut Creek. Julian Elischer responded by suggesting that interested high-volume customers and downstream vendors could pool their resources and pay a developer to work specifically on release management — a suggestion that was met with approval.

FreeBSD release engineer Robert Watson described the dilemma as a release engineering problem — with several causes — that he and the other release team members could address. Historically, he said, the FreeBSD base and ports collection were on a single, tightly-coupled release cycle — which often resulted in ports getting very little attention. In the early days, he added, the project used CVS version control, which made branching very expensive. Last but not least, the project has come to rely on a single "head release engineer" to steer all of the release schedules, which results in bottlenecks and slipping release dates. The CVS problem has been addressed by moving to Subversion, he said, and there is growing support for de-coupling the base and ports releases, but the project still needs to fix the single-release-engineer issue. Watson suggested mentoring-in new release managers from the developer community, each of whom would take responsibility for one minor release.

On the other hand, Igor Mozolevsky argued that the FreeBSD Problem Report (PR) patch system is broken, because it effectively requires end users to undertake a nagging campaign to even get a patch examined by a developer. That drives away users and is clearly sub-optimal for the project. Not everyone agreed with that assessment, although Matthew Fuller cited an example of a manpage fix collecting dust for three years.

Adam Vande More speculated that a bounty system would motivate quicker PR merges — but Matt Olander from iXsystems pointed out that he has set one up already. Watson again suggested changes that the project could make to its existing processes, starting with replacing the GNATS issue-tracker that no one seems to like, and adopting a more formal policy for trawling PRs and triaging bug reports.


Not everyone was on board with Kozubik's reading of the situation, however. Ivan Voras disagreed completely, saying that "the situation is actually quite good," and that "nobody would mind" if there were no more stable releases at all. "The 'releases' are for many people simply a periodical annoyance due to freezes," he said. Kozubik replied that "I could not have illustrated my point better, RE: FreeBSD becoming an OS by, and for, FreeBSD developers," and explained that most businesses will simply not use software that is not "officially" released.

Andriy Gapon defended the concept of a "by the developers, for the developers" mindset as the equivalent to being a community project. Projects that exist "for the users," he argued, tend to exist for commercial purposes, and be backed by profit-driven corporations. FreeBSD is more akin to the Debian project, he said, which is very much a "for the developers" effort, but still quite successful.

Adrian Chadd took issue with the question being raised at all, saying:

If you care this much to comment on it, please consider caring enough to step up and assist. Or, pay a company like iXsystems for FreeBSD support and get them to do this for you. Otherwise you're just re-iterating the same stuff I'm sure all the developers know but are just out of manpower/time/money/resources to do anything about.

But Doug Barton quickly responded "let's do away with the whole, 'If you step up to help, your help will be accepted with open arms' myth. That's only true if the project leadership agrees with your goals." The FreeBSD project needs to do some serious thinking about things like the role of "committer" and the meaning of "release," he said, including the difference between major and minor releases, all of which are terms with no formal definition. "We also need to take a step back and ask if throwing more person-hours at the problem is the right solution, or if redesigning the system to better meet the needs of the users *as a first step* is the right answer."

Engineering release-engineering

Barton suggested defining minor "point releases" as updates only to the FreeBSD base, not the ports collection or documentation. "The other thing I think has been missing (as several have pointed out in this thread already) is any sort of planning for what should be in the next release," he added. The current release schedule is a hold-over from the trouble-filled days the project experienced in the FreeBSD 5.x era, he said, but "the pendulum has swung *way* too far in the wrong direction, such that we are now afraid to put *any* kind of plan in place for fear that it will cause the release schedule to slip."

Watson concurred, adding that ""here's been an over-swing caused by the diagnosis 'it's like herding cats' into 'cats can't be herded, so why try?'" — and asking list members if they could come up with a tentative release schedule.

The debate continues, without a consensus yet on a release schedule. For his part, Kozubik favored immediately declaring FreeBSD 7.z end-of-life, marking 8.x as "legacy," and tagging 9.x as the only production release. Not everyone agrees with Kozubik's five-production-years-plus-five-legacy-years timetable; though there is support for a five-year total support life, and most developers seem to think that making minor releases every four months is doable.

Such a schedule would be roughly in line with what the commercial Linux distributors offer, and as Freddie Cash observed, the maintenance of a production release alongside a legacy release is similar to the process used by Debian. The big question remains whether the project will be able to hash out all of the details in time to commit to a concrete release schedule for 9.x itself — or whether the new release process will have to wait for FreeBSD 10.0.

Comments (13 posted)

Brief items

Distribution quotes of the week

You know what I love?

When reboots don’t go horribly wrong.

-- Seth Vidal

Fellow Anti-mergers, I understand the pain and anguish that systemd has caused you personally, and your families. Your hopes and dreams crushed, by someone with all the charm of a cheese grater across the knuckles. Your remaining life tainted by this putrescent subhuman who forced himself upon your internet.

Despite the privation we have all endured, please find strength to stop this nightmarish ravaging of our once-pure filesystems. For if he’s not stopped now, what hope for /usr/sbin vs /usr/bin?

-- Rusty Russell

At that time, Debian developers were busy breaking unstable as much as they could, as it’s tradition on the weeks following a major release...
-- Jordi Mallach

Comments (none posted)

Debian 6.0.4 released

The Debian Project has released the fourth update of Debian 6.0 (squeeze). "Please note that this update does not constitute a new version of Debian 6.0 but only updates some of the packages included. There is no need to throw away 6.0 CDs or DVDs but only to update via an up-to-date Debian mirror after an installation, to cause any out of date packages to be updated."

Comments (none posted)

Red Hat extends RHEL5 and 6 support

Red Hat has sent out a press release stating, in pure marketing style, that the support period for versions 5 and 6 of Red Hat Enterprise Linux has been extended to ten years. "Many of our customers have come to realize that standardizing on Red Hat Enterprise Linux improves efficiency and helps lower costs. With a ten-year life cycle, customers now have additional choices when planning their Red Hat Enterprise Linux deployment and overall IT strategy. We are pleased that customers are looking far into the future with Red Hat."

Comments (53 posted)

Red Hat Enterprise Linux 4 - 30 day End Of Life Notice

Red Hat Enterprise Linux 4 will reach its end-of-life on February 29, 2012. "For customers who are unable to migrate off Red Hat Enterprise Linux 4 before its end-of-life date and require software maintenance and/or technical support, Red Hat offers an optional support extension called the Extended Life-cycle Support (ELS) Add-On Subscription. The ELS Subscription provides up to three additional years of limited Software Maintenance (Production 3 Phase) for Red Hat Enterprise Linux 4 with unlimited technical support, critical Security Advisories (RHSAs) and selected Urgent Priority Bug Advisories (RHBAs). For more information, contact your Red Hat sales representative or channel partner."

Full Story (comments: none)

Distribution News

Ubuntu family

UDS Sponsorship Now Open

The next Ubuntu Developer Summit (UDS) will take place in Oakland, California May 7-11, 2012. "UDS is the most important event in the Ubuntu calendar. It is where we get together to discuss, design, and plan the next version of Ubuntu; in this case the Ubuntu 12.10 release." Canonical will sponsor the hotel and accommodation for a limited number of people to attend UDS. Applications for sponsorship must be submitted by February 22.

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

Comments (none posted)

The case for the /usr merge

Lennart Poettering has announced the posting of a summary of the motivations for merging several root-level directories into /usr. "A unified filesystem layout (as it results from the /usr merge) is more compatible with UNIX than Linux’ traditional split of /bin vs. /usr/bin. Unixes differ in where individual tools are installed, their locations in many cases are not defined at all and differ in the various Linux distributions. The /usr merge removes this difference in its entirety, and provides full compatibility with the locations of tools of any Unix via the symlink from /bin to /usr/bin."

Comments (267 posted)

Page editor: Rebecca Sobol
Next page: Development>>

Copyright © 2012, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds