LWN.net Weekly Edition for September 5, 2014

Rethinking the OpenStack development process

OpenStack is, by any measure, a successful open-source project—attracting dozens of companies and hundreds of individual contributors, who implement a number of services and core features. But such success has its drawbacks. Recently, the OpenStack project began debating whether or not it needs to make major changes to its development process, to hopefully reduce the backlog of contributions and bugfixes awaiting review. Everyone involved seems to agree that a more formal process is the answer, with some measure of priority-setting for each release cycle, but exactly what form the changes should take is less clear-cut.

The trouble with success

On August 5, Thierry Carrez raised the issue on the openstack-devel list. He noted several interrelated problems, including the fact that more and more contributors are putting time in on code that scratches their own itch (which he called "tactical" contributions), and fewer contributors are left to tackle the "strategic" issues. Those issues include somewhat contentious questions like which outside projects become "blessed" by the Technical Committee and make it into the OpenStack integrated release, and which are granted "incubator" status.

There is, naturally, a big advantage to being blessed and to receiving incubator status; both are perceived as validation of a project's value, and there is clear marketing clout that accompanies being designated part of OpenStack proper. But the desire of so many projects to make it to incubator or integrated status has caused considerable work to pile up for OpenStack's core team, and that endangers the project's ability to put out viable releases.

Carrez asked whether a moratorium on new feature work should be considered, and also whether it was time to tighten up the rules for officially blessing contributed projects.

By and large, the rest of the list subscribers seemed to agree with Carrez's formulation of the problem, though not necessarily on the best approach to fix it. Monty Taylor suggested it was time that the Technical Committee start being more selective about what contributions it accepts—by figuring out what exactly defines OpenStack itself and being prepared to say "thanks, but no thanks" to work that falls outside the definition. As an example, Devananda van der Veen cited the Heat and Ceilometer projects, both of which were blessed into the integrated OpenStack release while only used in a single (major) deployment, and which many in the community still seem to think have significant problems.

But what goes in to the OpenStack core and what stays out is a difficult question. As Sean Dague succinctly put it:

In contrast, Eoghan Glynn argued that cutting back on the number of projects was less important than catching up on the backlog of QA and review work that has accumulated over the past few development cycles. Taking a full release cycle to focus on solving existing quality and performance problems, as well as implementing in-project functional testing, would pay the most dividends in the long run.

Ground control

Michael Still, among others who agreed with Glynn on the value of taking a "time out" cycle, reported that the OpenStack Nova team had recently discussed the same issue, and had proposed a formal process to rate-limit the number of new features being considered for inclusion at any one time—the idea being that the number of feature "slots" (or "runways") available would be fixed, and in line with the number of reviewers who could oversee development during the cycle. The available runways could be divided up between new-feature work and efforts to make up "technical debt," and that division could be adjusted in either direction for a given release cycle.

For each new cycle, developers interested in getting their project accepted would have to define a specification, present it to the Technical Committee for approval, then begin work. But the project team would wait in a holding pattern until the Technical Committee decided that there was a free runway, at which point the project could land its code. The theory, of course, is that the gatekeepers know how many runways they can manage simultaneously, and thus how many new features should land for each release. But an important side effect of the plan would be that (in theory) developers whose project was left in a holding pattern would be free to work on something else for a while, thus maximizing the efficiency of the overall OpenStack contributor pool.

Quite a few list subscribers expressed support for the idea; Stefano Maffulli likened it to the kanban principle in manufacturing. But there were firm dissenters as well. Glynn argued that the plan hinged on developers voluntarily not pursuing their projects when those projects were in the holding pattern, which is not likely to happen often and which OpenStack cannot force them to do. Maffulli, though, countered that OpenStack actually can control quite a few contributors, in particular those who are assigned to the project by its corporate members and sponsors.

Others took issue with the idea at a more fundamental level. Daniel P. Berrange argued that treating stability work as an occasional goal for some release cycles is the wrong approach; rather, "I'd like to see us have a much more explicit push for regular stabilization work during the cycle, to really reinforce the idea that stabilization is an activity that should be taking place continuously." Others were concerned that the fixed number of runways was inflexible or would cause problems of its own. Kyle Mestery asked what would happen if one of the projects given a slot slowed down, either in development or in the review stage, and turned into a bottleneck. In reply, Joe Gordon said that the Technical Committee would just take the slot away and reassign it to someone else.

But the most often-cited objection to the plan was that it introduces a new bureaucratic process. The ultimate goal, many said, was just for the project to better communicate its development priorities for a given release cycle. Adding formal hoops to jump through might clarify the priorities, but it would do so in a decidedly inflexible way. As Berrange expressed that objection from the viewpoint of his own workflow:

Eventually, Russell Bryant suggested a less strict, alternative approach: focus on crafting a priorities list, expect reviewers to be guided by keeping an eye on that list, but do not allow the list block the development of other things.

Hashing out a plan

The plan, as outlined, requires the project to "pick several project priority themes, in the form of use cases, to help us prioritize work", then generate a list of those blueprints based on the themes. The list will be published, and volunteers will be encouraged, but not forced, to work on them. Similarly, reviewers "looking for direction of where to spend there blueprint review time" are expected to use the list, but, notably, if the community and the list diverge, "it means the core team is not aligned properly and should revisit the list of project priorities".

Compared to the fixed-number-of-slots approach, this plan is certainly more flexible. But, of course, it will also need to be put into practice at least once before any final judgment can be made. Perhaps, as some fear, a project as large and rapidly-growing as OpenStack will require a heavier hand to steer in a predictable direction. Several people in the mailing list discussion noted that the Linux kernel also revisited its procedures several times as its contributor base grew substantially, and today uses a rather formal hierarchy of subsystem maintainers.

But perhaps OpenStack's architecture is too different from the kernel's, and OpenStack users view it as more of a toolkit than a tool; in that case, a looser structure might make sense. In either case, as Zane Bitter put it, "this is what success looks like". Having so many contributions that they are difficult to wrangle is certainly a difficult problem to solve, but it is also one that few open-source projects consider a bad sign.

Comments (7 posted)

Haiku discusses a kernel switch

The Haiku project has been developing an open source, binary-compatible replacement for the BeOS operating system for the past thirteen years. The latest release is an alpha that came out in November 2012. The project has recently been discussing its priorities and future after a developer revealed a promising prototype of the BeAPI (i.e. BeOS API) that will run on top of Linux — rather than the Haiku kernel that has long been under development.

For those not familiar with BeOS and Haiku: BeOS was a multimedia-oriented desktop operating system developed in the 1990s by the now defunct company Be Inc. Its features were, at the time, quite forward-looking for desktop systems, including preemptive multitasking and symmetric multiprocessing. All the components of the OS, from the hybrid kernel (based on the work of former Be Inc. employee Travis Geiselbrecht) on up, were tightly integrated, as opposed to desktop Linux distributions that typically pull together disparate pieces of software.

Be Inc. struggled and was unable to obtain much commercial success. BeOS was not able to compete in the marketplace and Be Inc. ended up being acquired by Palm. Since 2001, BeOS enthusiasts have been trying to revive the project through the Haiku initiative. The project is supported by Haiku Inc., a not-for-profit organization which, among other things, holds Haiku-related trademarks and manages funding for the Haiku project.

LWN covered Haiku's last release, Alpha 4.1, almost two years ago. With no stable, complete release after such a long time, a conversation about the project's future was bound to happen; a possibly superior prototype was the perfect catalyst for that kind of discussion.

In an email to the haiku-development mailing list, developer Sia Lang announced "a working prototype BeOS API layer on top of a Linux 3.x kernel [...] after only a couple of months of development". Lang inquired about the status of Haiku to see if the Linux-based project would be worth further development: "If Haiku is close to release, I probably won't bother since it's still a lot of work, but if another seven years is going to pass by, I'll probably go ahead." In addition, Lang sharply criticized Haiku's decision to write its own kernel:

Adrien Destugues countered that Haiku has come a long way since 2007. "There were four "alpha" releases already, and we added support for WiFi, package management, and a modern web browser. This is quite an achievement for our small team of developers." While he did acknowledge the merits of having "the BeAPI available in other systems than Haiku", he argued that, since the Linux kernel targets a variety of use cases and hardware configurations, while Haiku's kernel is focused on tight integration and the desktop, Linux is sub-optimal for Haiku:

Lang replied that configuration was a one-time task: "Besides the "fun" argument, all your points speak *for* Linux (or a BSD for that matter). There would be a configuration tuned for BeOS workloads, and you would never have to do that job again." Lang felt that focusing on "fun" is sub-optimal for Haiku's users: "It's an important motivational factor. But the truth is, end users don't care about that. They want a BeOS clone that works".

Unless it uses a well-established open-source kernel, Lang feared, Haiku may never reach a general release: "Moving the considerable efforts to a kit layer on top of Linux or BSD seems like the only way to get *done* (with done meaning a healthy development and ecosystem with a maintainable and peer-reviewed kernel)".

In answer to Lang's question about a completion date for Haiku, Augustin Cavalier estimated a stable release within the next 14 to 24 months. This led Lang to decide to continue the project. "I'll put up a github repo when the source base is in good shape in case anyone wants to chip in." Cavalier later suggested that he and others might leave the project if Haiku adopted the Linux kernel.

Other developers weighed in with their critiques of Lang's ideas. Haiku treasurer and developer Ryan Leavengood defended the project's desire to make a kernel itself, while acknowledging the accomplishments of Lang's prototype:

Haiku Inc. president Axel Dörfer's views were shared by most other developers on the mailing list: "radical changes like this would change the identity of the project considerably (and therefore its complete community). [...] I don't mind Sia working on this at all, I might even use it when something useful comes out of it. I just wouldn't want to work on it anymore."

The message was not an appeal for developers to "jump ship", Lang said: "I am however asking the Haiku community to consider if the kernel choice made 14 years ago still makes sense. It's painful to leave a huge amount of work behind in the dust, but there's still so much Haiku work that would have a great life on top of a Linux or BSD based BeOS." In the end, the community seems to have decided to keep working on its own kernel — for now, anyway.

The heated discussion about Haiku's kernel reflects the issues of identity and purpose that any open-source project can face. In this case, the clash was largely about whether Haiku should aim for some type of practical impact in the marketplace, or whether the project should cater first and foremost to its developers, who almost entirely work on the project as a hobby in their spare time. There were also technical concerns about whether too much of the tightly integrated BeOS experience would be sacrificed by moving to Linux.

However, if Lang's prototype gathers a team after a source code release on GitHub, it could threaten the future of Haiku by outcompeting the project for developer mindshare. Regardless, it will be interesting to see what impact a "BeOS/Linux" system will have.

Comments (40 posted)

Toward federated services

Federation is not exactly a household word in free-software advocacy. However, a small but growing number of projects are using the term to describe de-centralized web applications based on open protocols. Such an approach, they maintain, is necessary to counter the lock-in created by centralized, proprietary sites such as Facebook and Instagram.

Although such centralized sites are sometimes applauded for running on top of free software, the challenges they create for free-software advocates are far from new. As early as 2008, Tim O'Reilly asked: "What good are free and open source licenses, all based on the act of software distribution, when software is no longer distributed but merely performed on the global network stage?".

Yet the Free Software Foundation has continued its license-based approach, promoting the GNU Affero General Public License (AGPL) as a way to bring free software to cloud services. However, the fact that less than one percent of projects were using the AGPL in 2013 (according to Black Duck Software) indicates that the AGPL is too little-used to be influential, even assuming that it might actually provide an alternative.

The identification and popularization of the concept of federation is usually credited to Evan Prodromou of StatusNet and Identi.ca. In searching for an alternative to centralized web services, Prodromou and projects like GNU MediaGoblin looked back to services like email and IRC, in which anybody can run their own servers, and users can run the software of their choice while still being able to communicate with each other. By contrast, on centralized sites, users can do neither. The closest the users of centralized sites can come to federation is to use tools such as Friendi.ca to post to multiple sites simultaneously; even then, proprietary or constantly changing APIs are frequent problems.

The problem with centralized services

Although centralized sites are often convenient, the trouble is that they contradict the ideals of free software for both developers and end-users. As Kenton Varda of Sandstorm explained:

At best, the only changes will be ones that the centralized site finds convenient in the short-term.

For end-users, centralized services raise both technical and privacy concerns. Technically, centralized services create a series of walled gardens that connect poorly, if at all — if, for instance, you have a Flickr account, you are unable to share photos directly with Instagram. Moreover, as GNU MediaGoblin's fundraising video explains, centralized services also mean centralized control. Users communicate through the service, instead of directly with each other, and become subject to whatever terms of service the service enforces. For example, content may be limited, or, as happened in the early days of Google+, anonymity may be prohibited. Should the service develop technical problems, it becomes unavailable to everyone, while in a federated network, if one node goes down, only those working with that node are affected.

However, perhaps the most important concerns about centralized sites is that their users lose control of their data. According to Prodromou:

In these situations, the idea of federation is seen by its advocates as being just as important as the code quality. The nodes of federated networks, their advocates maintain, should, in theory, be more respectful of user choice and privacy. Moreover, should one federated node prove unduly bureaucratic, users can always migrate to another one to continue receiving service.

Putting the theory into practice

Federation is far from a new concept. Prodromou suggested that the blogging networks that sprung up just after the turn of the millennium could be considered the first deliberate efforts to create federated networks. One or two efforts at federation, such as Diaspora*, have even received widespread media attention. To date, however, no implementation of federation has come anywhere close to achieving the success of centralized sites like Google+ or Twitter.

Part of the problem is that breaking monopolies by direct competition is difficult in any circumstances. Prodromou suggested that "it's a chicken and egg problem" to gain popular acceptance. Federated services need a large user base, but they are unlikely to attract users unless they already have a large base. Even dedicated free-software advocates are as likely to use Twitter as Identi.ca — not because they approve of Twitter's practices or privacy policies, but because their friends are using it.

Another problem is that projects that promote federation are not intended primarily for average users. Their main appeal is to the minority who want to run their own servers and perhaps fork the code. As Varda noted:

For example, MediaGoblin's project lead Chris Webber observed: "we have a lot of people say that they want to deploy their own MediaGoblin server, but they don't have the time" for either the initial configuration or the ongoing maintenance.

Moreover, user-friendly design is difficult to implement when a project is still in rapid development and is struggling to reach its first general release. As a result, efforts at federation have yet to reach their full potential. According to Prodromou:

Efforts to develop common protocols such as pump.io are being made, but attracting early adopters without full functionality is difficult. Although those working on common protocols include representatives from IBM and Mozilla, the majority are from smaller and more obscure projects, many of which are still in the early stages of development.

Yet even without these problems, federation is a a hard sell. The truth is, security and privacy are unappealing to many users. As Prodromou observed, promoting federation goes against the truism that, if developers want success, they need to:

All the same, those who are promoting federation remain optimistic. Because federation encourages code innovation, they hope that federated networks will eventually be able to offer features that centralized networks cannot. "End-users aren't going to be attracted to federated networks just because they're federated," Webber said, "but because there's software available that isn't available elsewhere." Prodromou agreed, suggesting that "the more we include games and rich content and [have] lots of interesting things happening, the more people come back, and the more that they invite their friends."

Even the dryness of federation as a subject may be less of an obstacle than anticipated — after all, Webber noted, if a subject as arcane as the HTML5 standard can attract widespread interest, so, too, might federation some day.

Whether federation can gain acceptance, much less counter the ills of centralized services, is still anybody's guess. For now, however, its potential is becoming as much a concern in some free software projects as the code itself. Federation may, in fact, be the long-delayed answer to the challenges offered from web applications, but whether it can be implemented widely enough to make a difference remains uncertain.

Comments (25 posted)