LPC: Development model diversity

There is a wide diversity in development models used by free and open source software projects. Allison Randal explored that diversity in her keynote talk at the Linux Plumbers Conference. As one would guess, there is no model that works for all projects and, in fact, projects can learn from the models used by others—which can help as the project evolves over time.

Since "the dawn of time", humans have been searching for a mythical "one true development model", but it does not exist, Randal said. This is not the fault of the computers, but is, instead, a "human problem". There has been much hope over the years that artificial intelligence will help to solve these kinds of problems, but that hasn't happened and may not, though she hopes things don't turn out that way. In the meantime, "if the code doesn't do what you wanted it to do, it's your fault".

Human problems

There are various limitations that projects have that stem from these "human problems". The first is time, because there is never enough of it to implement all the different ideas that project participants might have. Knowledge is another limitation because there is always more to learn: "If you haven't learned anything this week, you aren't trying hard enough", she said.

We also repeat the same mistakes over and over again because projects are limited by the memory of their participants. Not only do individuals repeat their own mistakes, but new project members repeat mistakes made earlier in the lifetime of the project. Distraction causes people to drop out of projects because of real life changes (families, jobs, and so on) or due to finding something new that interests them. In addition, participants may find their patience wearing thin, because they are "tired of answering the same user questions" repeatedly or are weary of the friction between various developers. All of these things get in the way of producing good code and sometimes lead to projects with great ideas that "never make it", she said.

In some ideal world, projects can predict the future perfectly which will allow them to plan for all of the obstacles that they will face. That prescience will also allow projects to anticipate exactly what its users want and to implement it so that it "maps exactly" to those wants. The project documentation would make it obvious why the project is relevant, how to use the project's code, and how to immediately start pitching in to help. Likewise, the testing, deployment, and maintenance phases of the project would proceed without a hitch.

There are several mainstream development models whose names are often heard, Randal said. One is the "waterfall" model, which essentially relies on planning perfectly in advance, whereas the "agile" model requires that you plan perfectly in small steps. Neither is very commonly used in FLOSS projects at least partly because they are "very process heavy". Those and other development models require a set of people in the project whose only focus is to keep the process itself going, which doesn't work very well for volunteer-staffed FLOSS projects. Even in projects where there is a good bit of funding behind them, we "rarely see heavyweight development models", she said.

Influences

There are multiple influences on how FLOSS project choose development models, but perhaps the largest is that the developers are the ones who set the direction for the project. There is typically a large group of geographically distributed participants, some of whom are volunteers, while others may be contributing for various different companies. It's not the companies that set the direction, though, developers choose the way by being interested in "working on this piece or that one", Randal said.

Projects also have various goals with regard to innovation. Some set out to "keep the state of the art moving ahead", while others focus on incremental improvements. In addition, some projects are driven by competition with other projects or proprietary software offerings. The competition for success is often what drives innovation within those projects.

Project forks can also influence the development model. Forks often get a "bad rap", Randal said, but they should really be seen as exploring another evolutionary path. A fork does not necessarily create a "permanent chasm" between the two paths. Aspects that are successful in the fork, both of the feature and development model variety, can and often do merge back to the mainline.

A "derivative" is essentially a different way to look at a fork, when everyone recognizes that there is value in exploring the different ways to achieve a shared "baseline goal", Randal said. Like with a fork, the experience gained by the derivative can make its way back to the original project.

Conflicts and governance

Some projects are "conflict-productive", while others are "conflict-averse". Either can work, but a project needs to recognize which fits it. While many projects work completely in the open, none are really totally public, she said. Discussions or decisions are often made between participants informally in conversations or meetings that are not recorded. The important part is that the majority of those discussions happen in a way that allows the information in them to flow to the public. A mailing list is a good method for doing so, though the traffic can grow to the point where it is too large for people to use, and then some kind of summary is needed.

The "benevolent dictator" model is very popular in FLOSS projects, partly because many projects start with just one person who becomes the obvious choice. A benevolent dictator typically has one of two roles, either a "tie-breaker" or a "trailblazer", she said. A tie-breaker can sit back and listen to the discussion and determine a way forward when there is a conflict among the project participants. The trailblazer, on the other hand, takes an active role in leading the project in the direction they think it should go. Elements of both roles are often seen in benevolent dictators of FLOSS projects, she said.

One challenge that some projects have already faced and others will have to deal with in the future is "what happens when the benevolent dictator retires?" It is a natural process that we already see with project contributors moving on, and project leaders will eventually want to do the same. The tie-breaker role tends to trickle down to other experienced project members even before a benevolent dictator steps down. The trailblazer role may be harder to replace.

Other projects don't have a single leader and will strive to reach decisions by way of consensus. The GNOME project is a good example of this, she said. There are challenges with that model because it can take a longer time to make decisions that way. In projects like that, there are often different people that take on the trailblazing role at different times for various pieces of the project.

Projects with a strict hierarchy are not seen very often for FLOSS projects because those projects tend to be more egalitarian, she said. The "JFDI" (just f* do it) model is fairly common because it allows developers to scratch their itch immediately. If the idea doesn't work out, or others don't like it, it can either be fixed or become an evolutionary branch that just dies off. That model is somewhat challenging as it requires participants to "embrace the chaos" that it can bring.

Any way that there is to structure a project can be found somewhere in the FLOSS universe, and "they all work to one extent or another", Randal said. In addition, the development model is not a good predictor of the success of a project. There are successful projects out there with bad code, bad organization, and/or bad structure, she said.

After a while, projects tend to reach a certain equilibrium where longtime members have set up a model and structure in a way that works. But as new people come on board, those things may need to change. The "model that works for your project today may not work in ten years", Randal said, in fact, it "almost certainly won't".

With that in mind, Randal challenged the audience to think about their projects and to try to determine the best features of their development models. Then she suggested that the audience look around for a project that does things in exactly the opposite way and try to learn how that project makes that work. It is important for projects and people to keep learning, adapting, and evolving; seeing what else is out there is part of that process.

Questions

Randal answered several interesting audience questions after she finished her talk. She noted that she has worked many different models along the way, and that there is none that stands out as "best"—or even necessarily "better"—it depends on the project and participants. There are some combinations that don't work very well together, as well as some models that will drive some people off.

The Linux kernel "is one of the best examples of a benevolent dictator model that works", she said, because Linus Torvalds "does such a good job of embracing chaos". Git is a big part of that because it encourages branching. But that also leads to challenges for those who want to use the kernel. Kernel integration teams have to be large to keep with the chaos: tracking patches and branches, following discussions, and so on.

Randal sees Torvalds as mostly being in the tie-breaker role, as it would be difficult to be a trailblazer for the kernel as a whole because it is moving so quickly. She doesn't foresee any major problems when Torvalds decides to retire as there are a number of kernel hackers that have established themselves as tie-breakers in various areas, and one or more of those folks could step up. Though "some people will be nervous" when that day comes.

Debian and Perl were two examples that she gave of projects that have made significant shifts in their development model over time. Debian has always been consensus-based—and still is—but recently Debian project leader Stefano Zacchiroli has been taking on a trailblazing role, which has been good for the project, she said. Randal took on the benevolent dictator role for the Perl Foundation some time ago in order to move it from that model to a committee-based one. Essentially, there was too much for a benevolent dictator to do, which was creating a bottleneck that might have seriously harmed the project, but a committee-oriented governance solved that problem.

She also pointed to Parrot as a project that was using a bad model for its development. Parrot switched to Git, but uses a centralized scheme with a "painful branching and merging strategy", rather than something more like the Linux kernel strategy. It's not Git's fault, she said, "tools can be misused". While Parrot is getting things done, it is not operating optimally which makes it hard for new developers and in general gets in the way of the development of the project.

[ I would like to thank LWN subscribers for supporting my travel to LPC. ]