The trouble with FreeBSD

He got involved with open source in the 1990s; he actually started with Linux, but somebody told him that "Linux is rubbish" and he should use FreeBSD instead. So he bought an iMac computer and got FreeBSD running on it; the project then punished him by giving him commit access. It is a great project, but it does have some problems relating to three factors in particular: FreeBSD is big, it's old, and its leadership can be slow to act.

How big is FreeBSD? The project's Subversion repository is currently about 3.1GB in size; a checked-out tree takes about 600MB. It consists of 71,100 files, about 32 million lines of code. It takes 20-30 minutes to build the whole thing, which is a big improvement from the old days, when it could take several hours.

The project is old, having gotten its start in November 1993; it was based on the 1992 386BSD release which, in turn, was based on much older code. So there is a lot of history. FreeBSD developers have only recently discovered automated testing, he said, but, for such testing to work, the software has to be structured properly, and FreeBSD isn't. Testing the kernel in an automated manner is especially hard. They do have some tests, though, many of which were borrowed from NetBSD, but they are focused mainly on user-space code.

With regard to slowness to act: like many community groups, FreeBSD is consensus-driven. The leadership is a nine-person elected team, but this team does not want to be seen as picking sides in any particular debate. The core team also does not want to be in control of the project's architectural direction. Originally, the core team was made up of the four founders of the project. The project as a whole followed the usual BSD model, giving commit access to good contributors, then appointing the best of those to the core team. Around 2000, though, the project switched to the current elected model. It works, but the consensus-driven model makes decisions slow.

Tools

For an example, consider the problems around tools and processes. A development project's source code is its core asset, and it must be well looked after. So the project needs a source-code management system. Initially, FreeBSD used CVS, which was a reasonable choice at the time, but the project stayed with CVS until 2008. Why didn't FreeBSD move off of CVS faster? It worked well enough for basic release management, but the real problem was that there was no consensus around what should replace it. BitKeeper was considered, but that was a proprietary system. Git and Mercurial came around later. The project considered Git, but didn't think it could handle a repository of that size, so it moved on.

The discussion went on for a long time. All communities have their perennial arguments that come back over and over; for FreeBSD, this was one of them. Developers did trial conversions to other systems, but consensus was elusive. When the decision was finally made, the project chose Subversion. It was not an optimal solution, but it did at least allow for changesets involving more than one file at a time, which was the biggest pain point with CVS at the time. In 2008, after eight years of arguments, Peter Wemm simply imposed a switch to Subversion; there has been relatively little whining since.

In comparison, Python started on CVS, moved to Subversion in 2004, then to Mercurial in 2009, and is now moving over to GitHub. The Python community's PEP process was helpful enabling the changes to happen. If, in the Python community, you want to make a change, there is a process to follow that forces you to think about it and make good arguments. Once the PEP is approved, you are clear to proceed. FreeBSD has no such mechanism.

In the Django community, developers realized that they were already using GitHub for everything except Django itself, so they simply pushed the change through and were done with it.

The source-code management debate was an example of the core team not wanting to take sides on an issue. The usual response in this kind of situation is to tell the developer to go and do a proof of concept implementation; that usually shuts them up, he said. But the only way to make a decision like this more quickly is to take a side. Doing so might have caused more "project angst", but perhaps that wasn't avoidable. Meanwhile, the project is talking about moving to GitHub. It has a repository there now, but nobody is paying attention to it, so if a developer submits a pull request, nobody will see it. With GitHub, he said, you have to either be there for real or not at all.

Rock-star developers

Another example has to do with community interactions, and the "not fun" task of dealing with bad behavior. A small community can usually talk things out fairly easily, but FreeBSD has hundreds of committers. FreeBSD leaders thus have to do a certain amount of human resources work. Developers tend to look down on the human resources department in their company, but it does have its value; it's how you don't end up in court. FreeBSD's disagreements are unlikely to end up in court, but you still need somebody to step in and straighten things out occasionally.

FreeBSD does maintain a committer's guide, which has two sections on interactions with other developers. It has advice like "do not impugn the intentions of somebody you disagree with", "accept correction", "ask for help", and more. There are also rules, including "respect other committers", "discuss significant changes before committing them", and "do not fight in public with other committers". There are some paragraphs on actions the core team can take to deal with violations; these include suspending commit access or removing a violator from the project's mailing lists. It could, he said, be the beginning of a code of conduct for the project.

"One of the more spectacular incidents" in this area involves the famous Matthew Dillon, who joined the project in 1994. He quickly showed himself to be a talented developer, with a special knack for dealing with esoteric virtual-memory bugs. But he was not a team player, preferring to be in charge. So, when he started working in any particular area, he would tend to take charge at the expense of anybody else working on that code.

Like most early multiprocessor-capable kernels, FreeBSD had a big kernel lock; Dillon wanted to get rid of it for the FreeBSD 5 release. John Baldwin had been already looking at breaking up the big kernel lock; Dillon joined the effort and things went well for a while. Then the two developers ran into a disagreement over how critical sections should be handled. Baldwin argued for simply disabling interrupts, but Dillon said that was too inefficient; he committed a different change involving a flag in the low-level interrupt handler. It was faster on the x86 architecture, but x86 is not the only architecture supported by FreeBSD.

This change led to a number of big arguments, with Dillon eventually backing out the change. A month-long acrimonious debate followed; eventually Dillon's change went in, but there were a lot of other, similar incidents. It was a classic case of a "rock-star ninja 10x developer" not getting along with the rest of the project.

The core team responded with a set of new rules about when commits could be made and the penalties for engaging in "commit wars". Eventually the problem came to a head when Dillon committed a change to what was supposed to be the FreeBSD 5 stable branch. After more argument, he backed it out with a commit that read: "Bow to the whining masses..."; it also threw in some unrelated cleanups "just for the hell of it". After that, the core team took his commit access away, and he eventually left the project.

Leadership is hard, Rice said. If the project had had an established code of conduct at that time, it might have been in a better position to deal with this rock-star developer problem.

Burning trash piles

After Dillon left, things calmed down for a while — until Gamergate ("a burning trash pile made of other burning trash piles") came along. One of the figures in Gamergate was Randi Harper, a former FreeBSD developer. She did a lot of much-appreciated work on the installer, then got a job at Amazon and, as often happens with Amazon employees, she stopped doing community work, though she still hung around on the mailing lists and such.

Harper, a combative personality, jumped into the Gamergate fray with gusto after being attacked. One day she decided to block the Gamergate "trash humans" out with a Perl script; it worked, causing them to lose their minds, Rice said, at which point they started attacking FreeBSD. The core team started getting emails asking it to deal with the problem, saying that Harper was dragging FreeBSD into the whole mess. Initially, they responded by saying it wasn't related to the project and there wasn't much they could do about it.

Things got worse when a FreeBSD developer joined the Gamergate side and started attacking Harper on Twitter; she attacked back. Things moved to a private FreeBSD IRC channel, which witnessed some heated discussions; the attacker then leaked a log of the conversation to Breitbart. That was a violation of the project's policies, so, at that point, the core team had no choice but to get involved.

Harper wanted three things from the core team: an official code of conduct for the project, a statement that the team was investigating this behavior and would not tolerate harassment, and for that person to be quietly removed from the project. The core team responded that the code of conduct was already in progress and that sanctioning the other developer was hard due to the fact that the bulk of the activity happened outside of the project's communication channels. Policing your own area is relatively easy; it's harder when things happen on sites like Twitter or Reddit. In any case, that became a moot point when the developer resigned.

The FreeBSD project does now have a code of conduct; it is, Rice said, a "patched-up rush job". Project members still honestly thought that "meritocracy" meant "inclusiveness" initially. They have since learned otherwise, and the code is being reworked. Meanwhile, Harper wrote a blog post about how badly the whole thing was handled; there were some valid criticisms in there, Rice said, with some things to learn from.

The real trouble with FreeBSD

So how can the problems with FreeBSD be fixed? Some of those problems are a function of the project's size, which is something that can't really be fixed. There are things that could be done, though. FreeBSD is still shipping sendmail, for example, which may no longer be necessary; taking it out would make FreeBSD a little smaller. FreeBSD is still a big, monolithic distribution, but there is some talk of splitting it up into packages. This is a place where interested people could help. There are also opportunities in automation, testing, and elsewhere.

Other problems are tied to the project's age, but that is hard to fix too. FreeBSD will not be getting any younger. It can be hard to change baked-in attitudes. For example, there is resistance to moving to GitHub because people in the core team and beyond are attached to running their own infrastructure. Such things can be worked around, but an attitude shift will be needed.

With regard to leadership, that could be fixable but it, too, would require an attitude shift in the core team. The team does not need to be an architectural leader, it needs to be a "scaffolding leader". It just needs to ensure that "the things the project needs to do a good job" are available. The core team should focus on how FreeBSD is made, rather than what it should be. Its job should be reducing friction for developers.

But that points to the real trouble with FreeBSD: it is made and led by volunteers. There is nobody whose job is purely to make FreeBSD more awesome. There are plenty of people tasked with making it more awesome for their employers, but that work doesn't always help in the long term. Those employers get a faster build system, but automated testing languishes. Somehow, the project needs more resources to be able to focus on making FreeBSD "a fun and awesome place" because that is how they can ensure that it will be around for a long time to come.

The full video of this talk is available.

[Your editor would like to thank linux.conf.au and the Linux Foundation for assisting with his travel to the event.]