SCALE: Developing a project community

The day before SCALE 11x in Los Angeles was devoted to a series of mini-summits; some centered around a single software project (such as Puppet or Ubuntu), but others were topical—such as the FOSS Mentoring summit. That summit was devoted to the mechanics of working with the human aspects of a free software project, from developers to end users. For example, Nathan Betzen of XBMC discussed the communication tools, from forum packages to external social networking services, and how the project makes use of each. Karsten Wade spoke about infrastructure volunteers (e.g. as system administrators), and how to recruit, train, and foster increased participation. Robyn Bergeron spoke about organizing and managing predictable release cycles in a project that relies on unpaid—and often hard-to-predict—volunteer contributors.

Communication tools

Betzen has been working with XBMC since 2008, and currently serves as community manager. His talk dealt with end-user-facing tools—blogs, forums, and so forth—as distinguished from development-centric tools like source code management or build systems. In large part, he said, this is of interest because XBMC is a user-facing application with a large community of users who are not engaged in software development.

He started off with general advice on "how not to suck" at user community interaction. This advice included sincerity in messaging (communicating why you honestly feel your project is great, rather than crafting an artificial message), restricting your posts to appropriate and relevant information (regardless of how awesome your cat photos may be), and seeking relevant input from the users. On that point, he said that starting an open source project is a public statement that you expect other people will want to work on the product. The earlier you accept that—including the fact that these others will bring different opinions and plans—the better your communication will be.

He then summarized the most common avenues for user communication, and what XBMC has learned about their suitability for different purposes. Web-based discussion forums are the "heart" of your community, he said: where everyone is going to go to talk to and ask questions of the development team, and where they will look for help. He gave no advice on the specific software package used, but said the important thing is that users can register and write their words on your footprint on the Internet. He gave some "best practices" advice, such as trying to shut down as few threads as possible (other than lawbreaking topics and bots). Even if your community becomes known for its off-topic discussions, people are still coming to the site and learning about your software. He also advised making forum rules early (and making them highly-visible), and making as much intra-team communication open as possible, even if that only means posting read-only threads summarizing team meetings.

Blogs are distinctly different from forums as a community tool, Betzen said. If comment threads on individual posts are the only place where discussions occur, the project suffers. First, individual posts come and go (including going off the bottom of the first page rather fast), and blogs allow only site administrators to start a topic of discussion. That prevents community members from speaking what's on their mind, plus it creates a bottleneck. "You want as many people as possible to do the work of making conversations."

He then discussed the relative pros and cons of the current social networking services, starting with "the great beast," Facebook. Facebook is a "not terrible" way to gauge the reach of your project, he said, and is considerably better than log-based web analytics. The reason is that every Facebook account is bound to a real person. Thus, in addition to calculating your own project's reach, you can compare it to the reach of your competitor projects. It is also a useful outreach tool, since it is one of the default places people unfamiliar with your project start a lot of their searches. It works well as a "personality outreach" tool, where you can ask end users to submit ideas and content of their own. For example, he said that XBMC regularly asks users to post photos of their own XBMC systems, and frequently gets submissions for themes, mascots, and other mock-up designs that eventually become reality.

In contrast to Facebook, Betzen said, Twitter is a terrible way to measure reach, since it is dominated by fake accounts and bots. "Half the people on Twitter aren't real people," he said. "You can become a 'Twitter master' with thousands of followers, most of whom don't know who you are and aren't even real." Still, he said, it functions well as a front-line support service, and is useful to visibly communicate with non-users and participate in the larger world.

Reddit can be a good tool, he continued, but it really works best for already-engaged projects. He advised against creating your own "subReddit," since that can be seen as self-aggrandizing, but added that the site can be useful as a persistent news outlet and to promote "causes" and campaigns. For example, he cited the case of a third-party port of XBMC to the original Xbox hardware: when PayPal unilaterally shut off the group's donation account without explanation, a Reddit furor arose that was credited for PayPal's decision to mysteriously reinstate the account (again without explanation).

Finally, he discussed Google+ and YouTube. Google+ is popular among developers but not with users, yet it has a killer feature with the "Hangout" video chat system. This allows people to talk directly to a project team, which is a great way to engage, and it can be used to broadcast live meetings. Similarly, YouTube videos offer some "magic", because there are frequently features or procedures that are clear in a screencast, but terrible to try to read on a "list of steps that was written on a wiki several years ago and hasn't been updated."

Infrastructure

Wade's session also dealt with the practicalities of managing a project, in particular how to organize and maintain infrastructure: the systems administration and support tasks that keep a project functioning smoothly (as opposed to the development of the actual code). He highlighted the importance of treating infrastructure as an avenue of open participation, just like hacking.

Wade's advice had two sides. First, using open tools protects the project (in that owning your data and other "stuff" is critical), and it allows other open projects to see how you do things and learn from them. Second, making infrastructure participation part of the project gives interested administrators a way to pitch in and support the project even if they are not prepared to be developers and (as with coding) it offers a good way for those interested in systems administration to learn and equip themselves with skills that will be valuable later.

He drew examples from his experiences on the Fedora distribution and on the oVirt project. The first piece of advice was to divide project infrastructure into three categories: core-essential, core-nonessential, and non-core. Core-essential pieces are those that are required to actually develop the software, such as source code management, build systems, and testing tools. Core-nonessential pieces are those necessary to keep project participation functioning, but are not related to the project code itself, such as configuration-management tools or mailing list software. Non-core pieces are those concerned with data and metadata, and are generally orthogonal to the system on which they run, such as wikis, documentation, and blogs.

These categories should be treated separately, he said, and doing so benefits both administrators and developers. Developers may prefer to take care of the core-essential pieces themselves, but they do not need to take time setting up Mailman configuration or provisioning database servers. Administrators, too, can divide up tasks in the layers, letting go of control "down to the bare metal" as administrators often want.

Wade instead advised projects to treat the "holy grail" of administration—root access on the server—just like it does the holy grail of development, commit access. By starting new infrastructure volunteers on non-core tasks (even wiki maintenance or web page editing), they can learn the system, gain experience and the trust of other administrators, and work their way toward the core systems. Having a transparent process and meritocractic system are ideals that define open source projects, he said, and they apply to infrastructure administration just as they do to anything else.

A lot of administrators treat the prospect of sharing root access warily, since it is tricky to trust volunteers that they have never met in person. But Wade argued that such trust-building is no different than the trust-building process required of new developers. It is just a mindset many administrators have not adopted. Projects can and should consider sandboxing systems and other tools to guard against accidental catastrophes, but ultimately administrators need to remember that systems administration is not a life-or-death endeavor, and if someone on the team does make a mistake, the project can always roll back the change. Of course, that advice does assume that the project is keeping backups, but that is hardly a new piece of advice.

He also offered several practical ideas for projects looking to get their infrastructure into better shape. The first is to start small and scale up as the project grows. Initially, the bare minimum that the project requires will have to do, even if that is a bargain-rate web hosting plan. The second idea is to find ways for interested sponsors to contribute to the infrastructure as a means of supporting the project. Providing tools or covering all or part of the hosting bills is a good way to let companies visibly support the project, and by allowing multiple companies to contribute, it shows that the project has broad support (as well as giving all vendors equal opportunities). He mentioned that several companies provide servers to the Fedora project; whichever one is responsible for serving up a particular page is indicated on the page itself.

Finally, he provided specific examples of how the oVirt and Fedora projects each split up their infrastructure organization. Fedora's core-essentials include the Koji build system, pkgDB database, Bodhi updater, Yum update system, source code management, testing infrastructure, and "Fedora people" web pages (the latter because they are sometimes used to provide package repositories). Fedora's core-nonessentials include Bugzilla, MediaWiki, WordPress, Planet, elections infrastructure, and mirror management system. The non-core pieces include web hosting and FTP mirroring. The oVirt core-essentials are fewer in number: Git, Gerrit, Jenkins, and Yum. Its core-nonessentials include Puppet, Foreman server manager, and Mailman. Its non-core pieces include MediaWiki and to some extent Mailman (here Wade observed that sometimes the borders between the layers can be fuzzy), and external services like GitHub and Identi.ca.

The audience asked several questions, such as how IRC fits into the mix. Wade replied that IRC is a tricky one, since it is used for communication but is also integral to the development process. The projects tend to use external IRC networks like OFTC and Freenode, but run their own logging and announcement bots (which he considers pieces of project infrastructure). Another audience member suggested that "core" and "essential" might be taken as loaded words, which Wade readily conceded (who wants to hear they have been deemed nonessential, after all?). He said he was open to better suggestions, but that the principle of layers and levels was the main point.

Release cycles

Bergeron discussed "project management" in what she called the classical, business-lingo sense—specifically, addressing the questions of "how do we get this puppy out the door on time?" and "how do we juggle the logistics?" In the business world, there are project management certifications and training classes, she said, but none of them talk about how to apply their tactics to the open source approach, which has peculiarities. First, open source projects are composed of many people (often even critical team members) who are volunteering their time, and cannot be dictated to like a salaried employee. Second, such projects must cope with constant uncertainty in turnover, such as people changing jobs, going away to college, or losing interest for other reasons. Bergeron offered her own advice on this type of project management based on her experiences as Fedora project leader, specifically with regard to managing a development and release cycle.

The first piece of advice is to have a schedule. Shipping on time is the holy grail, she said, but it is important to recognize that shipping on time at the cost of angering and demoralizing the project's members is not worth it—ideally, you want your project to grow continually. How the schedule is determined needs to scale with the project, as does how it is communicated to project members as a whole. For a three-person team, putting it on a single web page may be sufficient; for a large project it is not. It does, however, actually need to be published somewhere. Mentioning it in a blog post is not good enough, as eventually it will scroll off the bottom of the page.

The next piece of advice is to communicate everything. A lot of people assume that open source projects are constantly communicating, but there are things every project can do better. In brief, she said, you cannot remind anyone too frequently about anything. But clarity is the key, so although important things like the schedule need to be communicated broadly to avoid the "well nobody told me about that" problem, they also need to be published in a consistent, predictable manner. In other words, emailing it and posting it at wiki/Schedule one time, then blogging it and posting at wiki/Calendar the next is a bad idea. An audience member asked how to communicate constantly without it becoming annoying; Bergeron replied that that was a risk but it could be mitigated with a good tone and a smile.

The next piece of advice is to manage how much work takes place during the release cycle. Projects need to agree on what features and changes will receive attention, and not simply let everyone do whatever they want when they want to do it. But these plans also need to consider failure a possibility, and plan for incomplete efforts and how to roll back changes. Change-planning can be contentious, she said, but it is critical that the project have the conversations in the community, or else people will leave. Sometimes people hear that warning and think "I want X to leave, he's a damn fool," she said, but you never know who he will take with him or where he might go.

The fourth piece of advice is to learn to live with Murphy (as in Murphy's Law). Problems will happen, she said, but when they do they usually contain learning opportunities. In addition, coming through the problem together can be good for the community, as it fosters a sense of camaraderie. The final piece of advice was to consciously set expectations for each cycle. The best way to do that is to be transparent about the decision-making process and to be communicative. Surprises can scare people off, she said, but invitations to help out are awesome.

In the audience question-and-answer session, one person asked what tools there are for project management. Bergeron recommended TaskJuggler, among other open source options. Another audience member asked how projects can prevent "collisions" where more than one person wants to work on the same thing. Bergeron replied that Fedora has several teams that collaborate, which is one way to tackle the problem, but added that it was rarely a practical concern. Most of the time, there is far more work needing to get done than there are volunteers to do it.

All three sessions provided practical advice, and, judging by the number of audience questions (of which only a fraction were recounted above), that advice was useful for quite a few attendees. Since the free and open source software ecosystem is one that exists to produce software, it can be all too easy to spend too much time thinking about revision control systems and contributor agreements and conclude that those topics cover what it takes to manage a successful project. But as Wade said, what really makes a project "open" is how it functions, not what license it uses; that applies to communication, infrastructure, and scheduling just as it does to development.