From lab to libre software: how can academic software research become open source?

Academics generate enormous amounts of software, some of which inspires commercial innovations in networking and other areas. But little academic software gets released to the public and even less enters common use. Is some vast "dark matter" being overlooked in the academic community? Would the world benefit from academics turning more of their software into free and open projects?

I asked myself these questions a few months ago when Red Hat, at its opening of a new innovation center in Boston's high-tech Fort Point neighborhood, announced a unique partnership with the goal of tapping academia. Red Hat is joining with Boston-area computer science departments—starting with Boston University—to identify promising software developed in academic projects and to turn it into viable free-software projects. Because all software released by Red Hat is under free licenses, the partnership suggests a new channel by which academic software could find wider use.

This article looks at some successful academic projects that have entered mainstream use—ranging across computer history from the Berkeley Software Distribution (BSD) to Jupyter notebooks—and looks for the factors that might help make that transition work. The projects that I covered suggest the following rules of thumb:

• Academics, working in the reward system of academia, are not likely to carry through the conversion of software from a research project to a viable product of interest to a broad community.
• Funding, usually from government agencies or foundations, is key to the creation of high-quality products that can be widely adopted. This funding can help support the conversion to a useful free-software project.
• It helps a lot if the target users share some of the values and technical knowledge of the project leaders.
• Infrastructure software tends to succeed more than application-level projects, perhaps because it has a broader appeal.

We'll start off with some observations from free software advocates about why it's so hard to derive production-ready software from academic research.

Software success ≠ academic success

Academics are obsessed with publication. And almost universally, the academic publishers are interested only in research findings, such as: "Packets of a certain size maximize transmission throughput under such-and-such conditions." The publications do not include related data or source code, which are considered, at best, as ancillary and, all too often, as junk. (Things are changing a bit here, especially for government-funded projects, but mostly in the demand for open data, not open source code.)

The irony—and even tragedy—of this disregard for the infrastructure that makes their findings possible is that academics skimp on software quality measures, such as testing. Quite often, bugs in the software cause researchers to publish incorrect results.

In order to get code widely adopted by people outside the academic setting where it is invented, professors or students must first develop the code conscientiously to ensure that it's robust, extendable, and correctly solves the problem at hand. Then they must iron out the idiosyncrasies of the project for which the code was developed, generalizing it for a broader range of domains and purposes. They have to maintain a repository, solicit contributions, and vet those contributions. Ultimately, someone has to run a community that can debate changes and choose new directions. None of those time-consuming tasks has anything to do with publication or tenure. Academics are pressured to get interesting results, put out a paper, and move on to the next project.

James Vasile, a programmer and consultant in the open-source space, notes that academics are biased toward secrecy in their code, as in their research. Woe to them if they release code early in their research that helps competing scientists reach conclusions earlier and get published first. To prevent this career killer, they hold on to the code until they publish their paper or present their conference session. That could be years after they wrote the code, which is pretty late to create a public repository and develop a community around it.

Generally, Vasile told me, it's more likely for academics to share tools that enable their research as infrastructure, but that aren't the main goal of their research. The possibility of turning infrastructure into open source has parallels in commercial firms in a phenomenon I labeled "closed core" six years ago. Companies often want to keep their essential business software secret much like academics want to hold the code for their experiments close to the vest.

Vasile mentioned several other barriers that make it hard to develop open-source projects from academic code. Academics work slowly on code, not at the pace of a professional team. When organizations try to help through donations, universities take a big bite—often half—out of the funds. Finally, many of the tasks required to make software robust and useful are not academically interesting.

Marshall Kirk McKusick, one of the early developers and maintainers of BSD, furnished the additional insight that students usually haven't had the time to develop the skills of maintainable, extendable coding, so their work is quick-and-dirty and unsuitable for reuse. It may also contain useless stub code for features that were never fully developed and that probably never will be.

Additional barriers to freeing code were pointed out by Jeffrey Spies, co-founder and CTO of the Center for Open Science. Researchers rarely think about the traits of software that make it suitable for widespread adoption, such as maintainability or documentation. And even those who make the code available in a public repository have little incentive to foster a community around the software.

High-quality software development requires hiring high-quality software developers, which is difficult in academic settings. Good developers want to work in an environment that appreciates their contributions, a trait unlikely to attract them to academic environments that discount the importance of software quality.

Vasile, McKusick, and Spies presented daunting prospects for successful deployment of academic code. But some projects manage to surmount the hurdles and become free-software successes. Let's look at a few, and try to tease out what helped them succeed.

BSD

BSD, which came out of the University of California, Berkeley, is the crowning success of academically conceived software. McKusick provided a history of BSD's growth and adoption for the O'Reilly Media book Open Sources: Voices from the Open Source Revolution.

One factor that probably allowed BSD to gain wide adoption was its audience of system administrators, who had the skills to install a complicated and sophisticated piece of software on bare metal. Many in the community also submitted contributions to the code. For instance, improvements by the community made 4.2BSD networking more efficient and robust than the code that was originally contributed by Bolt, Beranek, and Newman, which had achieved fame by developing the foundations of the Internet for ARPA (the original name of DARPA). McKusick told me that hundreds of contributors were involved in developing BSD.

Was BSD denied access to sufficient capital? It seems to have been mostly a project of the Berkeley computer science department, although McKusick's article cites funding from DARPA during the critical transition from 3BSD to 4BSD. I see no record of Sun Microsystems, which used BSD as the basis of its SunOS operating system, ever giving money to the project, although it contributed a good deal of code and bug fixes.

Spark

We move now to another project that began at UC Berkeley, but in a very different time and context. A successor to the ground-breaking Hadoop, the Apache Spark cluster-computing engine is part of most "big data" strategies now. Among the projects I've researched for this article, Spark is probably closest to the kind of project that Red Hat will sponsor.

I spoke to one of the early organizers of the Spark project, Patrick Wendell, who left UC Berkeley along with some other team members to found the company Databricks, where he is now VP of Engineering. Wendell told me that Spark was a brainchild of an atypical research group at Berkeley called the AMPLab, where five or six faculty work with about 35 students at any one time on big data processing tools. The AMPLab had both public and private sponsorship, and researchers there were expected to produce software of use to a wide industry audience—as Wendell said, it's "baked into their philosophy." Although projects don't have to be released as free software, many researchers do so to gain the benefits of wide adoption and contributions from the field. For instance, the Berkeley team donated Spark to the Apache Software Foundation in 2013 and built a community of developers outside the AMPLab.

Hence, Wendell said, academics in the AMPLab can have impacts in ways that go beyond publishing papers. They measure success by the broad adoption of their work, not only by insights that get into conferences or journals. He agreed that building community and fixing bugs were not the most efficient path to publication, but for some academics that's fine. Working in the AMPLab does not preclude academic success either—for instance, the Spark project has generated lots of academic publications. Matei Zaharia, founder of the project, took a sabbatical to co-found Databricks but then returned to academia, where he is an assistant professor in the Stanford CS department.

PostgreSQL

This database-management system, perhaps as much as any free software, demonstrates how much can be achieved by developers in an open community. The long and complex history of the project, summarized on a project web page, involved a couple of failed commercialization efforts. But the code of the current PostgreSQL seems to be derived entirely from academic and community efforts, where a Berkeley database project called Ingres inspired another called Postgres, the genesis of modern PostgreSQL. The original developers were part of the same constellation of Berkeley researchers responsible for BSD.

But, as the history page notes: "In 1996, Postgres95 departed from academia". And in a podcast interview, Bruce Momjian suggested that not much of the work on modern PostgreSQL was done at Berkeley, and that PostgreSQL was really a community project from 1996 onward (2:42 into the podcast). He also highlighted the role played by college professors in the PostgreSQL community.

Paradoxically, the academicians don't seem to contribute much to the code, a recalcitrance that Momjian attributes to their lack of interest in practical use (6:30 into the podcast), echoing my conversations with Vasile and Spies. Major funding seems to have come late in the project. Recently, according to Momjian, a number of "big players" have offered support, including IBM, Amazon, and Microsoft (12:57 into the podcast).

Jupyter

Supple enough to be valuable to educators, conference presenters, and general authors alike in the computer field, Jupyter emerged from academic researchers at Cal Poly State University, San Luis Obispo and UC Berkeley. It has brought information presentation into the modern era of multimedia, interactivity, and collaboration. Originally designed to display and run Python code (and called IPython), it was eventually extended so that other computer programming languages could be supported, and its name was changed to Jupyter (still keeping "py" in the name to honor its Python roots and implementation). Jupyter is a central tool in use at my own employer, O’Reilly Media, as was described in a video keynote; it has many other users as well.

In an interview with one of Jupyter's earliest developers, Brian E. Granger, I learned that the Python origins of the project were crucial for historical reasons. Scientists, after years of using proprietary tools such as Mathematica and MATLAB, were turning to powerful Python libraries such SciPy, NumPy, and the many modules that rely on them. According to Granger, these libraries were developed in the early 2000s but were not ready for production use until later in the decade. Two other advantages enhanced their popularity: being cost-free and being easy to mix with other Python libraries for other tasks. Once the Python libraries became fixtures of many fields in science and engineering—particularly the new field that came to be known as data science—their users were open to the interactive educational tools offered by IPython.

It wasn't hard for people outside academia to appreciate IPython. Everybody in the field teaches a course sometimes, or just gives a conference presentation. IPython, and then Jupyter, cut hours from the time it took to put one's code and text into a spiffy presentation form. The project solved several scientific needs at once: repeating experiments in a reliable way, reproducibility of results by other researchers, and teaching or giving talks.

Granger makes no bones about the importance of funding for the success of his project. It benefited quite early from support by Joshua M. Greenberg of the Sloan Foundation, and now it is additionally funded by the Moore Foundation and Helmsley Charitable Trust. The project also has numerous sponsors and institutional partners and gets significant code contributions from about 25 full-time developers.

Conclusion

Each research project that experienced success in the larger software world has found its own path forward. The examples I cited in this article are by no means the end of the story. For instance, the co-founder of the R statistical language, Ross Ihaka, suggested (in a paper) that the developers maintained R for years in a "relatively closed process" and stumbled by necessity onto basic open-source practices such as establishing mailing lists and a group of core committers. This project looks like another example of academic software that could be quickly understood and adopted because the target audience closely resembled the developers and was technically adept.

I haven't covered the Linux kernel or GNU project here, because (in addition to them already being famous) they weren't academic projects, even though Linus Torvalds and Richard Stallman happened to be associated with universities when they launched the projects.

Combining what I heard from project leaders and from the other free-software leaders I interviewed, I suggest that an effort like the Red Hat one I mentioned at the beginning of the article would have the best chance of succeeding by following a few overarching principles. First, choose a project whose value can be quickly understood and embraced by its intended users. Bring in outside experts to evaluate the code for quality to make sure it's worth using; if not, it may make sense to launch a new code base with similar goals. The code must also be easy to generalize and extend. Finally, the project should be taken out of the academic environment as soon as possible (with a payoff to the university, if necessary) and assigned to a project leader who has experience building communities around projects and in recruiting companies or individuals to develop code and all the other infrastructure a free-software project needs.

I'll end with some optimism. Professors and students have routinely turned their ideas into proprietary software. But, given the ease of coding these days, and the resulting commoditization of software, some of these academics are likely to consider making the software free. Apache Spark, discussed earlier, is one example. Another is MapD, a major database project that benefited from advice by Michael Stonebraker, one of the field's leading researchers and entrepreneurs. This company open sourced its core product and has been funded to the tune of 25 million dollars. Fledgling organizations can now turn to organizations such as the Apache Foundation and the Software Freedom Conservancy for organizational advice. In a decade or so, we may know much more about what motivates researchers to open their code, and how they can do so successfully.

[This article is also available in a Portuguese translation by homeyou.]