Ten simple rules for the open development of scientific software

Linux distributions mostly provide only one version of a library at a time. Since a program can depend on many libraries, somewhere along the stack some code will be different from the original code almost immediately. If the code is run on a different distribution, it will be running using a different version of libraries that the software depends on; if only because the build flags are different.

We can pretend that this should not make a difference, but it does. The software stacks are so complex nowadays, that it's very hard to predict how software will behave.

The closest that scientific software can get to being reproducible is to provide the software compiled for a virtual machine with all the required libraries also implemented in that virtual machine and distributed in source and binary form with the software.

The best candidate for this is currently Java, where the entire set of code, libraries and required data and configuration files can be put in a single jar file. Java has a stable instruction set and is available on many platforms. The behavior of e.g. floating point operations is standardized and independent of compiler flags as it is for compiled code.

It's always infuriating to read a CS paper that contains an "experimental results" section, clearly indicating they implemented their idea, and not have the code.

* Many scientific programs are written by scientists -- and, more significantly, by newbie science grad students -- who have no training and little experience in writing readable and reusable software. I learned things like "write your program in small chunks that can be independently tested" by doing it wrong and learning from my mistakes. My mentors were all scientists, not programmers; they had more experience but no more training, and nobody even thought of looking at my code with the same red pen they used on my papers. And certainly there were no classes in writing good programs to go along with the classes in writing good papers! (Personally I should also credit the people of comp.lang.fortran and the GCC development list; most of what I learned about good coding style I learned from paying attention to discussions there -- but that's unusual.)

* Many scientific programs are written to solve an immediate single need to demonstrate a particular result. I remember a case where there was a filter shape that ideally would have come from an input file, but I was up against a hard conference deadline and hardcoding it was faster and simpler. Likewise, the code for my final dissertation work involved a horrible mess of Intel Fortran for Windows, a custom-built Cygwin G++, and hardcoded Windows-to-Cygwin directory name conversions in makefiles -- but it worked on my computer, and it got the job done and let me collect a lot of necessary data in short order. When you're making a one-time-use tool, there is very little added value in making it pretty, or making it reusable.

* Publishing software openly does not magically create a community that will spread the maintenance load. As a programmer I have worked on a significant piece of library software (Sourcery VSIPL++) with an open standard interface, a small standardization community, a commercial user base, and for quite some time a sponsored GPL implementation with an associated openly-archived mailing list. I believe we had a number of users of the GPL implementation, but in all the years we had it, we heard from very few of them, and received no patches or other maintenance help. The reality is that most scientific software is not useful to enough people to sustain a development community, and even where it is, sustaining a community requires substantial ongoing work.

* As a general rule of thumb that I've learned from commercial work on open-source projects (now that I'm a professional programmer), the amount of time that it takes to push a patch upstream to an open-source project and get it accepted is about the same as the amount of time that it takes to write something decent that works in the first place -- and that's by people who are good at open-source community politics, which itself is a learned skill. That will apply to people contributing to existing scientific projects, and it's probably a lower bound for the amount of work required to document the internals of a new piece of software and make it readable and portable so that it's something that other people could use.

The problem of making scientific software open source in a really useful way (rather than just a useless unreadable code dump) is difficult to solve because it fundamentally requires a lot more effort, and because it requires a significant amount of training of new scientists. At a rough estimate, it requires at minimum a full semester-long class and ongoing mentoring by experienced programmers, and about a doubling of the effort and time required for writing programs. With science Ph.D. times already extending well beyond what's sustainable, this implies a notable reduction in the amount of work that can be done in a Ph.D., and requires bringing additional programming expertise into scientific departments to teach the classes and provide the mentoring and perhaps reduce some of the programming load on the scientists.

(In support of that: For scientific laboratories, we have machine-shop classes for students and we have laboratory technicians and machinists to help the researchers build the laboratory equipment, because we have accepted that this is skilled labor that is best done with a lot of assistance by professionals. Why don't we do the same with software?)

This is a hard problem because there are real, and significant, costs involved in solving it -- both in funding, and in graduate student time. Platitudes, even well-explained ones, will not pay those costs. We need to be having the hard conversations about what the costs are, why they are important to pay -- and whether they truly are! -- and how university research departments can pay them.