A scientific basis for Open Source Software

My wife is a researcher in computational linguistics. I've seen her waste months of her precious time trying to figure out why published (and widely cited) results don't match up with their published algorithm. These are the kinds of questions that could be easily answered by examining source code and/or by experimenting on the executable, but are more or less impossible to answer without access to either. After watching this happen up close (I help her on occasion with programming work for her research, so I understand some of what she does), I became convinced that reviewers should start rejecting any paper that comes without full executable source code, on the basis that the results in the paper are not reproducible.

Now, maybe other fields than computational linguistics rely on simpler or more declaratively specifiable computation, but I doubt it. My guess is that if you pick a recent, widely cited paper from most modern fields of science and attempt to reproduce it, that it will take many months of programming effort on your part, and more than likely that you'll fail -- the ultimate results your attempted reproduction emits will not be identical to the original, and you'll be unable to determine why it is different.