A scientific basis for Open Source Software

You've got to be kidding. Results are produced by implementation, not by algorithm. I.e. by the algorithm plus the bugs present in said implementation. "We have reimplemented the algorithm and results differed from those given in the paper in the following respects" is a _lot_ weaker and less useful than "reviewing the implementation in appendix B of the paper reveals the following bugs and results are affected by those in the following respects".

While we are at it, rigorously assessing one's *own* code is much harder than doing that to code written by somebody else. There's a reason why mutual code review is useful...

If one doesn't want to sink down to the level of "soft sciences" (or alchemy, for that matter[1]), description of methods and materials should be detailed enough to make it realistically possible to investigate how do results depend on the details of those. Otherwise the results are unfalsifiable, with everything that follows from that.

[1] aka "your failure to reproduce my results only proves that you are less spiritually advanced than I am and need to work harder to elevate your soul to my level" - and no, that's not a parody. The discipline really had been infected by that from its inception and it had taken Boyle et.al. to dump that "spiritual" garbage. At which point it had become chemistry...