> The reproducibility argument is a weak one. The methodologies are
> described in any paper worth its salt so the software could be (and often
> should be) rewritten independently. Indeed such a rewrite is a better test
> of the original results since large codes are hard to rigorously assess if
> you did not write them.
Ideally yes. Unfortunately this doesn't happen, ESPECIALLY in computer science and engineering research. Half my dissertation was about how completely unreproducible results are. Small variations in assumptions resulted in large changes in outcomes. In my case changes to either or both of the hardware model and the compiler algorithm running on it gave wildly different results.
We think we have a good intuition of how software and hardware works. We don't. It is impossible to reproduce the models presented in papers precisely because describing them to an adequate degree would essentially be a replication of the source code.