Reproducibility

Posted Jun 5, 2025 9:42 UTC (Thu) by magi (subscriber, #4051)
In reply to: Reproducibility by randomguy3
Parent article: The importance of free software to science

I tend to agree, although the strict reproducibility (using the same environment and setup) clearly has to hold.

I think that in order to check reproducibility you need automatic tests. Chances are that you will end up with results that are not strictly the same (as in binary equality). The question is, are the results significantly different whatever that might mean in this context. I think this is really one of the big differences between scientific software and "normal" software. In addition what is the correct result anyway, given that simulations are used to explore a system that cannot be tackled otherwise. Another issue with testing scientific software is that the tests might require huge resources (think weather model) or closed data (think medical data).

Having good tests will allow people to convince themselves the software is doing what it is supposed to. It will also help port the software to new systems and deal with the software rot.