Reproducibility [LWN.net]

Reproducibility

Posted Jun 4, 2025 22:08 UTC (Wed) by randomguy3 (subscriber, #71063)
Parent article: The importance of free software to science

I'm suspicious of the idea that reproducibility means being able to run exactly the same software, compiled in exactly the same way. It doesn't look good for the robustness of the result if changing the software, or the implementation of the algorithm, breaks it.

I think the bigger win for using free software (along with open data) is making it harder to hide flawed analysis. When different experiment appear to disagree, it can provide a way of investigating why, and whether something underhanded (or incompetent) has been going on.

to post comments

Reproducibility

Posted Jun 5, 2025 0:20 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

> I'm suspicious of the idea that reproducibility means being able to run exactly the same software, compiled in exactly the same way. It doesn't look good for the robustness of the result if changing the software, or the implementation of the algorithm, breaks it.

Without the former, there is no point in even attempting the latter.

Let's say your new implementation produces different results with the same data. Maybe the problem is with your implementation, maybe the problem is with the algorithm, or maybe the problem is actually in the original, casting the original conclusions into question.

You often (usually?) don't know what differences from the original may turn out to be material, so ideally you'd try to precisely recreate the original results (ie with the same input and exactly-as-described procedure) before changing anything..

Reproducibility

Posted Jun 5, 2025 10:40 UTC (Thu) by farnz (subscriber, #17727) [Link] (2 responses)

The other critical point IME is that it's not unknown for algorithms to have external dependencies that aren't documented because "everyone" uses the same setup.

For example, I've had to work with code that started failing when recompiled for an SGI machine, rather than the Windows NT box it had been written for, and the Debian x86 boxes it had been ported to successfully. We instrumented the code base, and determined that the SGI machine was underflowing intermediates in floating point calculations, where the same code compiled for Windows or Debian was not - because intermediates were being kept in extended precision x87 stack slots, where the SGI was using double precision FPU registers.

Without the ability to compare, I'd just have a chunk of C code that the original author claimed worked "just fine" (and was backed by other users, who also found it working "just fine" on Linux and Windows NT), but which failed on my target system. With the ability to compare, I could find the problem, determine that the hidden assumption is that all FPUs are x87-compatible, and then work with the originator on a fix.

Reproducibility - NixOS

Posted Jun 5, 2025 17:51 UTC (Thu) by gmatht (subscriber, #58961) [Link] (1 responses)

NixOS (and Docker) could be useful to make it explicit what software you think "everybody" has; though, people who know about Nix may not be the ones who make this mistake.

Reproducibility - NixOS doesn't fix hardware assumptions

Posted Jun 5, 2025 18:38 UTC (Thu) by farnz (subscriber, #17727) [Link]

Historically, it's not just been software (which Docker helps with - since you're now running a static userspace, and NixOS can extend across the kernel) - it's been hardware and firmware as well. Nowadays, x87 is dead, but we're not at a stage where all CPUs behave identically for a given source code.

For example, there's still differences in memory consistency models between CPUs, so code that appears to be reproducible on one CPU model may fail on a different system, simply because it relied on hardware implementation details that aren't true of all implementations.

More insidiously, I've encountered algorithms that contain "harmless" race conditions that happen to be not harmless if you swap, or implementations that have problems if there's more than 256 CPU threads available to them (since you can fit the "number of available threads" counter in a byte, right?), or implementations that "know" that you can't have more than 48 virtual address bits (since that's the limit on the CPU they have today), or otherwise embed false assumptions about hardware.

Similarly, I've seen implementations that "know" that PCIe devices must have their BARs allocated below 4 GiB (since the firmware on their machine did that), or that the CPU numbers are contiguous, or that the CPU numbers convert to NUMA node numbers with a simple mask-and-shift, or that odd numbered CPUs are the SMT siblings of the even numbered CPUs.

All of these are details that are often consistent if you're comparing to a bunch of machines your institution bought - but they're details that can change once you change hardware vendors.

Reproducibility

Posted Jun 5, 2025 9:42 UTC (Thu) by magi (subscriber, #4051) [Link]

I tend to agree, although the strict reproducibility (using the same environment and setup) clearly has to hold.

I think that in order to check reproducibility you need automatic tests. Chances are that you will end up with results that are not strictly the same (as in binary equality). The question is, are the results significantly different whatever that might mean in this context. I think this is really one of the big differences between scientific software and "normal" software. In addition what is the correct result anyway, given that simulations are used to explore a system that cannot be tackled otherwise. Another issue with testing scientific software is that the tests might require huge resources (think weather model) or closed data (think medical data).

Having good tests will allow people to convince themselves the software is doing what it is supposed to. It will also help port the software to new systems and deal with the software rot.