Reproducibility - NixOS doesn't fix hardware assumptions

Reproducibility - NixOS doesn't fix hardware assumptions

Historically, it's not just been software (which Docker helps with - since you're now running a static userspace, and NixOS can extend across the kernel) - it's been hardware and firmware as well. Nowadays, x87 is dead, but we're not at a stage where all CPUs behave identically for a given source code.

For example, there's still differences in memory consistency models between CPUs, so code that appears to be reproducible on one CPU model may fail on a different system, simply because it relied on hardware implementation details that aren't true of all implementations.

More insidiously, I've encountered algorithms that contain "harmless" race conditions that happen to be not harmless if you swap, or implementations that have problems if there's more than 256 CPU threads available to them (since you can fit the "number of available threads" counter in a byte, right?), or implementations that "know" that you can't have more than 48 virtual address bits (since that's the limit on the CPU they have today), or otherwise embed false assumptions about hardware.

Similarly, I've seen implementations that "know" that PCIe devices must have their BARs allocated below 4 GiB (since the firmware on their machine did that), or that the CPU numbers are contiguous, or that the CPU numbers convert to NUMA node numbers with a simple mask-and-shift, or that odd numbered CPUs are the SMT siblings of the even numbered CPUs.

All of these are details that are often consistent if you're comparing to a bunch of machines your institution bought - but they're details that can change once you change hardware vendors.