Determinism, reproducibility, and numerical analysis

Posted Jun 7, 2025 16:48 UTC (Sat) by DemiMarie (subscriber, #164188)
In reply to: Determinism, reproducibility, and numerical analysis by farnz
Parent article: The importance of free software to science

At least CUDA explicitly does not guarantee reproducibility unless the hardware and software are unchanged. In fact, you have to opt-in to determinism in some cases.

Determinism, reproducibility, and numerical analysis

Posted Jun 9, 2025 9:54 UTC (Mon) by farnz (subscriber, #17727) [Link]

CUDA's non-determinism is in scheduling and memory access, not computation itself. This is the same guarantees as you get with a CPU; the computation is deterministic, but the point at which you work across CPU cores is non-deterministic.

However non-determinism is not the same as non-reproducibility. There exist plenty of non-deterministic algorithms that have a deterministically reproducible output; it's just extra analysis steps to confirm that your result is reproducible from a set of non-deterministic intermediate steps.

And note that for science purposes, we don't necessarily even need a deterministic output; if your prediction is "this simulation will never have an output whose mean is below 64.0", you can do an analysis that confirms that the algorithm's error bars on the mean are (say) ±31.5 due to non-determinism. Then, if the simulation gets an output of 127.0, you know that, while you can't reproduce the exact value, you can deterministically answer the question "is the mean of the output greater than or equal to 64.0" with "yes, because the worst case an attempted reproducer will see is a mean of 95.5, and the error bar on that will tell them that the algorithm's output must be above 64.0".