Determinism, reproducibility, and numerical analysis

Posted Jun 9, 2025 9:54 UTC (Mon) by farnz (subscriber, #17727)
In reply to: Determinism, reproducibility, and numerical analysis by DemiMarie
Parent article: The importance of free software to science

CUDA's non-determinism is in scheduling and memory access, not computation itself. This is the same guarantees as you get with a CPU; the computation is deterministic, but the point at which you work across CPU cores is non-deterministic.

However non-determinism is not the same as non-reproducibility. There exist plenty of non-deterministic algorithms that have a deterministically reproducible output; it's just extra analysis steps to confirm that your result is reproducible from a set of non-deterministic intermediate steps.

And note that for science purposes, we don't necessarily even need a deterministic output; if your prediction is "this simulation will never have an output whose mean is below 64.0", you can do an analysis that confirms that the algorithm's error bars on the mean are (say) ±31.5 due to non-determinism. Then, if the simulation gets an output of 127.0, you know that, while you can't reproduce the exact value, you can deterministically answer the question "is the mean of the output greater than or equal to 64.0" with "yes, because the worst case an attempted reproducer will see is a mean of 95.5, and the error bar on that will tell them that the algorithm's output must be above 64.0".