Who's afraid of a big bad optimizing compiler?

I expect one of the main reasons is that it's extremely difficult to define "the right thing". Firstly because different people will have different ideas of what's right; and secondly because concurrency and memory models are inherently complex and subtle topics, so it's always difficult to reason about them precisely. See how modern C++ tries to specify it with terminology like "is sequenced before", "carries a dependency to", "inter-thread happens before", etc, all with precise meanings that are almost impossible for a normal human to remember. Java tried to specify a memory model from the start, but got it wrong, and it took a decade to understand the mistakes and fix them.

Probably the other main issue is performance. Particularly with concurrency where CPUs often require explicit memory barriers if you want guaranteed ordering, and if the compiler added memory barriers around every memory access just in case you were using the same memory in another thread, it would be pretty slow. Even if it was only a tiny performance regression, many C programmers care a lot about performance (especially microbenchmark performance) and won't be happy with a new compiler that's measurably slower, and users won't be happy if their application runs slower after updating their kernel. They'd rather have a dangerous but fast compiler, and rely on the programmer being smart enough to avoid those dangerous cases.

1 while (!need_to_stop) /* BUGGY!!! */
2     do_something_quickly();

2 if (global_ptr != NULL &&
3     global_ptr < high_address)
4         do_low(global_ptr);

> I know, the first thing the article says is: "the C standard grants the compiler the right to make some assumptions that end up causing these weird/non-obvious things if you don't guard against them", but I must ask: why can't the compiler grow an optimization mode that does the right thing (and thus avoid all of the issues documented here) in the presence of global variables and concurrent execution? that situation (globals+concurrence) is so common, that solving it in the compiler would benefit most, if not all, C users, in both kernel and user space.

I'd say that C11/C++11 made a large step in that direction, by having a formalized memory model, and builtin atomics. Before that there really was no way to not rely on compiler implementation details to get correctly working (even though formally undefined) concurrent programs.

It does require you however to actually use the relevant interfaces.

I don't quite see how you'd incrementally get to a language that doesn't have any of these issues, without making it close to impossible to ever incrementally move applications towards that hypothetical version of C. I mean there's basically no language that allows to use shared memory and doesn't require escape hatches from its safety mechanisms to implement fast concurrent datastructures (e.g. rust needing to go to unsafe for core pieces). And the languages that get closes require enough of a different approach that it's hard to imagine C going towards it.

That's not to say that C/C++ have sufficiently progressed towards allowing to at least opt into safety. The C11/C++11 memory model and the atomics APIs are a huge step, but it's happened at the very least 10 years too late (while some of the formalisms where developed somewhat more recently, there ought to at least have been some progress before then). And there's plenty other issues where no proper ways are provided (e.g. signed integer overflow handling, mentioned in nearby comments).