This is why we can't have safe cancellation points

Hm, no, but the L2+ caches are unified on many models (e.g. on all the Intel x86-64 CPUs I have access to, Nehalem and later), and getting stuff from L2 cache is still immensely faster than getting it from RAM, fast enough that you can often consider it free for applications like this.