+1 Linux's low-memory thrashing behavior is a huge problem, much more than CPU starvation.
I've frequently wondered whether throttling memory allocations for memory-intensive apps would be a better solution, than the current system-wide thrashing that takes down *all* processes -- including the X server and ssh.
And now it occurred to me that a per-session memory cgroup could already solve this problem. Not sure how much overhead it adds, though.