This seems more a gradual development, with every thing on its own making sense at the time, but together still going in the wrong direction. It for sure isn't a big enough problem yet.
It would be nice to know what the reverse mapping is used for besides swapping. It seems that the reverse mapping of files is a solved problem, but that all this complexity is only for anonymous memory. Which can only go away is you have swap enabled, or some obscure option like memory hotplug. (In the case of hibernation you need to scan all pages anyway, so no need for this either.) So I guess that my main complaint is that this is done even when not needed.
For swap you only need a reverse mapping for pages that might get swapped out soon. I wonder if it's possible to create that mapping only for inactive pages instead of all of them all the time. Alternatively, swapping out can be done per-process, then there's no need for a reverse map. Shared pages are swapped out less quickly, but that's usually better anyway.
The swapping system needs an overhaul anyway, it's very slow currently. This way both the VM and the swap systems can be improved.
Improving the current code may not be easy, but it for sure is possible.