The case of the overly anonymous anon_vma

Posted Apr 15, 2010 22:02 UTC (Thu) by i3839 (guest, #31386)
Parent article: The case of the overly anonymous anon_vma

Is it just me who's wondering whether all this complexity is worth it?

What was it needed for again? To have a reverse mapping? Isn't that mostly useful for swapping and not much else? It seems silly to always add the overhead when it's almost never needed. Why not only have the overhead when actually swapping or whenever it's needed? Then this whole mess disappears from the common case and the complex stuff can be separated and isolated.

I have to read more code and think more about it, preferably with a clear mind.

The case of the overly anonymous anon_vma

Posted Apr 23, 2010 13:41 UTC (Fri) by rilder (guest, #59804) [Link] (1 responses)

Even I feel the same. The scenario "In a workload with 1000 child processes and a VMA with 1000 anonymous pages per process that get COWed," -- may not arise in a normal desktop workload. In such as case won't this introduce additional overhead, unless this feature is introduced as a CONFIG variable which I don't think it is.

The case of the overly anonymous anon_vma

Posted Apr 23, 2010 16:17 UTC (Fri) by i3839 (guest, #31386) [Link]

Not only that, if the anon_vma isn't used for file caches, but only for anonymous pages then the whole thing seems dubious when swap is disabled.

In addition to that, anon_vma apparently didn't solve the problem well, but instead of replacing it with something that does, more kludges are added on top of it. I got the feeling that there's some much more elegant solution waiting for someone to find it.

The case of the overly anonymous anon_vma

Posted Apr 24, 2010 20:59 UTC (Sat) by efexis (guest, #26355) [Link] (6 responses)

More complex doesn't necessarily mean less efficient. Having to search through all other pages belonging to all other processes, just to see who, if anyone, is sharing it with you, incures considerably more overhead. There're probably more reasons you'd want to do this than just at page swapout time too, maybe dealing with process memory limits - knowing who to charge the memory usage to, or with NUMA systems becoming more common, knowing whether it's worth moving a page to memory closer to the core running the majority of processes accessing it. Sure it's more complex to us, processing this extra information in our heads, but that little extra time can save a lot of time in the long run.

The case of the overly anonymous anon_vma

Posted Apr 24, 2010 22:48 UTC (Sat) by i3839 (guest, #31386) [Link] (5 responses)

You're forgetting the memory overhead (including cache misses), not everything is pure CPU cycles. Besides that, this adds a cost to most memory operations, even if it turns out it was never necessary. Why slow down the common case to speed up special cases in rare situations?

I don't have time to look further into the details, at least not this month. Hopefully next month.

The case of the overly anonymous anon_vma

Posted Apr 26, 2010 3:03 UTC (Mon) by efexis (guest, #26355) [Link] (4 responses)

"I don't have time to look further into the details"

Well my guess is that they did, and that code that mostly slowed the memory manager down for savings only in a corner case would have been thrown out by Linus, as is often the way.

The case of the overly anonymous anon_vma

Posted Apr 26, 2010 19:09 UTC (Mon) by i3839 (guest, #31386) [Link] (3 responses)

I hope you're wrong. :-)

This seems more a gradual development, with every thing on its own making sense at the time, but together still going in the wrong direction. It for sure isn't a big enough problem yet.

It would be nice to know what the reverse mapping is used for besides swapping. It seems that the reverse mapping of files is a solved problem, but that all this complexity is only for anonymous memory. Which can only go away is you have swap enabled, or some obscure option like memory hotplug. (In the case of hibernation you need to scan all pages anyway, so no need for this either.) So I guess that my main complaint is that this is done even when not needed.

For swap you only need a reverse mapping for pages that might get swapped out soon. I wonder if it's possible to create that mapping only for inactive pages instead of all of them all the time. Alternatively, swapping out can be done per-process, then there's no need for a reverse map. Shared pages are swapped out less quickly, but that's usually better anyway.

The swapping system needs an overhaul anyway, it's very slow currently. This way both the VM and the swap systems can be improved.

Improving the current code may not be easy, but it for sure is possible.

The case of the overly anonymous anon_vma

Posted Apr 30, 2010 20:02 UTC (Fri) by efexis (guest, #26355) [Link] (2 responses)

Hmm... I dunno, I would have thought that this isn't so much only needed for anonymous memory, but this may be the only reason why it's needed for anonymous memory, iyswim... that in other uses there are other reasons for it too, and so it can actually be simpler to just have it, than have it in some places and not in others. Anywhere you have copy-on-write pages it's going to be important, as the way you achieve that is to mark the pages read only. Someone tried to write to it, it causes a memory protection fault, which jumps into the VM code giving it the address of memory that the fault occured while trying to write. So where do you start if you don't have a back link to get from that address to the structures belonging to those using the page? You'd end up just doing loads of searching instead. The same goes for when it's something like memory mapped files, you need to know when the pages have been modified so you know it has to at some point be written back to disk. If you can't look up what's going on from the address it's going on at, things get way more complicated. Seems this is just basic accounting :-/

The case of the overly anonymous anon_vma

Posted May 1, 2010 12:17 UTC (Sat) by i3839 (guest, #31386) [Link] (1 responses)

It is. Reverse mapping for file backed pages doesn't need anon_vma stuff, it's a lot simpler.

You don't need a reverse mapping to do COW, when you get a pagefault you know the virtual address which caused it.

Reverse mapping is needed to find all virtual pages belonging to a certain physical one. That isn't needed often for anonymous memory.

The case of the overly anonymous anon_vma

Posted May 3, 2010 16:51 UTC (Mon) by efexis (guest, #26355) [Link]

You're right about the page fault bit, my mistake, a reference count would be sufficient there, the reverse map would be for changing the backing for purposes of swap or migration in a NUMA or HIGHMEM system where quick knowledge page<-->active sets is required.
Reverse mapping for file backed pages don't need anon_vma stuff obviously, because they're not anon. What they use isn't simpler though, as, for example, a library will often be shared by more address spaces than an anon page, it makes more sense to use a search tree (at least this used to be the case, according to a slightly dated early 2.6.x book detailing it. If it's now simpler than this, that would show kernel devs in fact simplifying things, not adding complexity).
If you use none of these, go into your kernel conf and disable CONFIG_SWAP, CONFIG_NUMA and any CONFIG_HIGHMEM entries. If nothing else uses it, it'll either be removed, or worst case, a few greps will show you where the functions are being used elsewhere to throw a few #ifdef's around (I imagine embedded comunity would have interest in already doing this).