Why struct meta_page? [LWN.net]

Why struct meta_page?

Posted Aug 3, 2007 6:00 UTC (Fri) by balbir_singh (subscriber, #34142)
In reply to: Why struct meta_page? by i3839
Parent article: Controlling memory use in containers

There's an OLS paper Challenges with the memory controller that describes some of the challenges we face as we design our memory controller. Why we don't directly extend struct page is explained in the paper (hint: number of struct page(s) is directly propotional to the memory in the system)

(Log in to post comments)

Why struct meta_page?

Posted Aug 3, 2007 14:35 UTC (Fri) by i3839 (guest, #31386) [Link]

Thanks for the pointer, interesting paper. It explains a lot, but it didn't convince me yet.

You started with an unmodified struct page. Then it turned out adding a pointer back to the struct meta_page was really worth it, so you did. Now you end up with:

/*
 * A meta page is associated with every page descriptor. The meta page
 * helps us identify information about the container
 */
struct meta_page {
	struct list_head lru;		/* per container LRU list */
	struct page *page;
	struct mem_container *mem_container;
};

and a struct meta_page *meta_page; in struct page.

That are 5 pointers, of which two are used to link a meta_page and its corresponding page to each other, which is a very big overhead and probably also makes the code more complex.

So the current method uses (1 * nr_pages + 4 * A) * sizeof(pointer) memory, where 'A' is the number of pages associated with a meta page.

Getting rid of struct meta_page and embedding lru and *mem_container directly in struct page will use 3 * nr_pages * sizeof(pointer).

Using that we can calculate which method is under which circumstances more memory efficient:

1 * nr_pages + 4 * A  <  3 * nr_pages
-> 4 * A < 2 * nr_pages
-> A < nr_pages/2

Assuming that 'A' equals the number of active pages, the conclusion is that a separate struct meta_page is only better when less than half of all memory is active. But:

1) That seems like an uncommon condition, so in general it's expected that it will be much more than that (e.g. on my system the ratio is 1/5 in favour of active pages, and the few other systems I checked also have much more active pages than inactive ones).

2) Optimizing memory usage makes only sense for when there is memory shortage, so to see what approach is most effective it should be compared under memory pressure conditions. If the number of active pages is low it can be expected that the memory pressure is low too.

What am I missing?

Why struct meta_page?

Posted Aug 4, 2007 9:30 UTC (Sat) by balbir_singh (subscriber, #34142) [Link]

Your calculation seems accurate, but remember not everyone wants to use the memory controller. For non-users of the feature, we could later add a boot option and the overhead would be (with config enabled)

nr_pages * sizeof(pointer)

without this, the overhead would clearly be

3 * nr_pages * sizeof(pointer)

I need to double check this, but the sizeof struct page is currently close to being aligned in one cacheline, continuous and uncontrolled extensions can hurt in the long run (may be even now).

Why struct meta_page?

Posted Aug 4, 2007 13:20 UTC (Sat) by i3839 (guest, #31386) [Link]

Currently it's 56 bytes for 64 bit systems and 32 bytes for 32 bit ones, so you're right that adding pointers is bad, but that's also true for the one pointer you add (less so for 64 bit systems though).

Can't you reuse the struct list_head lru in struct page for the container instead of using two lists? That would save two pointers. If you need to do global reclaim you can do it per container, ordered on how "full" they are.

Also, is the struct mem_container* pointer really needed? Don't you know which container is involved from the context it's used? And the few times you really need it, can't you walk the lru list to find it out? (E.g. add struct mem_container to the lru list and distinguish it by setting a bit in the lru pointer or some other trick).

Just throwing ideas at you.

Why struct meta_page?

Posted Aug 6, 2007 17:26 UTC (Mon) by i3839 (guest, #31386) [Link]

If the above method of reusing the current lru isn't possible, an alternative would be to keep around a shadow array of struct meta_page, assuming currently an array of struct pages is used starting at a known location. Then you can calculate the index of the other structure from the memory address of either structure, also making those two pointers redundant, with the added benefit that struct page size isn't increased and the memory is only allocated when containers are used.

Why struct meta_page?

Posted Aug 3, 2007 14:37 UTC (Fri) by i3839 (guest, #31386) [Link]

Unrelated to the above, your patch has

+#define PG_metapage 21 /* Used for checking if a meta_page */
+ /* is associated with a page */

but the flag only seems to be used for locking, not for checking anything like the comment says?