LWN: Comments on "Idle and stale page tracking"

Idle and stale page tracking

sethml — Sun, 09 Oct 2011 21:22:56 +0000

Because they have another option: move a process to another machine. Imagine you start a batch job that requires 10,000 processes. Some of your processes may get dumped on machines running web search. Now imagine one of those search processes gets a bunch of requests and its working set increases - your process may get killed to avoid making web search thrash. The controller for your job can then move that process's work somewhere else. This lets Google take advantage of underused resources on production clusters without disrupting production services.

Idle and stale page tracking

giraffedata — Fri, 07 Oct 2011 20:45:31 +0000

Why does Google partition the memory? If they're just going to move memory from partition to partition to make the utilization even, that seems to take back one of the typical benefits of partitioning memory.

This could be quite interesting for me.

ejr — Thu, 06 Oct 2011 14:55:53 +0000

I work on massive graph analysis. We assume the graph is in memory (for multiple reasons, and it's appropriate for our target) and ideally in the multi-TiB size. Many of our graph analysis kernels skip around the graph in an unpredictable manner and run for hours. Assuming that pages are allocated per socket on ye typical NUMA machines, the idle page data could give us a general way to measure aggregate load balance for memory operations or help us determine which old graph edges to drop when we need to add newer edges. Sure, we could collect this data ourselves, but it's always nice when someone does it for you...

Thank you for pointing this out!