The multi-generational LRU

Posted Apr 5, 2021 4:47 UTC (Mon) by yuzhao@google.com (guest, #132005)
In reply to: The multi-generational LRU by ibukanov
Parent article: The multi-generational LRU

Hi,

I'm one of the engineers working on this project. I want to clarify that multigenerational lru does NOT use more CPU. In fact, for all the workloads we could benchmark, it uses less, much less.

But then what's the catch? It retains more (lru) information for each page, in spare space in page->flags. It also stores extra information for each memcg, node, and process. Specifically, it has ~500 bytes per-memcg and per-node and ~50 bytes per-process memory overhead.

Please feel free to send your questions, suggestions, and concerns to page-reclaim@google.com.

The multi-generational LRU

Posted Apr 5, 2021 19:33 UTC (Mon) by ms-tg (subscriber, #89231) [Link]

Thank you for this clarification of the trade-offs and invitation for the community to ask further questions!

The multi-generational LRU

Posted Apr 8, 2021 14:01 UTC (Thu) by smurf (subscriber, #17840) [Link] (1 responses)

These memory overheads seem rather minor, not to say insignificant, compared to the benefits.

Did you identify any workload that fares worse under your algorithm? One may assume that Google has quite a few wildly different workloads to experiment with …

The multi-generational LRU

Posted Apr 13, 2021 9:09 UTC (Tue) by yuzhao@google.com (guest, #132005) [Link]

Sorry for the late reply.

Your assumption is correct. But our "wildly different workloads" are only a fraction of all real-world workloads. AFAIK, we don't use Postgresql, and we do plan to reach out to the Postgresql community and see if they could help us with benchmarking.

And we don't use io_uring (yet). Jens Axboe, the io_uring maintainer helped us identify a regression in buffered I/O and verify the fix. Now there is an improvement compared with the mainline: https://lore.kernel.org/linux-mm/20210413065633.2782273-1...

By design, the multigenerational LRU shouldn't make any workload worse -- why should it make worse decisions after it gathered more information? But that's just the theory :)

The bottomline is we'll do our best to fix any regressions the community reports to us. Again, the email address is page-reclaim@google.com.