ZONE_DEVICE and the future of struct page

Posted Mar 21, 2017 19:28 UTC (Tue) by roc (subscriber, #30627)
Parent article: ZONE_DEVICE and the future of struct page

Why would anyone let SPARC64 requirements drive a design decision like this? That architecture is marginal and dying. And in this case it sounds like it could be worked around with some effort by the SPARC64 maintainer(s).

ZONE_DEVICE and the future of struct page

Posted Mar 21, 2017 21:49 UTC (Tue) by willy (subscriber, #9762) [Link]

I don't think it's accurate to say "marginal and dying". For one thing, it is my strong suspicion that we will see persistent memory on SPARC64 CPUs given Oracle's focus. It has a public roadmap going out to 2021.

We wouldn't let, say, FR-V or Alpha disrupt persistent memory features, but I think SPARC64 is still relevant.

ZONE_DEVICE and the future of struct page

Posted Mar 22, 2017 6:33 UTC (Wed) by flussence (guest, #85566) [Link] (4 responses)

The kernel's golden rule is “don't break userspace”. Intentionally breaking an entire class of currently-working systems for the sake of being lazy is a pretty awful thing to suggest.

ZONE_DEVICE and the future of struct page

Posted Mar 22, 2017 22:04 UTC (Wed) by roc (subscriber, #30627) [Link] (3 responses)

It's not about being lazy, it's about tradeoffs.

Suppose for the sake of argument that the only two options are #1 drop SPARC64 support or #2 all architectures must waste 64 bytes per 4K page. If the "don't break things" rule forces choice #1 then that means Linux performance on much more common systems is being dragged down by legacy baggage (especially after more similar decisions accumulate).

Now suppose there's another option, #3 waste no memory and do a bunch of rearchitecting of the SPARC64 port to handle it. That sounds good, but what if the SPARC64 maintainer doesn't want to do the work? If you're 100% committed to "don't break things" then you can't motivate them by threatening to drop SPARC64 support. Instead the burden of reworking SPARC64 falls on whoever's implementing the core feature. That really sucks for various reasons, but in particular you're making core development that benefits many users much more difficult for the sake of some relatively very small number of users.

"Lazy" is a pejorative term implying moral deficiency. There's nothing morally deficient about being honest about taking these tradeoffs seriously.

ZONE_DEVICE and the future of struct page

Posted Mar 23, 2017 22:25 UTC (Thu) by djbw (subscriber, #78104) [Link] (2 responses)

In fact, it's not a waste. It's fundamental to many kernel paths. The DAX enabling without pages loses get_user_pages() support which disables not only DMA / direct-I/O, but also fundamental operations like fork and ptrace. We're already paying this 1.5% overhead for main memory, and my argument is that we should simply pay that overhead for persistent memory as well. It's not enough to convert some paths to use pfn_t and with new kmap() primitives, because that leaves us an ongoing maintenance burden of dual code paths as developers add new struct page usages. Unless we create a plan to get rid of struct page everywhere we should not special case persistent memory... especially when we have a mechanism to pay the overhead cost from pmem itself.

Once we mandate struct page for DAX this appears to open up several clean up opportunities like re-using more of the the core page cache implementation and unifying device-DAX / filesystem-DAX.

ZONE_DEVICE and the future of struct page

Posted Mar 23, 2017 23:39 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

There are problems with that. Persistent memory is fast and durable, but not as fast and durable as the regular volatile RAM. So you have to put your page structures in the main RAM and this adds up quickly - a 2Tb persistent array will require around 32Gb or page structures.

ZONE_DEVICE struct page != ZONE_NORMAL struct page in terms of write rate

Posted Mar 24, 2017 0:15 UTC (Fri) by djbw (subscriber, #78104) [Link]

I'm not convinced that's going to be a problem in practice. Consider that the bulk of what makes struct page a frequently accessed data structure is when it is used by the core mm for general purpose page allocations. The ZONE_DEVICE mechanism never releases these pages for that high frequency usage. Another mitigation is that struct page writes are buffered by the cpu cache, which further reduces the write rate to media.