get_user_pages(), pinned pages, and DAX

Hi dgc, I don't know where the "people are not looking at it" interpretation is coming from, and I can't parse that "not the approach that people have been looking at" quote because the fact is Ira and I were the only ones in the room besides Christoph trying to advance the XFS position. Ira has taken it further with a proof-of-concept implementation posted to the list, but the conversation devolved into a shouting match before we could even get to those details.

Given the strong feelings, and that POSIX seems to allow both behaviors, I'm of the opinion that the fsdax core should implement the lease capability as library functionality that a filesystem can opt-in to, and let the RDMA folks take it up directly with the filesystems if they disagree with the local fs policy, or otherwise leave the status quo of simply blocking indefinite pinning on dax mappings. I'm personally of the opinion that failing truncate() is an abdication of the kernel's responsibility to try to ensure forward progress of the most recently requested system state, but it's clear there will not be wider consensus on that opinion any time soon. So, in the interest of moving forward with *any* lease implementation for this problem I'd suggest just starting with XFS for now.

I think layout leases and some mmap of the block device could certainly address some use cases, and it would be better than the nothing we have today..

But the entire objection to any sort of lease has always been that the plan to SIGKILL userspace after a timeout is horrible and unworkable in the real world. If someone came up with a better alternative I haven't heard it..

I gather the way pNFS/etc handle layout lease revoke in-kernel is generally OK, but those techniques don't translate to userspace??

<sigh>

And what we have is a range of responses showing why this topic ends up with people shouting - everyone (except Dan) has (again) missed the point I'm trying to make. Layout leases have /nothing/ to do with specific technology issues (RDMA, GUP pinning, whether SIGBUS is appropriate or not, etc). I'm concerned about how we fulfill the primary responsibility of the kernel: managing user access to hardware robustly and safely.

We've been doing remote direct access to storage in the filesystem/storage world for a long, long time. e.g. clustered SAN filesystems. They have a protocol for access rights, and the node responsible for arbitration can kick out and fence any node that goes rogue or doesn't play by the rules. File layout leases are a similar abstract management protocol to co-ordinate 3rd party access to block devices and to provide notifications that collisions have occurred. The only difference is that layout leases don't do any of the enforcement side of the protocol - they are just tracking and notification and everything else is left up to the application.

So you can all argue over how hard it is to revoke GUP, but that completely misses the point - the point is that without a /notification/ that an access collision has occurred and we need to recall and re-let the leases to resolve the issue, we can't even begin solve this co-ordination problem.

Indeed, what happens when we have multiple different technologies all doing direct access to the storage hardware and they have to be co-ordinated? e.g. pNFS clients push data into the local filesystem, local GPUs or ASICs pull the data direct from the storage, then push the result back direct to teh storage, then another remote node pulls the result direct from the storage via pNFS... How are these all supposed to interoperate safely and correctly?

The fact we have new technologies like DAX, pmem, p2p DMA, etc does not change the fact we still need to co-ordinate all the different direct accesses being made by applications to the storage. If anything, the new technologies make access arbitration even more important to get right.

So, please, if you have an access co-ordination mechanism that can tie pNFS, SMB-direct, DAX, p2p hardware DMA, RDMA push and pull direct to/from storage (e.g. NVMeOF), etc togther with the local filesystem that manages the storage allocations with a coherent, sane, workable management strategy, I'm all ears. Nobody else is suggesting any sort of solution to these high level architecture issues....

If we are going to end up with systems that can make use of all these direct storage access technologies in a reliable, robust manner then we have to solve the general access arbitration and notification problem before anything else. Otherwise we'll just keep going down the road of "nothing works properly with anything else" as we have been doing for the past few years.

-Dave.