get_user_pages(), pinned pages, and DAX

<sigh>

And what we have is a range of responses showing why this topic ends up with people shouting - everyone (except Dan) has (again) missed the point I'm trying to make. Layout leases have /nothing/ to do with specific technology issues (RDMA, GUP pinning, whether SIGBUS is appropriate or not, etc). I'm concerned about how we fulfill the primary responsibility of the kernel: managing user access to hardware robustly and safely.

We've been doing remote direct access to storage in the filesystem/storage world for a long, long time. e.g. clustered SAN filesystems. They have a protocol for access rights, and the node responsible for arbitration can kick out and fence any node that goes rogue or doesn't play by the rules. File layout leases are a similar abstract management protocol to co-ordinate 3rd party access to block devices and to provide notifications that collisions have occurred. The only difference is that layout leases don't do any of the enforcement side of the protocol - they are just tracking and notification and everything else is left up to the application.

So you can all argue over how hard it is to revoke GUP, but that completely misses the point - the point is that without a /notification/ that an access collision has occurred and we need to recall and re-let the leases to resolve the issue, we can't even begin solve this co-ordination problem.

Indeed, what happens when we have multiple different technologies all doing direct access to the storage hardware and they have to be co-ordinated? e.g. pNFS clients push data into the local filesystem, local GPUs or ASICs pull the data direct from the storage, then push the result back direct to teh storage, then another remote node pulls the result direct from the storage via pNFS... How are these all supposed to interoperate safely and correctly?

The fact we have new technologies like DAX, pmem, p2p DMA, etc does not change the fact we still need to co-ordinate all the different direct accesses being made by applications to the storage. If anything, the new technologies make access arbitration even more important to get right.

So, please, if you have an access co-ordination mechanism that can tie pNFS, SMB-direct, DAX, p2p hardware DMA, RDMA push and pull direct to/from storage (e.g. NVMeOF), etc togther with the local filesystem that manages the storage allocations with a coherent, sane, workable management strategy, I'm all ears. Nobody else is suggesting any sort of solution to these high level architecture issues....

If we are going to end up with systems that can make use of all these direct storage access technologies in a reliable, robust manner then we have to solve the general access arbitration and notification problem before anything else. Otherwise we'll just keep going down the road of "nothing works properly with anything else" as we have been doing for the past few years.

-Dave.