Shared memory mappings for devices [LWN.net]

By Jake Edge
May 7, 2018

In a short filesystem-only discussion at the 2018 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM), Jérôme Glisse wanted to talk about some (more) changes to support GPUs, FPGAs, and RDMA devices. In other talks at LSFMM, he discussed changes to struct page in support of these kinds of devices, but here he was looking to discuss other changes to support mapping a device's memory into multiple processes. It should be noted that I had a hard time following the discussion in this session, so there may be significant gaps in what follows.

A device driver stores the device context in the private_data field of the struct file, Glisse said, which has worked well, but is now becoming a problem. There are new devices that developers want to be able to attach to an mm_struct. In addition, though, those devices are still being used by a legacy API that needs to be preserved.

Glisse said that his first idea was to associate the device context with an mm_struct. That led to various developers to try to better understand the use case. Ted Ts'o summarized what came out of that. He suggested that what Glisse wanted was for every mm_struct to have a unique ID associated with it and to store that unique ID in the device context. Any ioctl() that tried to access the device would only work if the unique ID in the mm_struct of the caller is the same as that buried in the device context. Glisse agreed that would do what he was aiming for.

Ts'o noted that you can't use the address of the mm_struct because that would vary between processes. It wouldn't necessarily be implemented as a unique ID, he said, but that is conceptually how it would work. Kent Overstreet suggested a simple global sequence number for mm_struct. Processes that shared them would have the same sequence number, so the ioctl() enforcement could be done.

After questions about where the changes might lie, Glisse said that he had not written any code yet, but that he did not think changes to the virtual filesystem (VFS) layer would be required. VFS maintainer Al Viro did not really think it mattered where the changes would be made, his question was whether the behavior is needed. Glisse said that it is; it will allow the legacy code to continue running on GPUs, while allowing for more modern uses of the devices going forward.

Index entries for this article
Kernel	Device drivers/Support APIs
Conference	Storage, Filesystem, and Memory-Management Summit/2018

to post comments

Shared memory mappings for devices

Posted May 7, 2018 20:31 UTC (Mon) by willy (subscriber, #9762) [Link] (10 responses)

Argh, I missed this session. Thanks for writing this up, Jake!

I have a preliminary patch to use an IDR to assign each mm_struct a u32 ID. There are various details around how quickly those can be reused. It actually saves memory (replacing the list_head with a u32). I'll try to get that finished up soon.

Shared memory mappings for devices

Posted May 7, 2018 20:42 UTC (Mon) by luto (subscriber, #39314) [Link] (2 responses)

On x86, we already have ctx_id. That could easily be moved to core code.

Shared memory mappings for devices

Posted May 7, 2018 21:02 UTC (Mon) by willy (subscriber, #9762) [Link] (1 responses)

I talked with Dave Hansen about ctx_id already. mm_id can replace it.

Shared memory mappings for devices

Posted May 7, 2018 21:37 UTC (Mon) by luto (subscriber, #39314) [Link]

Would it be u64? It’s quite important that it never gets reused.

Shared memory mappings for devices

Posted May 8, 2018 21:29 UTC (Tue) by mirabilos (subscriber, #84359) [Link] (6 responses)

What’s an IDR?

Asking for my acronyms database…

Shared memory mappings for devices

Posted May 8, 2018 21:33 UTC (Tue) by willy (subscriber, #9762) [Link] (3 responses)

https://www.kernel.org/doc/html/latest/core-api/idr.html

Shared memory mappings for devices

Posted May 11, 2018 7:37 UTC (Fri) by jem (subscriber, #24231) [Link] (2 responses)

Ironically, the documentation you linked to doesn't explain either what IDR or IDA are short for.

I miss Byte magazine, which meticulously spelled out an acronym the first time it appeared in an article, no matter how common. For example: "Random Access Memory (RAM)".

Shared memory mappings for devices

Posted May 11, 2018 14:16 UTC (Fri) by willy (subscriber, #9762) [Link]

Oh, I have no idea what either IDR or IDA stand for. It's not stated anywhere in the source code. I suppose I could search the mailing lists to find if there was any explanation when they were first published.

Shared memory mappings for devices

Posted May 15, 2018 23:52 UTC (Tue) by giraffedata (guest, #1954) [Link]

IBM used to do that in all its manuals as well, and it's an irritating style because acronyms are not abbreviations - they're words in their own right that are better known and have more meaning than the words from which they are derived. Sometimes the acquired meanings even contradict those words. The etymology of the acronym is still useful, of course, but the right way to do it is RAM (Random Access Memory), not Random Access Memory (RAM).

RAM is an interesting example, by the way, because words used illogically was another peeve Byte editors had and the magazine never used RAM to mean what most people meant by it: read/write memory. The term in Byte for that was "programmable memory." RAM would appear in Byte only in contrast with sequential access memory.

Shared memory mappings for devices

Posted May 11, 2018 16:44 UTC (Fri) by cladisch (✭ supporter ✭, #50193) [Link] (1 responses)

IDR is an ID allocator; the "R" stands for "radix tree", which is just an implementation detail.

IDA is an ID allocator that does not store a data poiner for each ID.

Shared memory mappings for devices

Posted May 14, 2018 13:16 UTC (Mon) by dezgeg (guest, #92243) [Link]

Maybe IDR could be made a backronym of say, "ID Registry" to hide the radix tree implementation detail.

Shared memory mappings for devices

Posted May 7, 2018 21:27 UTC (Mon) by spopuri (guest, #47635) [Link]

What is this for exactly? There have been other discussions about representing memory provided by an nvme drive with struct page structures. Is that the use-case? or is it more for things like HMM?

Don't make this visible to userspace

Posted May 8, 2018 2:39 UTC (Tue) by ebiederm (subscriber, #35028) [Link]

This sounds like yet another darn global id that someone is going to make visible to userspace
and then it will need to get namespaced and then it's utility for device drivers will be gone.

So be very careful and don't even consider making something like this visible to userspace unless you intend to make it globally unique and even then hesitate and make absolutely certain that is what you want.

Shared memory mappings for devices

Posted May 8, 2018 14:34 UTC (Tue) by Sesse (subscriber, #53779) [Link]

This isn't about this specific post, but it had to go somewhere: Thanks for the LFSMM coverage. Almost all the articles so far have been highly interesting.