Device-to-device memory-transfer offload with P2PDMA [LWN.net]

Device-to-device memory-transfer offload with P2PDMA

Posted Oct 3, 2018 11:38 UTC (Wed) by dullfire (guest, #111432)
Parent article: Device-to-device memory-transfer offload with P2PDMA

There seems to be a mistake in the article.
While I have not read the patch set, it would not make sense to require "all devices involved are behind the same PCI bridge".
I suspect the term "PCI host bridge" was intended (because that would have the effect that the paragraph describes). Furthermore, since in PCIe all devices have their own PCI bridge (devices, not functions. Also, as a quick overview, a PCIe switch is made up of set of PCIe bridges, one for upstream... and one for each downstream port), it would effectively be impossible to have two PCIe device ever use this functionality. Which would render it moot.

to post comments

Device-to-device memory-transfer offload with P2PDMA

Posted Oct 3, 2018 16:25 UTC (Wed) by mrybczyn (subscriber, #81776) [Link] (2 responses)

Yes, you're right. It would be more accurate to say "behind a host bridge". You will find more about it in the last part of the article when it talks about the use cases.

Device-to-device memory-transfer offload with P2PDMA

Posted Oct 4, 2018 21:13 UTC (Thu) by jgg (subscriber, #55211) [Link] (1 responses)

'behind the same bridge' is the right language, if not a little confusing. It doesn't mean 'behind the last bridge' but simply any bridge. Ie the upstream bridge of a switch is sufficient to satisfy the condition, even though there are later bridges before reaching the device.

Behind the same root port (for PCI-E) is not quite the same thing, ie two functions on the same device cannot do P2P DMA with this patch series if they are plugged directly into the root port.

All that aside, this series does have the requirement that the devices be behind a switch. You can't use it on a GPU and a NVMe drive plugged directly into root ports on your CPU, for instance. This greatly limits the utility, and hopefully will go away eventually when people can figure out how to white list root complexes and BIOSs that support this functionality.

Device-to-device memory-transfer offload with P2PDMA

Posted Dec 7, 2024 11:51 UTC (Sat) by sammythesnake (guest, #17693) [Link]

> two functions on the same device cannot do P2P DMA with this patch series if they are plugged directly into the root port

That seems like a fairly likely use case - passing off some data from one stage of processing to another, so hopefully this restriction is lifted soon. I imagine that's a direction in the developers' sights, though - I'm happy to assume that my negligible level of domain knowledge is outdone by theirs ;-)

A couple of possible factors that might make it less of an urgent need occur to me, though, how likely are these, I wonder...?

1. How common would it be for these related functions to be plugged into the root, rather than sharing a (device internal?) bridge?

2. I imagine such devices might simply share the memory between the stages and not need DMA at all for this kind of stage-to-stage handover...?