| From: |
| Bobby Eshleman <bobbyeshleman-AT-gmail.com> |
| To: |
| Donald Hunter <donald.hunter-AT-gmail.com>, Jakub Kicinski <kuba-AT-kernel.org>, "David S. Miller" <davem-AT-davemloft.net>, Eric Dumazet <edumazet-AT-google.com>, Paolo Abeni <pabeni-AT-redhat.com>, Simon Horman <horms-AT-kernel.org>, Andrew Lunn <andrew+netdev-AT-lunn.ch>, Gerd Hoffmann <kraxel-AT-redhat.com>, Vivek Kasireddy <vivek.kasireddy-AT-intel.com>, Sumit Semwal <sumit.semwal-AT-linaro.org>, Christian König <christian.koenig-AT-amd.com>, Shuah Khan <shuah-AT-kernel.org> |
| Subject: |
| [PATCH net-next v3 0/4] net: devmem: allow rx-buf-size > PAGE_SIZE per binding |
| Date: |
| Fri, 12 Jun 2026 09:25:56 -0700 |
| Message-ID: |
| <20260612-tcpdm-large-niovs-v3-0-a3b693e76fcb@meta.com> |
| Cc: |
| netdev-AT-vger.kernel.org, linux-kernel-AT-vger.kernel.org, dri-devel-AT-lists.freedesktop.org, linux-media-AT-vger.kernel.org, linaro-mm-sig-AT-lists.linaro.org, linux-kselftest-AT-vger.kernel.org, sdf-AT-fomichev.me, razor-AT-blackwall.org, daniel-AT-iogearbox.net, almasrymina-AT-google.com, matttbe-AT-kernel.org, skhawaja-AT-google.com, dw-AT-davidwei.uk, Bobby Eshleman <bobbyeshleman-AT-meta.com> |
| Archive-link: |
| Article |
Every devmem dmabuf binding hands the page_pool PAGE_SIZE niovs today.
On NICs that consume one descriptor per netmem, this caps a single RX
descriptor at PAGE_SIZE and burns CPU on buffer churn.
In this series, we add a bind-time netlink attribute,
NETDEV_A_DMABUF_RX_BUF_SIZE, that lets userspace request a larger niov size
(power of two >= PAGE_SIZE). Drivers must opt in via
queue_mgmt_ops.QCFG_RX_PAGE_SIZE.
Selftests use udmabuf, but udmabuf sgtables were previously hardcoded to
PAGE_SIZE. This series modifies udmabuf to respect folio sizes in its exported
sgtable. The result is that when backing udmabuf with MFD_HUGETLB 2MB pages,
the sgtable is populated with 2MB entries, allowing devmem's gen_pool to carve
out large (eg. 64K) niovs.
Measurements
------------
Setup: kperf devmem RX/TX cuda, 4 flows, 64 MB messages, 60s, dctcp,
num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov size,
mlx5.
niov RX dev Gbps RX flow avg Gbps app sys %
----- ---------------- ----------------- ----------------
4K 300.63 +/- 53.21 75.16 +/- 13.30 54.15 +/- 10.23
16K 321.35 +/- 28.20 80.34 +/- 7.05 41.05 +/- 8.87
32K 347.63 +/- 2.20 86.91 +/- 0.55 44.54 +/- 3.51
64K 332.11 +/- 14.26 83.03 +/- 3.56 35.47 +/- 3.11
RX app sys % drops ~19% from 4K to 64K.
kperf support (not yet merged):
https://github.com/facebookexperimental/kperf/commit/8837...
Signed-off-by: Bobby Eshleman <bobbyeshleman@meta.com>
---
Changes in v3:
- fix a bunch of non-reverse christmas tree declarations (Stan)
- remove extra uint32 cast for getpagesize() (Stan)
- remove overzealous strtoul checking (Stan)
- remove value checks that the kernel already performs on rx_buf_size
(Stan)
- Link to v2: https://lore.kernel.org/r/20260611-tcpdm-large-niovs-v2-0...
Changes in v2:
- Use NL_SET_ERR_MSG_FMT for sg alignment failure details (Stan)
- Keep -E2BIG (not a direct ask, but seemed preferred, Stan)
- Update udmabuf commit message and comments explaining why
"one sg ent per folio" is useful (Christian)
- Set/restore nr_hugepages in py harness (Stan)
- Link to v1: https://lore.kernel.org/r/20260603-tcpdm-large-niovs-v1-0...
---
Bobby Eshleman (4):
net: devmem: allow rx-buf-size > PAGE_SIZE per dmabuf binding
udmabuf: emit one sg entry per pinned folio
selftests/net: ncdevmem: add -b option to set rx-buf-size on bind
selftests/net: devmem.py: add check_rx_large_niov
Documentation/netlink/specs/netdev.yaml | 8 +++
drivers/dma-buf/udmabuf.c | 52 +++++++++++++++++--
include/uapi/linux/netdev.h | 1 +
net/core/devmem.c | 51 +++++++++++--------
net/core/devmem.h | 13 +++--
net/core/netdev-genl-gen.c | 5 +-
net/core/netdev-genl.c | 19 ++++++-
tools/include/uapi/linux/netdev.h | 1 +
tools/testing/selftests/drivers/net/hw/config | 1 +
tools/testing/selftests/drivers/net/hw/devmem.py | 12 ++++-
.../testing/selftests/drivers/net/hw/devmem_lib.py | 58 +++++++++++++++++++++-
tools/testing/selftests/drivers/net/hw/ncdevmem.c | 36 ++++++++++++--
.../testing/selftests/drivers/net/hw/nk_devmem.py | 11 +++-
13 files changed, 225 insertions(+), 43 deletions(-)
---
base-commit: 518d8d0199538a4d6d5e51064044ece71e0c42e7
change-id: 20260602-tcpdm-large-niovs-56523a3a1077
Best regards,
--
Bobby Eshleman <bobbyeshleman@meta.com>
