LWN: Comments on "Zero-copy TCP receive"

Zero-copy TCP receive

amarao — Thu, 26 Apr 2018 09:19:36 +0000

I may be naive, but I feel that hiding headers for any zero copy application is useless and adds compexity. Kernel can instead put a whole packet 'as is' into userspace memory and provide two 'data start/data end' pointers for each packet to help application to ignore headers.

Zero-copy TCP receive

sbates — Mon, 23 Apr 2018 02:18:53 +0000

Well I’d assume these proposed patches are for a variety of NICs from multiple vendors. The SolarFlare stuff is (I assume) specific to their NICs.

Zero-copy TCP receive

jmichels — Sun, 22 Apr 2018 12:51:21 +0000

Wouldn't this also have to out perform kernel bypass functionality offered by vendors such as Solar Flare?

Zero-copy TCP receive

cladisch — Sat, 21 Apr 2018 06:35:05 +0000

> Another possibility is just mapping the entire packet into the user-mode ring buffer

The FireWire driver does this for isochronous packets.

> and letting userspace skip over the embedded protocol headers

The FireWire host controller interface is standardized and must support scatter+gather, so the driver can instruct it to write the header words into another buffer so that only the actual data bytes end up in the mmap buffer. This requires that the header size is fixed and 32-bit aligned, so doing the same for a TCP/IP interface would require more flexible hardware support.

There is also a mode that dumps everything into the buffer, where the application has to parse out the packet metadata and headers.

Zero-copy TCP receive

epa — Fri, 20 Apr 2018 20:04:43 +0000

Could showing the header data to userspace conceivably introduce a security hole? It might be better to zero out those bytes (once the kernel has finished with them of course) before handing the page over to userspace.

Zero-copy TCP receive

ejr — Fri, 20 Apr 2018 16:55:47 +0000

Yeah, I'd expect something more along the lines of vmsplice(..., SPLICE_F_GIFT).

Zero-copy TCP receive

alkbyby — Fri, 20 Apr 2018 16:49:48 +0000

It is a little odd to see such unusual band controversial change that also introduces new API, to be merged so quickly.

Zero-copy TCP receive

quotemstr — Fri, 20 Apr 2018 15:08:27 +0000

> TCP is a little different though as you are seeing the abstraction. With I think tcp segment offload or the ingress equivalent I think it is very likely you will get the kind of packets needed in this case.

Another possibility is just mapping the entire packet into the user-mode ring buffer and letting userspace skip over the embedded protocol headers --- sort of as a hybrid between a conventional network stack and a user-space network stack.

Zero-copy TCP receive

quotemstr — Fri, 20 Apr 2018 15:06:26 +0000

Right. An ioctl could at least return a proper error code (say, EAGAIN), and there's precedent for an ioctl consuming data. There's no precedent for mmap being destructive!

Zero-copy TCP receive

post-factum — Fri, 20 Apr 2018 12:56:14 +0000

Would speculative page faults patchset be able to address the contention for the mmap_sem lock?

Zero-copy TCP receive

k8to — Fri, 20 Apr 2018 07:04:23 +0000

Ring buffers are pretty efficient for some patterns of data passing, but I don't see how you could do a zero-copy ring buffer.

Zero-copy TCP receive

ebiederm — Fri, 20 Apr 2018 03:46:04 +0000

It might be worth comparing this to how PF_PACKET sockets works with mmap.

Those I believe implement a shared ring buffer between kernel and user space. Something like that might be possible.

TCP is a little different though as you are seeing the abstraction. With I think tcp segment offload or the ingress equivalent I think it is very likely you will get the kind of packets needed in this case.

Doing anything more complicated (aka a ring buffer) I suspect would be quite a bit harder to implement and more fragile than what has been implemented here. As this sounds like it is just taking packets right out of the existing packet queue.

Zero-copy TCP receive

josh — Fri, 20 Apr 2018 02:47:20 +0000

The underlying memory map would still need to change each time, to avoid copying the underlying pages. That said, using mmap repeatedly for this does seem quite strange.

Zero-copy TCP receive

luto — Fri, 20 Apr 2018 02:26:15 +0000

Wouldn’t it be better (faster and saner) to mmap a magic region for zero copy reception and then use ioctl to materialize the data into it?