Zero-copy network transmission with io

Zero-copy network transmission with io_uring

Posted Jan 12, 2022 12:25 UTC (Wed) by al4711 (subscriber, #57932)
In reply to: Zero-copy network transmission with io_uring by smurf
Parent article: Zero-copy network transmission with io_uring

> What exactly is the question?

My question is what's the benefit of zero-copy data when the decrypt/encrypt step is in between.

Maybe I misunderstand the benefit, so please let me draw a picture.

client -> data -> nic -> kernel -> reading data and write data to nic buffer -> client

When we look now into the decrypt/encrypt step is this my understanding.

client -> data -> nic -> kernel -> server reading data -> decrypt/encrypt -> write data to nic buffer -> client

Could the ktls help in this case?

to post comments

Zero-copy network transmission with io_uring

Posted Jan 13, 2022 1:40 UTC (Thu) by neilbrown (subscriber, #359) [Link]

> My question is what's the benefit of zero-copy data when the decrypt/encrypt step is in between.

"Zero copy" is a marketing term. A more accurate term would be "reduced copy".
You might image an naive protocol stack where a copy happens when moving from each level to the next. Then the data is copied onto the network fabric, copied off into the destination, and copied back up the stack.

At any stage there is a potential benefit in avoiding the copy (and also a cost, so small messages are likely to be copied anyway).

Encrypt/decrypt may require a copy that would not otherwise be needed - though it may be possible to encrypt-in-place or encrypt-and-copy for one of the unavoidable copies (like copying onto the networking fabric). But that doesn't mean there aren't opportunities to reduce copying when encryption is used.

And also, encryption is not always used, even though it should always be available. On the open Internet, or in the public cloud, encryption is a must-have. In a private machine-room with a private network, there is minimal value in encryption, and there may be great value in reducing latency. In that case, it may be possible and beneficial to eliminate all the memory-to-memory copies ... particularly when an RDMA network fabric is used which allows the receiver to tell the sender when in memory to place different parts on an incoming message.

Zero-copy network transmission with io_uring

Posted Jan 13, 2022 13:55 UTC (Thu) by farnz (subscriber, #17727) [Link]

This does also reduce the number of copies when using kTLS. "Zero copy" is a bit of a misnomer - it's only there to eliminate memcpys from user owned memory to kernel owned memory, not all copies.

The point of "zero copy" is that in a normal transfer, data is copied from the user buffer to a kernel buffer, then the network card does DMA from the kernel buffer to its own transmit buffer. "zero copy" reduces that to a copy from the user buffer to the NIC's transmit buffer.

With kTLS, "zero copy" is a win with or without expensive NICs:

With expensive NICs, the NIC can do the encryption during DMA from CPU memory to the transmit buffer. You thus avoid copying the data into the kernel, and just have the NIC read and encrypt during DMA.
With cheap NICs, the kernel has to do a copy. Without zero copy, it copies plain text from user buffer to kernel buffer and then encrypts from kernel buffer to network buffer. With zero copy, it encrypts from user buffer to network buffer. In either case, the NIC will then DMA the network buffer into the on-chip transmit buffer.