Mazzoli: How fast are Linux pipes anyway?

When writing, luks/raid/integirty add far more complexity to the the request.

But even if you're not taking that into account, reads are by their very nature synchronous - the program MUST wait until it gets the information it wants, while a write can wait. So the elevator algorithm prioritises reads over writes ...

And then there's buffer(/cache)-bloat ...

Cheers,
Wol

The particular semantic that vmsplice() has over write() is that modifications to the buffer after making the call are not allowed. Or rather, they will show up in the recipient when they ought not to. We can't optimise write() to a pipe like this, because we'd need to shoot down TLB entries to prevent subsequent modifications to the buffer from showing up. That's why the new syscall was needed.

write() takes the address of a buffer that the user owns. Because it can not make any assumptions about how long that data is going to remain valid after the syscall, it has no choice but to copy. Because the memory was allocated ahead of time by the user, it can't really make any smart decisions about page size either. Then on read, it has to do the same thing again.

I guess for pipes specifically, one could imagine a flag which would make write() block until there is a corresponding read() call on the other side of the pipe, which would eliminate one copy. But with differing buffer sizes, non-blocking IO and other considerations I presume that'd be a lot of complexity that's unlikely to be worth it, especially considering the fact that you apparently can't even compute fizzbuzz fast enough to completely saturate a pipe like this. [1]

In that sense I'd say that the premise is wrong and write() is actually already plenty fast enough by default.

[1] In scenarios where this does matter there are usually already plenty of better, specialized solutions like mmap, XDP, dpdk, netfilter, etc.

There's a better command for that nowadays: `lscpu`. It's much easier than trying to make sense of /proc/cpuinfo (and much more reliable than cpuinfo's unfixable legacy format trying to describe modern CPUs; it incorrectly shows my desktop as having 12MB of L2 when there should be 6, and also omits the L1 and 64MB of L3 entirely).

In the original article, he achieved the speed by changing the problem from "transferring bytes via Linux pipes" to "transferring pages via Linux pipes". You can't move bytes at 35GiB/s purely locally within the same program, so claims of doing it over pipes makes you wonder what he's smoking. Turns the title involved a little poetic licence. Perhaps the article could have been better called "zero copy via pipes".

In case it tickles someone's interest: when you transfer pages, you are transferring ownership (permission to write) of the page. The HN discussion centred on how that transfer of ownership was managed. In the article it was dealt with by not handling it at all. This is just a benchmark program, and so he could get away with writing to the pages once before he started and then never writing them them again during the test. But that means he was transferring a constant over the pipes over and over again by vmsplice()'ing a page that never changed. The HN discussion revolved around whether it was possible to write to (ie, later reuse) the transferred pages in a way that didn't create a race between the reader and writer and didn't need to set up some second channel (eg, a futex) to avoid the race condition. It centred around transferring ownership of the page to the kernel. It hadn't reached a conclusion when I read it.

The HN discussion was more interesting that the article in some ways, because it explored whether what was really just a clever sleight of hand could ever be useful in practice.