Speeding up Linux disk encryption (Cloudflare)

That was fun! Always love a good optimisation story :) The response from the dm-crypt maintainers was not fun at all, though:

> If the numbers disturb you, then this is from lack of understanding on your side. You are probably unaware that encryption is a heavy-weight operation...

Am I being a snowflake for reading this as just a little condescending?

It's interesting to see most of the problems were caused by various queues in I/O path. Buffers made sense for HDDs, but are impediments for contemporary storage. Biggest part of fix was ripping out the queues.
Once again bufferbloat is to blame, and once again storage layers are emulating what networking did earlier.

For benchmarking and debugging, it's probably worth having 'none' as one of the available ciphers, as long as nobody will get confused enough to pick it in production. Nowadays hardware-accelerated crypto is very fast, as the article notes, but it's still nice to eliminate that variable altogether when hunting a slowdown.

That's a great blog post to read. My main thoughts were:

That's what you get with one size fits all. The kernel is supposed to support everything from spinning disks, tiny 32 bit systems up to nvme, hundreds of gigs of RAM. It would be a miracle if all scenarios performed ideally. Still it performs well enough in many cases.

There is certainly happening a lot of bit rot in the kernel. The resources I have had at work have been always orders of magnitude smaller than what Cloudflare seems to have. Still we have identified similar problems, where 10+ year old code just doesn't work very well. With small resources all you can do is go away (not from Linux, but from a certain fs for example) or make a really dirty hack that you don't dare to show anybody else, even if it happens to work in your system.

Anyway, with Linux all these options exist, at Cloudflare scale and for the 0.3 person kernel teams. If you don't like it, go out, get/buy/write a kernel and report when you are happier :) I'm willing to listen, but I don't hold my breath until that.

Years ago doing a hours-long copy operation on a dm-crypt spinning disk, I found that setting the cpufreq governor to 'performance' instead of 'ondemand' sped it up significantly (on a CPU without AES-NI). I think that part of the time the CPU was doing encryption and part of the time the CPU was waiting on I/O, and the average CPU load was too low for the ondemand governor to use a higher frequency, even though overall throughput benefited significantly from a higher CPU frequency.

Cloudflare says they've been running this in production, but would it be safe for a random sysadmin to use? Assuming they have a good backup policy in place? :)

The patches look remarkably simple, to my untrained eye. I realize they're a bit kludgy and not a good long-term solution, but can anybody familiar with dm-crypt comment on whether the patches as-is are likely to have any stability / data corruption issues?