Distributing filesystem images and updates with casync
Distributing filesystem images and updates with casync
Posted Jun 29, 2017 13:42 UTC (Thu) by nix (subscriber, #2304)In reply to: Distributing filesystem images and updates with casync by mezcalero
Parent article: Distributing filesystem images and updates with casync
Yeah. The rolling hash mechanism is an ingenious way to convert a hash or checksum (like, say, BuzHash or Adler32) into a content-sensitive chunk-boundary detector. It amazes me that, with this algorithm out there for decades, people are *still* writing deduplicators and other chunkers that just split into N-byte-sized blocks; a rolling hash is almost always a better way. (With a modern memory hierarchy it's nearly free, too -- you need to incur delays to get the thing you're working over into cache no matter what, and compared to that the overhead of rolling-hashing vanishes in the memory latencies.)
