Merkle trees

Just be careful about your block size. If the data you need to dedup is offset by 1, 2, ... blocksize-1 bytes then you're out of luck. The larger the block size the less likely you will be able to deduplicate more -- I think this is a corollary to Zipf's law. Yet higher block sizes reduce overhead of storing the merkle tree. Could still be interesting and worthwhile though!