Merkle trees and build systems [LWN.net]

Merkle trees and build systems

Posted Jun 8, 2020 3:38 UTC (Mon) by pabs (subscriber, #43278)
In reply to: Merkle trees and build systems by mathstuf
Parent article: Merkle trees and build systems

This talk reminds me of how modern backup systems like restic and borg store filesystems; similar to git but without the commit hash chain (just independent snapshots) and with an additional layer of splitting files into chunks using rolling hashes.

Sadly the restic storage design misses out splitting directories into chunks of filenames, which means that there is some inefficiency around directories with many files in them.

I wonder when git is going to adopt the file chunking stuff.

Merkle trees and build systems

Posted Jun 8, 2020 12:19 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (1 responses)

Are they always stored as a single object then I assume? I wonder if statistics on how large directory blobs are in a repository could be made. I doubt they tend to approach normal chunk sizes often which means that, statistically, you're unlikely to find a chunk boundary in a directory blob in the first place.

Merkle trees and build systems

Posted Jun 8, 2020 12:30 UTC (Mon) by pabs (subscriber, #43278) [Link]

Some details are in this bug:

https://github.com/restic/restic/issues/2446