Integration into file formats.

Posted Jan 15, 2026 11:50 UTC (Thu) by Wol (subscriber, #4433)
In reply to: Integration into file formats. by himi
Parent article: Format-specific compression with OpenZL

> magic in the storage layer to handle that kind of deduplication would definitely be nice.

Isn't this inherent in one the file-systems? ZFS springs to mind?

Some filesystems I believe keep a hash of disk blocks, and if two blocks have the same contents, the overlying files will be changed to point to the same block. Within this, they can either "check on write" and so dedup on the fly, or they do a post-hoc dedupe pass. Either way, I'm sure this functionality is available in at least one regular linux file system.

Cheers,
Wol

to post comments

Integration into file formats.

Posted Jan 15, 2026 14:51 UTC (Thu) by willy (subscriber, #9762) [Link]

This is the kind of thing that sounds seductively attractive and then you actually try to do it and the metadata needed to keep track of everything blows up exponentially (literally, not in the modern meaning of "a lot"). And fragmentation increases massively, which turns out to matter even on NVMe drives.

There's specialist cases where this makes sense, but it's no free meal. Or maybe it is a free meal, in the sense that the drinks now cost 50% more.