Integration into file formats.
Integration into file formats.
Posted Jan 15, 2026 2:19 UTC (Thu) by himi (subscriber, #340)In reply to: Integration into file formats. by jepsis
Parent article: Format-specific compression with OpenZL
You can do something similar as it stands with existing compression libraries, but it's a lot of work for not much gain over using a general tool for whole-file compression. What the OpenZL project brings is a body of knowledge about data compression in general that can be used to inform the way that you set up your data streams to allow the best possible results, and a bunch of code that makes it easy to create a highly specialised compression pipeline - if that gets you something two thirds the size of the old '.foo.gz' files that can be compressed and decompressed in half the time it may well be worth the effort.
The body of knowledge could also feed more broadly into file format design choices - if laying your data out one way versus another costs you (say) 10% in terms of zstd compressed file size, that's kind of useful to know even if you're not going to try and make a super-specialised compression tool. As far as I know that sort of knowledge base doesn't exist at present.
