Consider the resources required for decompression

Posted Jan 18, 2026 16:55 UTC (Sun) by jreiser (subscriber, #11027)
Parent article: Format-specific compression with OpenZL

In the 1970's a friend worked for a US company that designed and built large power transformers and related equipment, then shipped it overseas to Africa for use in the electrical grid of developing countries. I learned that the most important design factor was not volts, amperes, electrical phases, materials, or operating efficiency. Rather, the most imporant constraint was the width of the railroad tunnels between the receiving seaport and the installation. If the equipment could not be transported to the end destination, then nothing else mattered.

Analogously for lossless data compression, then the most important design constraint is the resources required for decompression. If the receiving environment does not have enough RAM for working storage, or enough ROM to store the decompression program, then the compression ratio does not matter. Perhaps a datacenter has vastly more than enough RAM and ROM, but many other environments do not. A child's toy, household appliance, home internet router, electric bicycle, industrial automoton, etc, often operate in resource-poor environments. Think of a 16-bit microcontroller with a total of 128kB of RAM and 256kB of ROM. Even a low-end cellphone with 3MB of RAM and 1MB of ROM can be a tight fit.

Because the output of OpenZL must name (or include) the schema, then there is an opportunity to approach this situation
by labeling each compressed output. "This data was compressed by version 1.2.3 of implementation FOO of Standard BAR. The reference decompressor requires 10MB of working storage and 38kB of code." For any compression system, it would be a great improvement if the constraints of decompression could be expressed as explicit parameters to the compressor. "For decompression, then my embedded device allows 20kB of RAM and 10kB of ROM. Please meet these constraints, or tell me how close you can come."

to post comments

Consider the resources required for decompression

Posted Jan 19, 2026 1:26 UTC (Mon) by excors (subscriber, #95769) [Link]

> If the receiving environment does not have enough RAM for working storage, or enough ROM to store the decompression program, then the compression ratio does not matter. ... Think of a 16-bit microcontroller with a total of 128kB of RAM and 256kB of ROM.

In that case (or even smaller), I've found Zlib is surprisingly good. The uzlib implementation takes about 2KB of ROM (on Cortex-M4), and about 1KB RAM + window size (max 32KB but you can probably reduce it to 16KB or 8KB without much impact on compression ratio). If you're storing compressed data in flash (e.g. when downloading a firmware update) then you want to minimise `code_size + uncompressed_size * compression_ratio`, and I suspect it's going to be hard to beat uzlib with a more sophisticated algorithm until you're getting up in the hundreds of KBs of data.

It would be nice if there were more algorithms (with size-optimised implementations) competing in that space, and benchmarks showing what's the best tradeoff in different ranges. It sounds like that's not what OpenZL is interested in though, since it's currently designed with a universal decompressor that will presumably be huge.

Consider the resources required for decompression

Posted Jan 23, 2026 6:44 UTC (Fri) by kmeyer (subscriber, #50720) [Link] (2 responses)

I don't think OpenZL is trying to be a solution for this increasingly irrelevant class of microcontroller. You can get relatively sophisticated ARM SOCs for cheap these days.

Consider the resources required for decompression

Posted Jan 23, 2026 9:12 UTC (Fri) by farnz (subscriber, #17727) [Link] (1 responses)

While the CPU core has become much better since then (typically a Cortex-M0 or a RISC-V thing, CPU core clock speeds below 40 MHz are now virtually non-existent), the cheap end is still very constrained for RAM and Flash.

Once you're looking at the under $0.25 per MCU range, 16 KiB flash is still common, but 2 KiB RAM isn't that uncommon - and these MCUs are remaining relevant because they're incredibly cheap in high volumes, use very little power and are still reasonably capable.

Consider the resources required for decompression

Posted Jan 23, 2026 11:54 UTC (Fri) by excors (subscriber, #95769) [Link]

Yes, I wouldn't personally care about 16-bit processors nowadays; but there's plenty of low-cost battery-powered IoT devices that can't afford megabytes of always-on DRAM, and I don't imagine energy efficiency will improve enough to change that any time soon. They'll probably have a (relatively) fast 32-bit core that can do plenty of sophisticated computation and data processing, but very little storage, so compression algorithms are both possible and useful.

(E.g. the "mainstream" STM32G0 series has 64MHz Cortex-M0+, and ranges from 8KB RAM / 32KB flash ($0.60 direct from ST) up to 144KB RAM / 512KB flash ($1.75).)

I wouldn't be surprised if this was becoming _more_ relevant over time, since many people see value in monitoring and controlling every device over the network, so they need more microcontrollers.