Weekly Edition
Return to the Development page
| |||||||||||||
ImageZero for lossless photo compression
Christoph Feck has announced
the first release of ImageZero, a lossless photo compression tool.
"Being twice as fast as PNG when decompressing (and more than 20
times faster when compressing) it achieves compression ratios that are near
or better than PNG for natural photos, sometimes even better than JPEG-LS
for very high quality photos." The code is available on gitorious. (Thanks to Paul
Wise).
(Log in to post comments)
ImageZero for lossless photo compression Posted Jan 30, 2012 14:56 UTC (Mon) by sztanpet (subscriber, #60731) [Link]
Looks quite nice, if seemingly a bit niche (targeting only photographic images.)
https://kdepepo.wordpress.com/2012/01/30/fast-lossless-co...
ImageZero for lossless photo compression Posted Jan 30, 2012 23:22 UTC (Mon) by andreasb (subscriber, #80258) [Link]
That's a somewhat peculiar way to put it. Photographic images is the same niche that JPEG targets, still JPEG is pretty popular…
ImageZero for lossless photo compression Posted Jan 31, 2012 11:28 UTC (Tue) by man_ls (subscriber, #15091) [Link] Photographic images and lossless compression is somewhat of a niche. Lossless JPEG is not all that popular, even on high-end cameras -- they all seem to use a RAW format with little or no compression. It would be nice if they could all use the same standard; but as explained in the fine comments below, this format does not seem to supply the additional features required: more than 8 bits per channel or storing sensor information instead of combined pixels. Not to speak about camera-specific features as geometry or distortion correction.
ImageZero for lossless photo compression Posted Jan 31, 2012 18:12 UTC (Tue) by rahvin (subscriber, #16953) [Link]
The RAW format manufacturers use is far more than just uncompressed images, it's the direct output from the image sensor & processor on the camera including all the raw data before white point and a dozen other digital manipulations are applied. Some versions are even in a compressed format. They are larger than uncompressed images precisely because there is more information stored, the camera's processor throws away a LOT of data.
Digital imaging is hard because the digital sensors don't see like people or even film stock (color space and even interpretation of the color can be completely different, more sensitive in some cases and less sensitive in others), as a result several important manipulations are done on the raw sensor output when generating a JPG, what the RAW format does is preserve the original sensor data so that as an example, if the camera miscalculates the white point in the image it's possible to take the RAW image and apply another white point to the original data without a lossy compression in the way borking up the resulting image. For me at least about 1/3 of the pictures I take with my early model DSLR have a warped white point (typically bluer than it should be) when compared against a uniform source.
I too would prefer to see a uniform RAW format, but they might simply be hindered by the different sensors requiring the storing of different data because it is the raw output from the sensor.
ImageZero for lossless photo compression Posted Jan 31, 2012 19:16 UTC (Tue) by man_ls (subscriber, #15091) [Link] Yes, and sometimes the camera even applies geometric transformations to correct lens aberrations. I remember a friend's Panasonic GF1: the same picture in JPEG showed a straight horizon line, in RAW it was clearly curved. Sometimes the transformations are different for every color, therefore correcting chromatic aberrations. This means the firmware was aware of the specific lens characteristics and corrected it for the JPEG!
ImageZero for lossless photo compression Posted Jan 31, 2012 19:42 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
don't try for a uniform RAW format, that will only limit the capability of future devices that may have information that doesn't exist today.
Instead try for a documented, and identifiable RAW format from every manufacturer. Then you can have projects like dcraw that can manipulate the data.
ImageZero for lossless photo compression Posted Jan 31, 2012 23:50 UTC (Tue) by man_ls (subscriber, #15091) [Link] At least a common step for compressing image data (even if the "image data" can be anything from image pixels to a row of sensors of non-square dimensions) might be useful, I guess.
ImageZero for lossless photo compression Posted Feb 1, 2012 0:09 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
even defining a common compression method would prevent you from using a better one later on.
or you make the format 'extensable', meaning you include tags in there to indicate versions and if there is a new version, you won't know how to read it.
this gets you back to exactly the same place as having all sorts of different versions of the format from the manufacturer, you just squash them all together with version id's in one file. But now you have to worry about coordinating the versions between all the different manufacturers.
or you make one tag be a manufacturer tag, and have everything keyed off of that (so that version 55 from manufacturer A is different from version 55 from manufacturer B), but at that point, what is your advantage over just letting them be different?
ImageZero for lossless photo compression Posted Feb 1, 2012 0:18 UTC (Wed) by man_ls (subscriber, #15091) [Link] You are over-thinking the whole thing: you have convinced me already that a common RAW format is not a good idea. I just suggest that this format might be a useful arrow in a camera manufacturer's quiver, as a step to apply to raw image data in their RAW algorithms. It might even simplify the public RAW format specification (if your wish came true), as one piece of the puzzle would already be widely known.Nothing would prevent the manufacturer to change the format (including the compression) for their latest and greatest camera.
ImageZero for lossless photo compression Posted Feb 1, 2012 2:50 UTC (Wed) by andreasb (subscriber, #80258) [Link]
> don't try for a uniform RAW format
You don't have to try, Adobe's DNG is already here. I use my camera's native raw format, but AFAIK the DNG output would be fully equivalent content and feature wise. The native format may have slightly better compression (lossless of course) of image data than DNG, if I remember right.
ImageZero for lossless photo compression Posted Feb 2, 2012 15:32 UTC (Thu) by gidoca (subscriber, #62438) [Link] Yes, but DNG doesn't support all camera sensor types on the market, e.g. Sigma's Foveon X3, see here.
probably won't go anywhere Posted Jan 30, 2012 15:15 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
Lacks a really compelling use case. It's a little faster, it can offer a little better compression ratio sometimes, it may not trip over patents (well, it doesn't deliberately implement anything famously patented). Doesn't do anything that was previously impossible. It's just not enough to overcome the inertia.
Another issue is that if you're handling raw photographs, rather than photographic images that are "finished" (for which JPEG will often be adequate) then you probably want "raw" data rather than RGB images. The "raw" data reflects the many curiosities of physically building image hardware devices, as opposed to a theoretical abstraction where you get a bunch of RGB pixels ready for display. If this new format isn't suitable for compressing "raw" data, then it seems to fall between two stools, on the one hand unsuitable for final publication (existing widespread deployment of JFIF JPEG wins and PNG steps in on the rare occasion that's not adequate) and on the other not really a complete replacement for professional photographers (doesn't preserve everything from the "raw" image).
RE: probably won't go anywhere Posted Jan 30, 2012 15:42 UTC (Mon) by dnewcomb (subscriber, #3312) [Link]
I would be interested if this could be adapted to multiband ( 4 or more) geospatially referenced images. The current commonly used compression options are compressed tiffs ( packbit,lzw,jpeg) , or the proprietary ecw or MrSid formats.
RE: probably won't go anywhere Posted Jan 31, 2012 18:18 UTC (Tue) by rahvin (subscriber, #16953) [Link]
There are far more than 4 Geo-referenced raster formats. Heck you didn't even list the most popular these days, Geo-referenced JPG, it's handled the same was as GeoTIFF in that there is a world file called a .jpw with the same name as the JPG file.
In fact with Geo-referenced JPG and TIFF I question the need for anything else.
probably won't go anywhere Posted Jan 30, 2012 17:59 UTC (Mon) by LightDot (guest, #73140) [Link]
Twice as fast as PNG when decompressing and more than 20 times faster when compressing is *quite* a bit faster than png, not a little faster. If that's actually true, of course, and doesn't cause other downsides.
That being said, I agree that any established format has a large inertia and might be hard to dethrone. Still, this sounds blazingly fast.
probably won't go anywhere Posted Jan 30, 2012 18:29 UTC (Mon) by boudewijn (subscriber, #14185) [Link]
I'm really interested in this work. It's little-known, but the interchange format that links MyPaint, Gimp and Krita together, openraster, uses png files embedded in an uncompressed zip container for every layer. This format could potentially make applications handle ora much, much faster. It might also be really interesting for the native file format of Krita.
probably won't go anywhere Posted Jan 30, 2012 20:16 UTC (Mon) by jmalcolm (guest, #8876) [Link]
That sounds like a nice use case. The author mentioned that being a backing store for imaging processing apps was one of the natural places for this tech.
probably won't go anywhere Posted Jan 31, 2012 23:47 UTC (Tue) by man_ls (subscriber, #15091) [Link] Does it support more than 8 bits per channel? I haven't seen a clear answer on the linked page.
video? Posted Jan 31, 2012 7:15 UTC (Tue) by eru (subscriber, #2753) [Link] I wonder if the speed might make the method interesting for loss-less compression of video? Such a format could be useful during editing.
probably won't go anywhere Posted Jan 31, 2012 7:32 UTC (Tue) by spaetz (subscriber, #32870) [Link]
> Lacks a really compelling use case. It's a little faster, it can offer a little better compression ratio sometimes,
While it may not be suitable to replace a graphic format due to network effects anytime soon. I can see how a self-contained use as e.g. a thumbnail database (think Gnome) where I want to create 1000 thummbails of my picture folders fast, or a storage backend for an image processing software would work out nicely.
"20 times faster when compressing" becomes really interesting when you talk about large amounts of images.
probably won't go anywhere Posted Jan 31, 2012 8:14 UTC (Tue) by PO8 (guest, #41661) [Link]
First place I'd think of using a weird lossless format with 20x compression speed is as internal storage for digital cameras. The problem there is reading the pixels out of a giant sensor and into a small flash image store quick enough to be ready for the next picture. If the compressor works well on raw images (see comments above) you could store them internally in this and then convert to PNG or JPEG on download from the camera.
There are a host of similar applications, e.g. image/frame compression on tiny ultra-low-power surveillance devices connected to ultra-slow networks. I've played with stuff like this, and right now I know of nothing that is both very CPU-light and does a reasonable job of compression (better than, e.g., souped-up RLE schemes).
All of that said, I'm still waiting either for a good description of the algorithm or until I get time to read the source. To my mind, an interesting question is whether the design and coding techniques used in IZ can also lead to dramatic speedups over current implementations when applied to PNG.
ImageZero for lossless photo compression Posted Jan 31, 2012 5:18 UTC (Tue) by PaulWay (✭ supporter ✭, #45600) [Link]
I must be thick, or missing something - I can't see where the actual compression is done. I can see that we go through transforming the image into the predicted values of each pixel based on some previous one; I can't see how that saves any bits anywhere. Any pointers on how this compression actually works would be appreciated :-)
Have fun,
Paul
ImageZero for lossless photo compression Posted Jan 31, 2012 7:53 UTC (Tue) by price (subscriber, #59790) [Link] Another version of the announcement suggests that it uses Huffman coding. That's commonly used in compression to take a bunch of smallish numbers (such as a clever prediction scheme might produce) and then transmit them in a correspondingly small number of bits each.
ImageZero for lossless photo compression Posted Jan 31, 2012 10:25 UTC (Tue) by PO8 (guest, #41661) [Link]
After a quick read of the IZ compression code, and checking the PNG spec, it sure looks to me like the IZ compressor code could easily be turned into a PNG compressor with no loss of performance (and the IZ decompressor code inserted as a fastpath in a PNG decompressor to handle the case of IZ-compatible compression). You'd have to either implement a fast version of standard Deflate LZW compression or define the compression IZ uses (apparently Huffman with a fixed table?) as a new compression type (there's a byte in PNG for this). The filtering looks offhand just like the PNG "Average" predictor, so that shouldn't be a problem.
Seems like it would be better to be compatible with existing decoders, and it doesn't look like it would cost anything.
I'm guessing the speedup comes from a combination of two things: the clever branchless implementation and using small-table Huffman coding vs large-cache LZW. Both are optimizations for modern processors. It's possible that by cleverly reimplementing the inner loop of a PNG engine in the style of the IZ code, and by strongly limiting the cache size of the LZW compressor in the encoder, you could get an existing full PNG engine to achieve comparable results.
ImageZero for lossless photo compression Posted Feb 2, 2012 5:17 UTC (Thu) by josh (subscriber, #17465) [Link]
Sounds like an interesting student project. :)
|
|||||||||||||