Working with raw images on Linux

Most digital cameras are set to produce JPEG files; for many applications, such files are more than good enough. But most decent cameras support other formats, and a vendor-specific raw format in particular. The raw format contains something close to what was measured by the sensor, with a minimum of processing in the camera. These files are large, unwieldy, and in a proprietary format, which argues against their use in many situations. But, by virtue of holding the original image data, raw files give the photographer a much wider range of options later on. Much of the processing normally done in the camera (white balance, histogram adjustment, etc.) can be tweaked later on. For this reason, people who do photography for a living often prefer to record in the raw format.

Even for the rest of us, who have no hope of earning a living that way, raw files can keep creative options open. For people who like to play with HDR techniques there is an additional advantage: the camera typically record 12 to 16 bits of data for each channel - rather more than fits into a JPEG file. That, in turn, means that the dynamic range of raw files is significantly higher - assuming, of course, that the camera has a sensor which can meaningfully record data at that resolution. The extra range can be used to increase detail in images in a number of ways, including the use of tone mapping techniques.

Raw file formats are created by camera manufacturers, who generally feel no need to document their work. They will usually sell you a tool for decrypting their raw files - but, strangely enough, Linux support is usually missing from the feature list. Fortunately, the free software world benefits from the work of Dave Coffin, who has set a task for himself:

The result is dcraw, which comes awfully close to meeting that goal. It supports a huge list of cameras, and it does so at a high level of quality - arguably better than the vendor's tools. It is a command-line tool, aimed at batch operation or invocation from other programs; dcraw can be run from a gimp plugin, for example. Just about anything one wants to do with a raw image file is supported by dcraw.

The only downside is that processing raw images can be an interactive process. If one wants to make adjustments, a command-line tool can get tiresome after a while. The answer to that complaint is the UFRaw tool, which is built on dcraw. UFRaw allows adjustment of the white and black points, gamma curve, white balance and more - all with immediate visual feedback. When the desired result is achieved, it can be saved in a number of formats.

UFRaw is not perfect. It's one of those applications that thinks it's clever to remember where the last image was stored and put the next one in the same directory. Your editor, instead, expects programs to default to the directory they were started in, or, failing that, to the directory where the source file was found. It's aggravating to save a file then have to figure out where the application decided to put it. UFRaw is doubly obnoxious in this regard because it immediately exits after saving the file. The non-resizeable window is also annoying. One assumes these little difficulties can be dealt with eventually; meanwhile, the core functionality is good stuff.

What sort of results can one expect? Here are three versions of the window view photo featured in the HDR article:

Original	UFRaw edited	Tone mapped

(See this page for larger versions of the pictures).

Some quick editing with UFRaw was sufficient to bring out a fair amount of detail in the plant in the foreground - though the background lost some contrast as a result. The tone-mapped photo does better at maintaining contrast throughout the frame. The end result is not as complete as the full HDR image (visible here), but it does show that raw files contain information which can be recovered later on to improve the picture. Taking a single raw image is much easier than the full bracketed HDR technique, and it allows tone mapping techniques to be used on subjects which stubbornly refuse to stand still for a few minutes while several shots are taken.

One thing worth noting in conclusion: we should not take our ability to work with raw images for granted. Vendors like Nikon and Sony are known for encrypting their raw formats. The language they use to justify themselves will look most familiar; consider this advisory from Nikon regarding its NEF format:

In other words, photographers are being locked out of their own images for their own benefit. All of the usual counterarguments apply here; photographers might just have their own idea of where there benefit lies. And what happens to those raw images a decade or two from now, when the vendor has long since ceased to support the format and, even if one can find one's single legal backup copy of the software, it refuses to run on currently available systems? Fortunately, we have dcraw, which will document the reading of these formats indefinitely.

So far, vendors' attempts to encrypt raw files have been broken in short order. Chances are that trend will continue. But there is little difference between breaking into a raw image file and turning off the copy protection bits inside a PDF file. The stage is clearly set for an ugly battle, probably involving the DMCA, when some vendor decides to turn nasty.

Photographers have been worried about this issue for a few years now; efforts like the OpenRAW project have been working, with little success, to get camera manufacturers to open up their formats. Adobe has been pushing its Digital Negative format as a standard; it would be a step in the right direction, but this format still has mechanisms for the embedding of vendor "private" information. At this time, there does not seem to be a clear solution in sight. We must deal with cameras just like we deal with many other types of hardware: we have to figure out how it works ourselves.