Shoot two exposures at once with Magic Lantern

Developers from the Magic Lantern (ML) project have enabled a previously undiscovered capability in high-end Canon digital cameras: the ability to record images and video at two different exposure levels in every frame. Post-processing is required to blend the exposures into a seamless image, but the result is a single shot that covers essentially the entire range of brightness that the camera sensor is capable of seeing. That means high contrast scenes are visible without sensor noise, without motion artifacts, and without stacking multiple shots in a high dynamic range (HDR) blending program.

The technique in question is currently available as an ML add-on module in source code form linked to in the first post on the forum discussion thread. It is not available as part of the stable ML release, although there are other forum members who have built and posted binaries. A separate utility (currently Windows only, but usable with Wine) is required to convert the camera output into a standard image or video file.

Sensors and sensibility

ML began as an effort to implement extra features for Canon's EOS 5D Mark II cameras—specifically, features to improve the camera's usability when shooting video. Subsequently, the project has taken on creating builds for other camera models with similar internal chip architectures, such as the 5D Mark III and the 7D, and developers have crafted modules that extend the cameras' functionality in ways beyond user interface improvements. The 5D Mark III, in particular, has gained some significant functionality through the ML team's work. For example, back in May the project implemented a raw video recording mode that allowed users to film uncompressed video (at 24 frames per second) at greater-than-1080p resolutions.

Canon's factory firmware shoots video at 1080p, tops, in H.264 compressed format. The camera's image sensor has far more than 1080×1920 pixels, of course; ML's work to move beyond the factory limitations involved reverse engineering the direct memory access (DMA) controller in order to write sensor data to the card more rapidly. As part of that process, ML began mapping out the camera's registers. Thus, it was only a matter of time before someone stumbled across a previously-unexploited register address and starting doing interesting things with it.

ML developer Alex "a1ex" Dumitrache discovered one such register on the chip that controls the CMOS sensor's "ISO rating" (which emulates the light sensitivity of different film speeds). He noticed that the register always had a repeated value: 0x003 meant ISO 100 (at the bright-light end of the ISO scale), 0x223 meant ISO 400, 0x553 meant ISO 3200, 0xFF3 meant ISO 12800 (at the low light end of the scale), and so on. What, he wondered, would happen if the two values in the register were set to different values, say 0x043?

The answer, it turns out, is that half of the sensor's scan lines will be read at ISO 100, and the others at ISO 1600. These lines are interlaced in pairs, with lines 0 and 1 at one ISO setting followed by 2 and 3 at the other setting, and so forth. This pairing is because the sensor's red, green, and blue pixels are arranged in pairs of lines; a two-by-two square of four pixels is required to get a full RGB triple for any one sample point. The reason that this interlaced setting is possible is that the 5D Mark III has an eight-channel sensor readout, where most Canon cameras have a four-channel readout. The 7D model also has an eight-channel sensor readout, so Dumitrache was also able to perform the dual-ISO sensor setting on that camera. So far, those are the only two camera models to support the feature.

Dumitrache wrote up the finding in a white paper [PDF] that goes into considerable detail on what happened next. Saving the half-and-half exposure frame as a file was no trouble, but it took far more work to figure out how best to interpolate the data in a meaningful way. If the two ISO settings are significantly different (as they presumably would be for the stunt to be worth doing), then an object that looks properly-exposed in one would look way too bright or too dark in the other. The darker exposure has less sensor noise, but the brighter exposure has better saturation and color fidelity. In addition, the two exposures are recorded with mismatched white points and black levels, and despite the nice round numbers of ISO ratings, the CMOS image sensor does not respond in simple ways to exposure changes.

He eventually worked out an algorithm to mix pixels from both exposures in the image midtones, and transition smoothly to a single exposure each for the shadow and highlight areas. This initially meant that highlights and shadows were at half the vertical resolution of the midtone parts of the image, with existing lines doubled to fill in for the missing ones. Naturally, this line doubling can create image artifacts (especially in areas like high-contrast edges).

Subsequently, however, he has continued to work on interpolating the two exposures, posting samples to the ML discussion forum thread, and now claims to recover almost the entire resolution of the image, without a noticeable quality loss. There are a variety of sample images in the discussion thread that showcase how the technique compares to a single-ISO exposure. A particularly good example is Jay Birch's photo. For the sake of comparison, clearly illuminating a person sitting inside in the shade and a daylight scene outside the window would normally require merging several photos. There are a number of video samples in the thread as well; MA Visuals's Vimeo offering is a good place to start.

Usage and impact

For now, Dumitrache's command line conversion tool cr2hdr is required to convert the dual-ISO frames into Digital Negative (DNG) format. The tool is licensed as GPLv2+, but it is still under development. So far the blending algorithm is hardcoded in; as he makes adjustments he updates the posted version of the file, but there are no user-tweakable switches.

The frames are recorded to the storage card with the .CR2 file extension used by the cameras for normal shots, but they obviously deviate quite a bit from the "correct" CR2 format. Once converted, stills can be opened in any raw photo tool that supports DNG, which includes most of the free software applications (such as UFraw, Rawstudio, or Darktable). Video editing may be trickier; at least, no one on the ML forum appears to have attempted any dual-ISO video on Linux.

Using the dual-ISO module on the camera is straightforward: one sets the "base" ISO on the camera's physical dial, and the second (or "recovery") ISO in the dual-ISO menu added by the module. The technique does not work with the camera's "Auto ISO" mode. But the big plus is that a shot taken in dual-ISO mode is captured all at once. There are other ways to take two exposures and blend them together, but they come with serious trade-offs.

For example, ML can automatically "bracket" shots: record two sequential frames at different exposures, so that they can then be blended together with a software tool like Hugin or Luminance HDR. ML can even record "HDR video," which it does by taking alternate frames at different exposure levels, then merging them together in pairs.

The drawback of both of these options is that the two exposures merged are taken at different times—only slightly different times, admittedly, but still different. Blending them when they are not perfectly aligned costs image detail. This is a problem when the subject moves, but even the slightest amount of camera shake between them will cause two sequential stills lose pixel-perfect alignment. Thus, even though the ML dual-ISO shot sacrifices some scan lines in the extreme tones of the image, the two-shot approach loses resolution when the images are merged as well. For video, the problem is even worse, since a moving object will introduce jitter when the sequential frames are merged. There are other products on the market (such as the top-dollar RED digital cinema cameras) that can shoot "HDR video", but they too merge sequential frames.

In addition, Dumitrache contends that the dual-ISO blending algorithm in his command-line tool produces more natural results than merging two bracketed shots anyway. Almost everyone is familiar with the "radioactive" look of poorly-merged HDR conversions; the dual-ISO approach does not suffer from that flaw. In part, Dumitrache is simply confident that his blending algorithm produces nicer results than many tone-mapping alternatives, but the math is different, too: his does not have to blend any overlapping pixels. Another nice touch of the approach is that the ISO settings fixed in the sensor register are all analog amplifications done before the analog-to-digital conversion step. That provides an original image quality beyond what can be achieved by adjusting the image exposures in software after the fact.

The practical questions for interested photographers to ask are when taking a dual-ISO shot makes sense and what settings to use. The 5D Mark III's sensor is reported to have a 14-stop dynamic range (i.e., total light-to-dark range), but any one ISO setting only captures a portion of that. Shooting at ISO 100 grabs almost 11 stops, with higher ISOs performing worse and worse. To get the other three stops missed by ISO 100, Dumitrache recommends shooting a recovery exposure of ISO 1600. At higher speeds, there is not any more dynamic range to squeeze out of the sensor, and there is noticeably worse rendition of the midtones.

As to what pictures to take, there are certainly a handful of obvious scenarios in which a dual-ISO image is sure to result in a better photo than a single exposure (one might note, for instance, that almost all of the sample images in the forum thread involve an interior room next to a bright window). But if the original scene doesn't have 14 stops of dynamic range in it, trying to capture them is overkill. That does not detract from the practical value of this new feature, however: the ML team has come up with a remarkable new feature, one that—so far—no other product has matched, free software or otherwise.

It would be nice to see support for these dual-ISO exposures in the various open source photo and video editing applications, but it is not yet clear how feasible that is. On the one hand, ML is a niche project for aftermarket features only. But on the other, if the ML project keeps up this kind of development, it might not stay a niche project for very long.