Development

Raw photo manipulation with Darktable

The Darktable project released the latest update to its photo editor on November 8. Version 0.9.3 is a minor update to the stable 0.9 series, but I had not spent much time with the application since the 0.7 cycle, so I took a fresh look. There are few surprises in this series; the developers continue to add features designed to enable photographers to develop creative images. As in past releases, there is particular emphasis placed on darkroom styles and techniques that hearken back to pre-digital days — which gives Darktable its own niche, but may not be what the majority of digital camera owners are looking for.

Defining Darktable

Darktable is a self-described "photography workflow application" and raw converter. Even within photography, the term workflow application means different things to different people. To some it means using a single application to perform every step from importing images from the camera, to sorting and editing them, to exporting final images for print or online publication. To others it just means the process of sorting, grading, and batch-processing images. To still others, it is little more than a catchword to draw a line between "professional" and "hobbyist" class programs.

Darktable includes camera offloading and image collection management features, so it can handle most if not all of the steps a photographer might use to turn a shooting session into a finished product (in fact, it is one of the few open source applications that can do tethered shooting using a USB-attached digital camera). Nevertheless, it places far more emphasis on the editing and creative interpretation features than on other steps of the process, which makes it feel like more of a super-powered raw editor than anything else. By comparison, Digikam has significantly more features on the collection-management front, including geolocation, facial recognition, and powerful search capabilities. Darktable will import your image metadata into an internal database, but provides only a rule-based filter to narrow down how much of your collection is visible in the "light table" view at once. Currently Exif is the only metadata format supported, while many other open source tools are now also supporting XMP.

It also provides little in the way of batch-processing, which I consider to be a key workflow feature. In other raw editors, such as Rawstudio, you can mark individual images for batch export as you edit, and when finished start the processing queue with one click. Currently in Darktable, you still need to return to the light table view, select the images you wish to export with the cursor, and use the "export selected" widget. You can work around this restriction slightly by using the tagging or star-rating features to flag the images you wish to batch process later, but that limits your ability to use tags or ratings for other purposes.

Even on fast systems, the multiple sets of image transformations required to export each photo can add up with even a small handful of high-resolution images. Being able to queue these operations for later — ideally even with distinct export parameters and destination folders — is a time saver. Darktable, like most raw converters, shows you a preview image of each correction you make in the editor, but it does not perform the full-size operations until file export time.

Still, none of that is to say that Darktable is not a capable application — it just provides more power to the image-creation pipeline than it does to other steps. But if you have a very large collection of photos to manage, you will probably prefer to use Digikam to juggle them until Darktable beefs up its suite of library-management tools.

The grand tour

I have not found an explicit history of the application's name, but it is clearly a portmanteau of darkroom and light table, which are its two primary modes (and, of course, the inverse of Adobe's "Lightroom"). Light table mode is used to sort images, including importing them from camera or disk (thankfully, Darktable does not attempt to copy images already on disk into its own special folder, rather it scans them and ingests the metadata for its database), tagging and editing image metadata, and exporting to common output formats like TIFF and JPEG. It can export in 16-bit-per-channel mode to formats that support it, and includes support for the OpenEXR high dynamic range file format, as well as direct export to Flickr, Picasa Web Albums, and HTML galleries.

Minimal editing capabilities (such as rotation and applying pre-defined "styles") are available in the light table mode; all serious editing is done in the darkroom mode. Both modes use a custom dark, neutral-gray UI theme rather than the system GTK+ color scheme. This is a growing trend among photo and creative applications, enough so that GTK+ implements a prefer-dark-theme property that applications can use to automatically request a dark variant. Darktable implements its own color scheme, however.

Switching to darkroom mode is done by double-clicking on an image thumbnail or by clicking on the mode text-label in the upper-right corner. This is where Darktable begins to diverge from the UI conventions followed by most other raw editors. The others tend to provide a long, scrollable column of image-adjustment tools down the left- or right-hand side of the editor, and you can adjust any of the settings exposed or leave them as-is. Darktable uses a plugin system instead; each of the adjustment tools is a separate plugin that you must activate in order to manipulate it.

The available plugins appear in a grid at the bottom of the right-hand column; when activated each plugin's controls appear in the main tool strip, but in order to apply the plugin to the selected image, you must also click on a "power button"-style icon next to the plugin name. That is more steps than should really be necessary, and I think it reflects the underlying design of the code base more than it does a particular UI theory. Darktable has a fast image-processing core, but all of the editing features are implemented as separate plugins. That makes for faster development, but in this case it complicates things for the end user.

What is more frustrating is that Darktable has several more peculiar UI quirks that make its features hard to discover. First, all of the aforementioned plugins are shown in an unlabeled grid, one in which each icon is a slick-looking minimalist logo, and all of which look virtually indistinguishable. You will have to hover the mouse over every icon long enough for its tooltip label to materialize. It would be simpler to list all of the plugins in a menu, but Darktable (deliberately?) has no menus. That has its own side-effects, such as making the preferences dialog the only place to discover keybindings (more on that below...). In other parts of the interface, Darktable uses text labels instead of icons (such as the selection buttons), which makes me suspect that the symbolic plugin icons are a purely cosmetic choice.

The Darktable window also uses triangular "expander" buttons on the top, bottom, left, and right edges to selectively show or hide sub-panes (such as the tool strip or navigation/image-info box). That is not bad in and of itself, but the triangles stay on the edge of the screen whether the pane they control is collapsed or not. As a result, it was about an hour before I discovered that the arrow at the bottom edge did not control the filmstrip widget also on the bottom of the screen, but instead toggled the visibility of a one-line color-selector tool which, at the bottom of the window, is nowhere near the other tools. The filmstrip can be hidden, but only by opening the preferences dialog. The preferences dialog, though, is only accessible through an unlabeled "gear" icon placed next to the image-sort-field selector, which itself is only visible when you expand the top-edge window pane.

A good UI-review or sprint with some usability experts would fix a lot of these issues and make Darktable significantly easier to navigate. On the other hand, some of the actual image-adjustment tools have very perplexing interfaces of their own that I am less sure how to fix, because I am still not sure I understand them. Most are normal sliders, spinboxes, or selection menus, but a few head in different directions. For example, the color-correction widget is a square grid of colors, into which the tooltips tell you to "draw a rectangle to give a tint." It is clear that whatever colors you catch with your rectangle somehow affect the tint, but exactly what they do is a mystery (a shift? a rotation? remapping the entire color space?) — and because the grid is unlabeled, you do not have precise control over the effect. Furthermore, because of the grid's fixed orientation it is impossible to draw rectangles around certain color combinations, such as blue-to-orange.

The "color zones" plugin's widget is arguably more confounding. It has three separate grids, one each for lightness, saturation, and hue, all unlabeled, and each incorporating a flexible adjustment curve, a set of control points bound to the bottom edge, a circle that hovers around the cursor, and a fluttering gray polynomial curve that surrounds the circle but changes in size and shape depending on some combination of the circle's diameter and the shape of the adjustment curve. Confused? You are not alone. Neither the online documentation nor the PDF manual explain the design at work; you can learn that the circle is a "virtual color filter" and that you can control it's diameter with the mouse scroll wheel, but little else. Simply labeling the UI elements used in these widgets would be a good start, but the lack of clear documentation makes them unusable.

Image is everything

All user interface issues aside, I do like Darktable, both for what it can do, and how quickly it can do it. It provides tools that allow you to focus on creating the look that you want in the output image, rather than exposing only low-level adjustments. Other raw editors give you control of de-noising and exposure; Darktable does, too, but it also has a zone system plugin and the ability to do split-toning (which Photoshop veterans will recognize as "duotones").

Among the effects plugins are several common photographic tools, such as a graduated neutral-density filter. In the on-camera world, this is a lens filter that darkens one part of the image, then fades naturally out — it is what people routinely use to bring the too-bright midday sky back into the same range as the ground. There are also composition guides (golden ratio, rule-of-thirds), lensfun-based distortion corrections, grain simulation, and a "fill light" plugin. For the most part, the controls are easy to use and the changes are rendered on the preview image with little noticeable lag.

The application also provides more control than its competition over common tasks like converting an image from color to grayscale. Other programs do this with a simple switch to monochrome — sometimes offering the choice between using the slightly-different luminosity or brightness value of each pixel to determine its final tone. Darktable tries to emulate the control provided by shooting in black-and-white in the first place, using various filters to alter the light. In his demonstration videos, developer Pascal de Bruijn said that he used actual black-and-white film to calculate the characteristic curves used in the presets (although he did not specify the type of film).

Granted, to a non-photographer, some of the plugins implemented in Darktable may not mean much at first, but they are still valuable. For example, there is a "Velvia" plugin which is intended to emulate the distinctive look of Fujichrome Velvia film, which pulled extra saturation out of normally neutral scenes. You could achieve the same effect by manually manipulating the saturation settings, but doing so without blasting-out cartoon-like colors is difficult. People who have never shot Velvia film and never care to start will probably just be happy to have a simple-to-use saturation preset.

The list of actual changes in the 0.9.3 series is modest; there is an "invert" plugin specifically built for scanning and converting film negatives, a new de-noising filter, and new presets for several of the existing tools (such as the "real world" black-and-white film conversion). Two new file import features will save obsessive photographers some time: a checkbox to recursively import a tree of directories, and a checkbox to ignore JPEG files and only import raw images.

There is also a new row of buttons above the existing plugin categories (which were generic groupings like "color," "correct," and "effect"). The "active" button shows you only the plugins that are currently activated, which can help if you turn on too many and get lost, and the "favorite" button allows you to mark your own custom set of plugins for quick access. As the number of plugins continues to grow, this will only be more valuable. Minor enhancements have landed for other tools, such as the crop/rotate plugin, which now works more like other image editors, allowing you to draw a line on the screen that should be horizontal and have Darktable automatically make the proper adjustment.

This release will probably not shake off Darktable's reputation as a tool tailored for photography geeks. The image manipulation functions are maturing, which you can see in the accumulation of carefully crafted presets. They allow users to quickly apply effects without having to understand the complex curves built in to the tools themselves. However, if the application is ever going to break into mainstream acceptance, it will have to undertake a serious attempt to fix its user interface. The tool grids, menu-free design, and everything-in-a-single-window approach may look slick, but they slow down the user's actual completion of tasks, which is an unforgivable sin among GUIs.

Every raw converter on the market struggles with the problem of how to fit a dozen adjustment widgets onto the screen along with the image they affect. Sure, none of them has solved the puzzle outright, but going-it-alone in user interface design rarely works either. Even Blender — which for years argued that everyone would eventually adapt to its strange conventions if they only sat down and used it long enough — finally undertook a redesign in the 2.5x series. Like Blender, Darktable offers a unique set of features, they just aren't as easy to utilize as they should be. Some work on that front would grow the user base, and hopefully allow the team to build out other areas of photography workflow, such as collection management and batch processing.

Comments (3 posted)

Quotes of the week

-- Dan Williams

-- Alan Cox

-- Tom Christiansen

Comments (none posted)

IMAPClient 0.8 released

Version 0.8 of IMAPClient, the "easy-to-use, Pythonic and complete IMAP client library with no dependencies outside the Python standard library" has been released. New features include OAUTH support, full NOOP support, IDLE support, Sphinx-based documentation, folder renaming, and more.

Full Story (comments: none)

Introducing Mozilla Conductors

Many projects have grappled with the problem of improving the quality of communications on their mailing lists, IRC servers, and so on. The latest effort in this area is "Mozilla Conductors," an effort introduced by Mitchell Baker and to be headed up by Stormy Peters. "How do we improve our ability to disagree vehemently and simultaneously remain civil and productive? As in so many things, the answer is to empower people who are already doing some of this and to explicitly make this a valued contribution."

Full Story (comments: none)

Open64 5.0 released

Open64 is a GPLv2-licensed compiler for the x86 and Itanium architectures with a long history; the project has just announced the 5.0 release. Much of the development work going into this release seems to be aimed at the exploration of interesting optimization techniques, but there doesn't appear to be a lot of information out there on how Open64 compares to other open compilers. The Open64 wiki has some more information about the project.

Comments (21 posted)

SciPy 0.10.0 released

SciPy, a collection of mathematics, scientific, and engineering tools for Python has released version 0.10.0. " SciPy 0.10.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation." Some of the new features include a new build system (Bento), sparse eigenvalue problem solver functions, support for simulating discrete-time linear systems, and more.

Full Story (comments: none)

Tomahawk 0.3

Version 0.3 of the Tomahawk media player is now available. New features include a "one-click" resolver gallery, various types of popularity charts, "lazy lists," and more.

Comments (none posted)

Valgrind-3.7.0 is available

Version 3.7.0 of the Valgrind runtime analysis tool is now available. New features include preliminary support for Android on ARM, support for IBM z/Architecture (s390x) running Linux, MacOSX 10.7 and XCode 4 support, better memcheck tests, increased performance in Helgrind, and more. See the release notes for all the details.

Full Story (comments: none)

Development newsletters from the last week

• Caml Weekly News (November 15)
• Perl Weekly (November 14)
• PostgreSQL Weekly News (November 13)

Comments (none posted)

Federico Mena-Quintero talks about the Document-Centric Desktop (der Standard)

der Standard interviews GNOME co-founder Federico Mena-Quintero. "The latest thing is that now things have to go through the design team first, and I don't think that is a good thing; there should not be a central body of control that decides how things are done, because that simply doesn't scale. And it also doesn't teach people in how to do design properly. I really would like to move to a model where, instead of having a central body of people who can veto things in or out, we can have a shared understanding of what constitutes good design and implementation. And then, we can evaluate proposals on all their merits and modify them instead of just saying, 'Oh no, I don't like this, because it doesn't follow my design criteria'."

Comments (9 posted)

Introduction to Kdenlive (opensource.com)

Opensource.com has posted a lengthy introduction to kdenlive (a video editor) as the beginning of a series of articles on this tool. "A good video editor is one that is suitable for anyone wanting to edit video, with powerful features that enable the video professional to do any task required of the job, yet with the simplicity that allows a hobbyist to quickly cut together footage off of a phone or point-and-click camera. Kdenlive can be both of those things, but regardless of the scope of your video project, there are right and wrong ways of doing things. Over the course of five articles, we will review the practical usage and the common set of best practices that will ensure your projects are successful."

Comments (none posted)