Automated image testing with Verified Pixel

Verifying the authenticity of a digital image is no simple task, given the array of image-manipulation tools available and the difficulty inherent in tracing the provenance of any digital file. But attempting to establish the origin and veracity of a photo is not a lost cause. The non-profit organization Sourcefabric, which produces open-source journalism tools, has developed a web-based utility called Verified Pixel that attempts to make assessing an image's reliability an attainable goal.

The Verified Pixel site notes that news organizations are increasingly struggling with how to verify the authenticity of smartphone footage and other images captured by eyewitnesses' cameras in the field. A lot of news organizations solicit user-contributed images, which results in a glut of input and not enough time to perform forensic analysis of every photo. This is in spite of the fact that there are several well-known forensic tools available.

In April 2015, Sourcefabric and Eyewitness Media received a grant from the Knight Foundation to develop an image-verification web service to address the problem. Verified Pixel is the result of that effort. After teasing the prototype on Twitter in October 2015, Sourcefabric began rolling out access to a test server to beta testers in January 2016. I requested an invite, then spent some time kicking its tires and asking questions of the development team.

The source code is available on GitHub, although the dependencies are likely large enough to ward off casual users. Verified Pixel is implemented as a module for Sourcefabric's newsroom-management tool Superdesk (although the beta is currently a standalone web application, the intent is likely to make Verified Pixel a standard Superdesk component). The server side is written in Python, with an AngularJS client front-end. Like Superdesk, Verified Pixel is designed to be run on an organization's own server; the test server used for the prototype was maintained by Sourcefabric simply to solicit feedback on the program's functionality.

The idea is for Verified Pixel to provide a consistent, multi-user workflow: any user can upload images to the database, and a flexible battery of verification tests will be run to assess each image's reliability. Users can then flag individual images as suspect or potentially valid based on the scores of the validation tests. Authors or editors can subsequently select images to use taking those test results into account.

My pixel is my passport, verify me

The current battery of tests includes three online services: Google's reverse-image search (which will find suspiciously similar images if they exist), TinEye (which will attempt to locate probable duplicates of an image and determine which is the oldest), and Izitru (which runs a set of forensic tests to determine if an image has been edited). In addition, Verified Pixel automatically extracts any Exif data from the image and highlights key fields (for example, plotting GPS locations and camera direction, if available, on a map).

All users can add comments to images in the database, and the image database can be searched on all metadata fields, including the verification-test results. Images can also be grouped into collections several layers deep; the topmost layer is called a "desk," which is representative of a separate office within a publication's newsroom.

The Google and TinEye search results can help attest to whether or not an image has been published previously, which presumably would not be the case for a user-submitted photo of a current event. Such a test would still have value even for images that are not coming in as part of a breaking news story, such as detecting stock photos that were presented as original coverage.

The Izitru service runs six forensic tests and returns a "trust rating" on a one-to-five scale that corresponds to whether or not the image is in a "direct from camera" state or has been modified after the fact. Unfortunately, the service does not detail what the six tests are, although the Verified Pixel test-results page provides some rough descriptions. They include tests of how JPEG data is packaged within the file (based on differences between how cameras and software package JPEGs), tests to detect the traces of camera sensor patterns, and tests that look for artifacts of recompression. Founder Harry Farid has authored a lengthy list of research papers on image forensics that likely cover the same ground.

These tests also depend, at least in part, on analysis of known camera characteristics. The FAQ page notes that "each digital camera has distinct ways of applying JPEG settings when saving a file. Other differences result from artifacts that are introduced when images are saved multiple times." The site elsewhere says that it "relies on a database of 'signatures' that describe the distinct ways that different camera models store their JPEG files" and that cameras not in the database will likely not receive the highest trust rating.

So Izitru can rate the likelihood that an image is unmodified; if the image has been modified, though, further testing is required to determine what the modifications were. Simple resizing is not particularly bad, while adjusting color curves or editing for content are far more serious issues. Izitru is limited to analyzing JPEG files, which may be of concern to some users who would be interested in support for camera raw files. But the JPEG limitation is in line with what many news organizations ask from user-contributed content; in November 2015, the Reuters news service announced it would only accept image files from freelancers that were shot as JPEG.

Looking forward

For free-software developers, however, the service's proprietary and secret test battery is likely a larger issue. The good news is that Verified Pixel is designed to support pluggable test modules, and Sourcefabric's Douglas Arellanes said support for additional verification services is still to come. In an email, Arellanes said that the project spent a fair amount of time working with both OpenCV and ccv on modification-detection and other tasks using machine vision, but ultimately found that it was not a priority for users:

Still, he said that he hopes to attract code contributions to support open-source libraries, since there is a lot that could be applicable. The team has an informal wishlist of additional tests it would like to see, such as automatic image tagging to recognize images containing traumatic content and a method to compare weather data (based on geolocation and timestamps) with the apparent conditions indicated in an image. The traumatic-content issue is one that Verified Pixel developer Sam Dubberley has explored in his own research.

But, for now, the project is focused on getting real-world feedback from news organizations. Arellanes said that "we need to see where the bottlenecks are for image verifiers - is it in workflow and getting the images into newsrooms' CMSes? Is it in making metadata easier? This is what we'd like to glean" from testing. He also added that Verified Pixel may be useful beyond the initial newsroom use case. "We think it has uses in a human rights context as well as in any situation requiring image verification - insurance claims, for example."

The beta runs its test battery on new uploads quickly, and searching and sorting are both painless. Given that the Google and TinEye assessments rely on a massive corpus of published-image data, it is hard to imagine comparable tests that do not dictate the use of a proprietary service. The Izitru modification-detection tests, though, could face stiff competition from other libraries if the project attracts a development community. It the meantime, it is clear to see how automating all such tests and collating the results can help users—in a newsroom or elsewhere—simplify the job of sifting through digital images of unknown trustworthiness.