Ultra-fast RAW to JPEG conversion with exiv2

Finally you escaped from your dusty appartment and spent a wonderful day shooting photos in the wild with your brand-new SLR. To capture all the magnificient details of nature, you decided to shoot in Camera RAW, and not in crumpy-old JPEG format. While waiting for the train home, you quickly want to upload the best shots to your flickr account. Since each RAW photo has a filesize of 10 Megabytes, this isn’t the most bandwith-friendly uploading format. Furthermore, your tiny netbook offers only little CPU horsepower. So you are desparately looking for an easy and performant way to convert dozens of RAW photos back to JPEG.

Imagemagick to the rescue?

When it comes to batch-file processing, Imagemagick is often the obvious choice. No nagging dialogs or splash screens, just a pure and simple command line interface.
In fact, converting a bunch of Camera RAW files to JPEG is as simple as typing:

mogrify -format jpg *.*

in a shell/console window (of course you need to cd to your RAW photo directory before). Unfortunately, this simple approach has some downsides:

  • Imagemagick depends on UFRaw to be able to process RAW photos. If this package is not installed, Imagemagick exists with a rather cryptic error message.
  • On my quite up-to-date laptop (Intel Core2 Duo T9300, 3 GB Ram, Ubuntu 9.04, ImageMagick 6.5.0-2), the conversion process took 65 seconds for 7 RAW files under almost 100% CPU load. That’s 10 seconds per RAW Photo. But for a low-end-netbook, you might have to multiply these numbers by 4 or 5, since the CPU/memory is much weaker.

So, waiting 40 seconds for one photo to convert is not your preferred choice?

Why so slow?

Don’t blame Imagemagick for its poor performance. Converting RAW photos to other formats is actually a computationally intensive task, since the RAW format doesn’t store ready-to-display pixel/color-channel data, but just the information captured by the photo sensor of the camera. You can think of it as some sort of “digital negative” that needs a huge amount of processing, before it becomes a viewable image. The benefit of this “raw” approach is that you can massively influence the development process. With JPEG coming out of your camera, the possibilities are much more limited.

Speeding up the conversion

But back to our problem. If the conversion process is that CPU/memory-hungry, how can we improve it?
The answer is easy: by circumvention. If we look closer at the contents of a RAW image file, we find

“[..] optionally a reduced-size image in JPEG format which can be used for a quick and less computing-intensive preview[..].”

Luckily, “optionally” means de-facto-standard, since the RAW image formats of Nikon, Canon, Olympus, Pentax and many other manufacturers support this feature.
So we don’t need to compute the JPEG representation of our RAW file, we can simply “snip out” an already embedded JPEG from the RAW file. We? Let’s delegate this to a specialized program.

Almost-realtime-”conversion” with exiv2

Since version 0.18, the excellent exiv2 utility can not only extract EXIF/IPTC information from an image, but also dump out embedded JPEG preview files from RAW photos.
The workflow is straightforward: Open a shell/console window and cd to your RAW photo directory.
First we need to find out, which previews are embedded in our RAW file:

exiv2 -pp RAW_CANON_350D.CR2

exiv2′s response tells us that there are 2 embeded preview images, one thumbnail with 160×120 pixels, and a larger preview with 1536×1024 pixels:

Preview 1: image/jpeg, 160x120 pixels, 5164 bytes
Preview 2: image/jpeg, 1536x1024 pixels, 151637 bytes

Preview #2 is just fine for our purpose (uploading to flickr), so we extract it with

exiv2 -ep2 RAW_CANON_350D.CR2

which creates a file called RAW_CANON_350D-preview2.jpg in the current directory. The option ‘-ep2′ simply means “extract preview #2″. To batch-process all our RAW files in the current directory we can type:

exiv2 -ep2 *.CR2

because all RAW files from one camera manufacturer have the same preview image structure. Don’t forget to adapt the file extension to the one of your manufacturer, e.g. *.ORF for Olympus, *.NEF for Nikon.
If you want to extract the preview images to some other directory, the exiv2′s ‘-l’ option is perfect:

exiv2 -ep2 -l previews *.CR2

would extract the JPEGs to the already existing subdirectory ‘previews’.

Downsides

exiv2′s extraction approach is ultra-fast, but has the downside that its limited to the embedded previews of the RAW file. You can’t generate JPEG dumps of an arbitrary size using exiv2. However it is possible to process the extracted previews with Imagemagick (e.g. to scale them down further, apply better compression). Furthermore you have to dispense with the power of RAW processing. However, our described method is just right when you need a quick conversion and/or only have limited CPU horsepower.

About Franz Buchinger

Digital Photography/Python Aficionado from Vienna, Austria. Tamed his chaotic photo collection with some magic metadata spells. Decided to share his humble EXIF, IPTC and XMP knowledge in this blog.
This entry was posted in EXIF-Hacks. Bookmark the permalink.
  • http://www.picurl.org/blog/2010/04/05/pyexiv2-the-best-choice-for-photo-metadata-manipulation-in-python/ pyexiv2 – the best choice for photo metadata manipulation in python « picurl

    [...] pyexiv2 also supports Camera RAW (.NEF, .CRW, .ORF, etc) files and even gives you access to their embedded preview images, which allows you to perform very fast RAW-to-JPEG conversions. [...]

  • Ken Rahaim

    While imagemagick is indeed slow at converting raws to jpg for the reasons stated, good old Alchemy is lightning quick at creating jpgs from raws? Why is this? It is not using the embedded jpg. I tested that by trying to convert a dng to raw that didn't have an embedded jpg and alchemy did the conversion just as quickly (1 second). So what is alchemy grabbing to do its conversion? Its not the raw itself. I did a jpg conversion leaving the same pixel dimensions as the original raw (4000×3000 pxls) and the resultant jpg was very pixelated. So, I assume alchemy is using some other embedded (thumbnail?) jpg to do its “conversion” with.

  • fbuchinger

    Hi Ken, I think you are confusing “Camera Raw” with a “raw” image file format, that simply writes out pixel values without compression or metadata. “Camera Raw” files such as .CRW, .ORF store the data captured by the camera chip. “Camera Raw” files are read-only, because the original chip data can't be derived from the pixel values of an image.

  • http://velmont.no/ Velmont / Odin

    Oh my. I'll use this in my “photo gallery / management” web application.

    Lifebox, which I'm using now, is pulling out ufraw for all my pictures, and although I've got a fast nice server, having thousands and thousands of images it still is too slow. So, I'll write my replacement to use your technique, very nice.

blog comments powered by Disqus