While there are a lot of command line tools for metadata manipulation, only a few python libraries deal with this subject. A recent Article on EXIF Extraction in Python introduces some of them, but unfortunately doesn’t cover the newly-released pyeviv2. So I decided to spread the word about it.

What is pyexiv2?

pyexiv2 is a Python binding to exiv2, the C++ library for manipulation of EXIF, IPTC and XMP image metadata. As mentioned above, it allows you to perform many metadata manipulation tasks, such as correcting wrong photo timestamps, assigning GPS coordinates to your photos or simply dumping out the EXIF metadata generated by your camera.

Why should I use pyexiv2 instead of EXIF.py or similar libs?

• pyexiv2 also supports Camera RAW (.NEF, .CRW, .ORF, etc) files and even gives you access to their embedded preview images, which allows you to perform very fast RAW-to-JPEG conversions.
• It is able to decode many important Makernote Tags, such as LensTypes and ProgramModes
• Being C++-based, pyexiv2 offers ultra-fast read and write support for metadata. In my tests, pyexiv2′s underlying library exiv2 was magnitudes faster than exiftool when it came to writing metadata
• It provides a convenient Python API, no need to call external tools (as with various python-to-exiftool wrappers)
• pyexiv2 automatically converts many tag values to approbriate Python datatypes (e.g. Exif.Image.Datetime returns a Datetime object), freeing you from the parsing date strings like '2004-07-13T21:23:44Z'. Of course, you can also access the raw value of a tag.
• [UPDATE]: I almost forgot to mention the most important merit of pyexiv2: it has a very robust metadata parser that is tested against hundreds of sample photos from different camera models. While I appreciate the various pure-Python EXIF parsers for educational purposes, they never came close to pyexiv2′s stability.

What is new in version 0.2.0 ?

pyexiv2 0.2.0 handles XMP metadata and supports reading images from stream using the from_buffer method. This means that you can provide pyexiv2 with a variable holding the image data, in prior versions, only the filepath of the image was accepted. Furthermore, pyexiv2 is now compiled against libevix2 0.19, which brought huge performance improvements for reading metadata. Finally, it exposes all recent enhancements of the libexiv2 API, such as preview image access.
pyexiv2 is a complete, non-backwards-compatible rewrite and features an improved, better documented API. It was compiled and tested under Linux and Windows.

Installation & first steps

Windows folks can download the .exe-Installer (requires Python 2.6 or newer on your system), Linux folks have to get the source code and install the required dependencies before compiling pyexiv 0.2.0. There are also pre-built packages in debian experimental, but these are not suitable for production environments.

The excellent introductory tutorial on pyexiv2′s website will make you quickly proficient with the library.

Unfortunately, some open questions remained: I couldn’t evaluate my assumption, because I don’t own a PowerShot. Another interesting, yet unanswered question is how quickly the CameraTemperature adapts to the environment temperature. Luckily I spend this weekend at my mother’s house and could conduct the necessary tests with her Powershot A2000 IS.

My test setup

My test setup was fairly simple: turn on the Powershot for 5 minutes and take a picture to my digital thermometer every 20 seconds. I planned to conduct this procedure inside my appartment (approx. 24 degrees Celsius room temperature) and outside (approx. 3 degrees Celsius). Before running my tests I allowed all devices an acclimation time of 10 minutes. I deliberately choose this period to be rather short, because I wanted to test how fast the CameraTemperature adapts to a changing environment temperature (think of quickly leaving your warm room to take some pictures of the snowman the kids have built).

Although the Powershot A2000 IS is just pocket size, I didn’t put it in my jacket to avoid any accidential heating by my body.

Interior Test

The interior test ended quite promising: I took 15 pictures, the first 7 yielded a camera temperature of 20 degrees, the other 8 of 21 degrees. My thermometer was showing constantly 23.3 degrees. Wow! This means just a difference of 2-3 degrees.

The thermometer reports 3.7 degrees celsius, but the CameraTemperature yields 18 degrees for this picture.

The disappointing Exterior Test

The exterior test caused more headaches (not just because of the cold October ). Even after the acclimation period of 10 minutes, my reference thermometer was still showing the interior temperature of 23.3 degrees. When I “rebooted” it by removing the batteries, the temperature first dropped down to 8, then to 3 degrees. This was the accurate temperature according to the local weather report.

However, for my total 20-minutes-stay outside, the camera temperature never dropped below 17 degrees. Quite far away from the 3 degrees actual exterior temperature.

This diagram shows the slow acclimation time for the CameraTemperature. Coming from a previous interior temperature of 23.7 C, it was exposed to an exterior temperature of 3.7 C for 10 minutes. After that, numerous shots were taken at the same temperature. The points indicate the recorded CameraTemperature for each shot. After 20 minutes, the CameraTemperature just dropped to 17C

Conclusions

The results from the exterior test seem to indicate that the camera needs a long acclimation time until its temperature reachs the external temperature. In my tests, the temperature dropped by just 5 degrees in the first 20 minutes, which equals to 25% percent of the total temperature difference of 20 degrees. If we assume a linear progression, the camera needs to be exposed at least for 4*20 = 80 minutes to an environment until its temperature equals the environment temperature.

UPDATE:Also quite a few Canon IXUS and EOS models as well as many Pentax cams seem to write this tag.

Hot or cold? - The CameraTemperature tag recorded by Canon Powershot cameras allows you to conduct temperature-based searches.

Got footage?

If you don’t own a PowerShot, you can quickly download the necessary footage: Phil Harvey hosts a huge “Meta Information Repository”, that contains sample images from many camera manufacturers, including Canon. Simply download the Canon Archive and extract it to your hard disk. Note: to save bandwith, the actual image has been replaced by a tiny placeholder, but the original metadata is still in place. This is fully sufficient for our experiments.

As I don’t own a PowerShot by myself, I also made use of the footage in the repository. If you are a PowerShot user and want to create footage by yourself, be warned: don’t expose your camera to extreme temperatures as this might rapidly shorten its life expectancy.

Extracting the Camera Temperature

First make sure that you ‘ve installed Exiftool 7.97 or newer on your system. When in doubt, simply run

exiftool -ver

from your shell or command prompt – this command displays exiftool’s version number. Then navigate to a folder containing unmodified photos taken by a Canon Powershot camera and run

exiftool -CameraTemperature -ext JPG .

and you should get a list like

Camera Temperature : 23 C
======== CanonPowerShotSD890IS.jpg
Camera Temperature : 26 C
======== CanonPowerShotSD900.jpg
Camera Temperature : 25 C
======== CanonPowerShotSD950IS.jpg
Camera Temperature : 30 C


Exiftool lists the camera temperature in degrees celsius. The -ext JPG . part of the command line makes exiftool look for all JPG files in the current directory, no matter if the extension is upper- or lowercase. By appending -r you can expand the search to subdirectories.

Filtering by temperature

With a bit of command-line-magic, you can also filter your photos by temperature:

exiftool -cameratemperature -if '$cameratemperature# < 20' -ext JPG .

will list all photos that were taken at a camera temperature below 20 degrees celsius, whereas

exiftool -cameratemperature -if '$cameratemperature# >= 20' -ext JPG .

displays a list of all photos taken at 20 degrees or more.

These two filtering commands deserve some explanation: the -if parameter tells exiftool to process only files that meet a certain condition. Also the inner part of the if clause is a bit special: '$cameratemperature# < 20' consists of the tag name cameratemperature (the prepended $ is needed to identify a tagname within a string). The appended # tells exiftool to convert the tag value to a number (e.g. 19 instead of '19 C'). The rest of the clause is quite easy: '< 20' stands for "all values smaller than 20", while >=20 means “greater or equal to 20″.

ThermoTagging hot or cool photos

As exiftool “decodes a riddle wrapped in a mystery inside an enigma” (quote Phil Harvey), it can extract such esoteric tags like Canon:CameraTemperature that are buried in the depths of propriertary Makernotes. Most (if not all) other Metadata readers and image management applications fail decoding this information. So it would be nice to convert the camera temperature to a format that all other applications can access. The most common way to do so is to create an IPTC Keyword for the information contained in the tag. IPTC Keywords are read by most image editors including Photoshop, many photosharing services (including flickr and picasa) auto-convert them to tags during photo upload.

Converted Thermotags at flickr.com

Storing descriptive tags

exiftool -keywords+=cold -if '$cameratemperature# < 15' -ext JPG .

adds the IPTC keyword "cold" to your photo, if the camera temperature was less than 15 degrees. Just modify keyword and if-clause to tag "hot" photos
To test your results, you can use


exiftool -keywords -cameratemperature -if '$keywords eq "cool"' *.jpg
======== CanonEOS_KissX2.jpg
Keywords : cool
Camera Temperature : 18 C
======== CanonIXY_DIGITAL20IS.jpg
Keywords : cool
Camera Temperature : 17 C
[...]

This commands lists all photos with the keyword “cold” and extracts their keywords and Camera Temperature.

Storing precise values

If you want to store the precise degree value, you can use:

exiftool '-cameratemperature+>keywords' *.jpg

which copies the Camera Temperature value directly to an IPTC Keyword:

exiftool -keywords CanonPowerShotSX200IS.jpg
Keywords : cool, 18 C


Is the camera temperature really identical with the outside temperature?

Although I don’t own a Canon Powershot, I wanted to verify Phil’s assumptions on Canon’s CameraTemperature. In short, he claims that the CameraTemperature is close to the outside temperature, if the device was left in an environment for a certain time. According to him, taking a lot of images in a short time additionally heats up the camera by some degrees. While my tests can’t prove the equality of the temperatures, they indicate a strong correlation (maybe with some offset factor):

• Range of the CameraTemperature exiftool dumped out values between 12 and 39 degrees celsius for Phil’s sample images. This indicates that the camera temperature is maybe a bit above the outside temperature (why are there no pics with values less than 10 degrees?)
• Correlation of creation date and CameraTemperature When I compared the temperature with the creation date (Exif:DateTimeOriginal), it turned out that all photos with value less than 20 degrees were taken in autumn or winter
• Visual correlation I extracted a few thumbnails from the “hottest” (> 35) degrees and coldest (< 15 degrees) values) and tried to find visual hints for heat or coldness (e.g. snow, summer clothes etc). My verdict is that there are no contradictory elements, but you can judge for yourself by looking at the hottest or the coldest thumbnails.

Acknowledgements
I want to thank Phil Harvey for developing Exiftool and his suggestions on this article. Vesa Kivisto also deserves a honorary mention for the decoding of the CameraTemperature tag.

Estimating the “true age” of a camera

The date of purchase isn’t a very reliable indicator, as the usage intensity of cameras can vary greatly. A 2-year-old DSLR that is mainly used at holidays or family events is surely in a better condition than one that serves as bi-weekly-party snapshot device.

It would be nice to have something like a mileage indicator for DSLRs, that more or less counts every picture ever taken with a particular cam. Luckily, such a magic indicator already exists, although its information isn’t always easy to obtain: The Shutter Count measures the number of shutter releases during the lifetime of your camera. A shutter is a moveable device, that exposes the camera sensor to the light during the shooting of the picture. Roughly spoken, the number of shutter releases approximately corresponds to the number of pictures taken with your camera.

Accessing the Shutter Count with exiftool

Unfortunately, the Shutter Count isn’t very easy to access. Only a few DSLR models from Nikon, Canon, Pentax and Samsung (green model names) dump this information out with their EXIF metadata. If you intend to buy such a camera, you can ask the seller to mail you an unmodified photo directly taken from the camera.

Jeffrey Friedl's Exif Viewer reports the Shutter count

You can then either upload the photo to Jeffrey Friedl’s Exif viewer or install the famous ExifTool metadata extractor on your system (Win/Mac/Linux). Open a shell/console, navigate to the directory where you stored the photo and type

exiftool THE_UNMODIFIED_IMAGE.JPG | find "Shutter Count"
on Windows or


exiftool THE_UNMODIFIED_IMAGE.JPG | grep Shutter Count
on Linux or Mac OS.

If a line like
Shutter Count: 2391

appears at your command prompt, your attempt was successful. If nothing appears, your model likely doesn’t include the shutter count in its metadata. NOTE: Only a few EXIF viewers will extract the Shutter Count value, as it is stored in the so-called Makernotes (proprietary extensions of the EXIF standard). To get reliable results, please use the tools I mentioned above.

The Firmware way

This Olympus E-500 has a Shutter Count value of 2265 (screenshot of cam display, yellow highlighted)

Interpreting the numbers

You had a hard time to retrieve the shutter count, but what does the value mean? Is the cam still a teenager or already almost in heaven? Oleg Kikin maintains a Shutter Count life expectancy database. Users report their shutter count numbers and whether the cam is still alive or already trash.

Life expectancy of a Nikon D70 based on the shutter count value (clicks). Only 37% of all models with more than 100.000 pictures are still working.

Imagemagick to the rescue?

When it comes to batch-file processing, Imagemagick is often the obvious choice. No nagging dialogs or splash screens, just a pure and simple command line interface.
In fact, converting a bunch of Camera RAW files to JPEG is as simple as typing:

mogrify -format jpg *.*

in a shell/console window (of course you need to cd to your RAW photo directory before). Unfortunately, this simple approach has some downsides:

• Imagemagick depends on UFRaw to be able to process RAW photos. If this package is not installed, Imagemagick exists with a rather cryptic error message.
• On my quite up-to-date laptop (Intel Core2 Duo T9300, 3 GB Ram, Ubuntu 9.04, ImageMagick 6.5.0-2), the conversion process took 65 seconds for 7 RAW files under almost 100% CPU load. That’s 10 seconds per RAW Photo. But for a low-end-netbook, you might have to multiply these numbers by 4 or 5, since the CPU/memory is much weaker.

So, waiting 40 seconds for one photo to convert is not your preferred choice?

Why so slow?

Don’t blame Imagemagick for its poor performance. Converting RAW photos to other formats is actually a computationally intensive task, since the RAW format doesn’t store ready-to-display pixel/color-channel data, but just the information captured by the photo sensor of the camera. You can think of it as some sort of “digital negative” that needs a huge amount of processing, before it becomes a viewable image. The benefit of this “raw” approach is that you can massively influence the development process. With JPEG coming out of your camera, the possibilities are much more limited.

Speeding up the conversion

But back to our problem. If the conversion process is that CPU/memory-hungry, how can we improve it?
The answer is easy: by circumvention. If we look closer at the contents of a RAW image file, we find

“[..] optionally a reduced-size image in JPEG format which can be used for a quick and less computing-intensive preview[..].”

Luckily, “optionally” means de-facto-standard, since the RAW image formats of Nikon, Canon, Olympus, Pentax and many other manufacturers support this feature.
So we don’t need to compute the JPEG representation of our RAW file, we can simply “snip out” an already embedded JPEG from the RAW file. We? Let’s delegate this to a specialized program.

Almost-realtime-”conversion” with exiv2

Since version 0.18, the excellent exiv2 utility can not only extract EXIF/IPTC information from an image, but also dump out embedded JPEG preview files from RAW photos.
The workflow is straightforward: Open a shell/console window and cd to your RAW photo directory.
First we need to find out, which previews are embedded in our RAW file:

exiv2 -pp RAW_CANON_350D.CR2

exiv2′s response tells us that there are 2 embeded preview images, one thumbnail with 160×120 pixels, and a larger preview with 1536×1024 pixels:

Preview 1: image/jpeg, 160x120 pixels, 5164 bytes
Preview 2: image/jpeg, 1536x1024 pixels, 151637 bytes

Preview #2 is just fine for our purpose (uploading to flickr), so we extract it with

exiv2 -ep2 RAW_CANON_350D.CR2

which creates a file called RAW_CANON_350D-preview2.jpg in the current directory. The option ‘-ep2′ simply means “extract preview #2″. To batch-process all our RAW files in the current directory we can type:

exiv2 -ep2 *.CR2

because all RAW files from one camera manufacturer have the same preview image structure. Don’t forget to adapt the file extension to the one of your manufacturer, e.g. *.ORF for Olympus, *.NEF for Nikon.
If you want to extract the preview images to some other directory, the exiv2′s ‘-l’ option is perfect:

exiv2 -ep2 -l previews *.CR2

would extract the JPEGs to the already existing subdirectory ‘previews’.

Downsides

exiv2′s extraction approach is ultra-fast, but has the downside that its limited to the embedded previews of the RAW file. You can’t generate JPEG dumps of an arbitrary size using exiv2. However it is possible to process the extracted previews with Imagemagick (e.g. to scale them down further, apply better compression). Furthermore you have to dispense with the power of RAW processing. However, our described method is just right when you need a quick conversion and/or only have limited CPU horsepower.