Updating metadata in Darktable

Question

Bear with me.

I've been using Darktable for a while now along with Olympus' own software and digikam for uploading, keywording etc and GIMP for any retouching I can't easily achieve in Darktable. I use Olympus Viewer to add a caption which is placed in the User Comment metadata section, digikam reads this and places it in its database caption and creates a local XMP file with this caption in. I think Darktable reads this and when a file is exported the caption is added to the metadata in the Description section (by the way Olympus place "OLYMPUS DIGITAL IMAGE" in this field). This is picked up by flickr so is quite handy.

The thing is I recently altered the User Comment on my RAW's from a day out after I had processed but not exported them with Darktable. Digikam can be made to re-read the the metadata but I couldn't find a way to force Darktable to do this. I tried removing them from the collection and then re-scanning the directory but the captions didn't update. I didn't want to delete the XMP's and start all my processing again. In the end I used digikam to alter the metadata in the jpg's I had exported.

Really sorry for the long winded post, but I wondered if anyone had any insights.

Edit after further reading on darktable.org:

In addition to the sidecar files, darktable keeps all image-related data in its database for fast access. An image can only be viewed and edited from within darktable if its data is loaded in that database. This automatically happens when you first import an image or at any later time by re-importing it (see Section 2.3.1, “Import”). In the latter case the database gets updated with data that darktable finds in the sidecar files belonging to that image.

Once an image has been imported into darktable the database entries take precedence over the XMP file. Subsequent changes to the XMP file by any other software are not visible to darktable – any changes get overwritten the next time darktable synchronizes the file. This behavior can be changed in the preferences dialog (see Section 8.2, “Core options”). On request darktable looks for updated XMP files at startup and offers a choice whether to update the database or overwrite the XMP file.

and

look for updated xmp files on startup

Check file modification times of all XMP files on startup to find out if any got updated in the meantime by some other software. If updated XMP files are found a menu opens for the user to decide which of the XMP files to be reloaded – replacing darktable's database entries by the XMP file contents – and which of the XMP files to be overwritten by darktable's database. Activating this option also causes darktable to check for text sidecar files that have been added after import time – see option “overlay txt sidecar over zoomed images” in Section 8.1, “GUI options” (default off).

Looks like as long as I set the database to update I should be OK, but this doesn't seem to be happening.

Answer 1

I've been doing some experimenting and it seems that the basic problem is with the way I was using digiKam.

In Configure - digiKam / Metadata / Behaviour there are several settings for reading and writing metadata. The really important one is : Update file timestamp when files are modified. Without this darktable doesn't know that the file has altered and will not prompt you to update the database with the "reload selected xmp files" button in the "updated xmp sidecar files found" dialogue box at start up.

---

Further findings from my experiments....

In the end I turned all the options on in Configure - digikam / Metadata / Behaviour and set "Write to sidecar files" to "Write to image and XMP sidecar", even though darktable really only interacts with the XMP file once it has been created. I guess this is an insurance against corrupting your image file.

digiKam has a setting in Configure - digikam / Metadata / Behaviour "If possible write Metadata to RAW files (experimental). Using Captions / Tags Description and Information tabs worked with title, caption, author, name, position and copyright all being recorded. However when I tried using the Image / Metadata / Edit All Metadata command my 10.7 MiB .ORF file was corrupted to 21.8 KiB, presumably all the image data is lost. This happens even when I try to edit only one item with all other checkbox's cleared.

When checking the .ORF files in Olympus Viewer which I had altered in digiKam, the only fault I can find is that the Lens Information which should read something like OLYMPUS 11-22mm F2.8-3.5 is messed up with a series of unreadable characters. This seems to be the Lens Model makernote, which digiKam displays correctly and darktable ignores preferring the Lens Type makernote (which reads "Olympus Zuiko Digital 11-22mm F2.8-3.5").

This may be due to the sync settings and certain fields being limited to a finite number of ASCII characters or the corruption of the makernotes.

digiKam uses exiv2 and their site Exiv2.org suggests that the closed, proprietary format of exif makernotes means that any change of an exif tag could move the makernotes field and corrupt it. Phil Harvey's Exif Tool site explains that the default behaviour of his metadata editing software is to work on a copy because of the complexities involved.

On the digiKam site digiKam Configuration a to do is described which will enable users to direct where digikam stores / retrieves tags ratings and comments.

I also found that I could use darktable's lighttable - import - image on a file already in the database to force an update, though this only seems to work once and then crashes darktable the next time you click on the button. Darktable's lighttable - import - folder does the same.

Finally, here are the locations that the software is writing data to.

When using the digiKam / Captions/Tags / Description tab:

Title is written to:

• IPTC : Object Name
• XMP : caption
• XMP : Title

Captions is written to:

• Exif : Image Description
• Exif : User Comment
• IPTC : Caption
• XMP : notes
• XMP : Description
• XMP : User Comment
• XMP : Image Description

Caption Author is written to:

• XMP : author
• XMP : Captions Author Names

When using digiKam / Captions/Tags / Information / Rights tab:

Names is written to:

• Exif : Artist
• IPTC : By-line
• XMP : Creator
• XMP : Artist

Position is written to:

• IPTC : By-line Title
• XMP : Authors Position

Copyright is written to:

• Exif : Copyright
• IPTC : Copyright
• XMP : Rights
• XMP : Copyright

When using darktable / lighttable / metadata editor:

Title is written to:

• IPTC : Object Name
• XMP : Title

Description is written to:

• Exif : Image Description
• IPTC : Caption
• XMP : Description

Creator is written to:

• Exif : Artist
• IPTC : By-line
• XMP : Creator

Publisher is written to:

• XMP : Publisher

Rights is written to:

• Exif : Copyright
• IPTC : Copyright
• XMP : Rights

Note darktable only writes this to the sidecar .XMP file and embeds it in any exported files processed from the RAW. digiKam can read this information and embed the data in the RAW image metadata by using Image / Reread Metadata from Image to update the database then Image / Write Metadata to Image to embed it. However this means the data is in actual fact written to more places.

Title is written to:

• IPTC : Object Name
• XMP : Title
• XMP : caption

Description is written to:

• Exif : Image Description
• Exif : User Comment
• IPTC : Caption
• XMP : notes
• XMP : Description
• XMP : User Comment
• XMP : Image Description

Creator is written to:

• Exif : Artist
• IPTC : By-line
• XMP : Creator

Publisher is written to:

• XMP : Publisher

Rights is written to:

• Exif : Copyright
• IPTC : Copyright
• XMP : Rights

UPDATE:

Darktable's behavior appears to have changed. Currently on v3.0.2 the metadata editor in lighttable view writes the following meta data to exported jpg's.

Title is written to:

• xmp.dc.title

Description is written to:

• Exif.Image.ImageDescription
• Xmp.dc.description

Creator is written to:

• Exif.Image.Artist
• Xmp.dc.creator

Publisher is written to:

• Xmp.dc.publisher

Rights is written to:

• Exif.Image.Copyright
• Xmp.dc.rights

A new feature in the export tab (accessed through the cog icon next to the export button is the "edit metadata exportation" dialogue. A bit of a mouth full but a powerful tool none the less, this dialogue allows you to override the metadata editor which when combined with export presets allows the user to create different export settings for different file uses. [

Now the jpeg contains the metadata as set in the export dialogue, though notice how the xmp.dc.creator and xmp.dc.publisher entries have been appended. I believe this is because multiple instances are allowed (gnome image viewer list these as XMP IPTC - dc:creator[1], XMP IPTC dc:creator[2], etc).
If the general settings in "edit metadata" are all switched off the metadata is presented as shown in the dialogue, but almost nothing else is included (no camera settings, creation date or time and so on). note I haven't searched darktable to narrow down tab results this time.

The power of this feature comes in when you use it on a redefined tag with no formula (1), text (2) or one or more of the darktable variables (3). Using no formula prevents output of the meta tag and the variables are listed in the manual pdf (there are more than shown in the hint text).