Transferring images with XMP sidecar files

[Graham Dicker]

Following a hard disk failure I have installed OpenSuse 12.2 on a new hard
drive. This comes with Digikam 2.6.0 and have restored from backup a bunch of
images used with the Digikam version that came with Opensuse 12.1. These
images have captions stored in the XMP sidecar files but the new Digikam does
not display them. How do I tell Digikam to use the existing XMP files please?

Graham Dicker
 
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Volker Henn]

Dear Graham,
seems to be a bug, doesn't work either on my PC with similar configuration.
Volker

Am Sonntag, 3. Februar 2013, 11:53:28 schrieb Graham Dicker:

> Following a hard disk failure I have installed OpenSuse 12.2 on a new hard
> drive. This comes with Digikam 2.6.0 and have restored from backup a bunch
> of images used with the Digikam version that came with Opensuse 12.1. These
> images have captions stored in the XMP sidecar files but the new Digikam
> does not display them. How do I tell Digikam to use the existing XMP files
> please?
>
> Graham Dicker
>
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Sun, 3 Feb 2013, Graham Dicker wrote:

> Following a hard disk failure I have installed OpenSuse 12.2 on
> a new hard drive. This comes with Digikam 2.6.0 and have restored
> from backup a bunch of images used with the Digikam version that
> came with Opensuse 12.1. These images have captions stored in the
> XMP sidecar files but the new Digikam does not display them.
> How do I tell Digikam to use the existing XMP files please?

Dear Graham,

Welcome to the club:-)
I've recently reinstalled my OpenSuse 12.1, shipped with Digikam
2.2, to OpenSuse 12.2, shipped with Digikam 2.6, and of course got
the same issue, previous 2.2 database isn't recognized and DK 2.6
rescans folders from scratch, thus loosing all metadata.

And, as Volker says :

On Sun, 3 Feb 2013, Volker Henn wrote:

> Dear Graham,
> seems to be a bug, doesn't work either on my PC with similar
> configuration.

Re-reading from sidecar files doesn't work. I don't know if this has
been fixed in recent DK versions, I don't know if it's a Digikam
related problem or a problem from the libexiv2 used by DK,
but metadata is lost anyway.

(BTW, we recently had a thread on this list, related to database
troubles and data/metadata integrity. And it's confirmed that this
is an important issue because user data saving (and metadata edition
costs hours of work) accross years, versions changes, etc.,
should be considered an important topic. My personal choice was
to keep metadata in images files, it's the only reliable way as for
today. When rescanning new collections, Digikam imports correctly
XMP data embedded in files.)

Back to your problem. If DK can't read back its sidecar files,
the solution could be to move metadata into the images files.
Phil Harvey's exiftool can do such things but it's not
straightforward at all. Here are some guidelines :

1. I assume you have a full backup of all your collections,
images files, sidecar files et al., just in case:-)

2. Select a test image and do a copy of the associated sidecar file,
say test.xmp

3. Exiftool can rebuild a XMP section from a XMP/RDF xpacket input,
but the sidecar files generated with Digikam (via the libexiv2) are
standard XML files, not xpacket.

4. Open the test.xmp with your favorite text editor. You should get
something like :
<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
     <rdf:RDF
     xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
     <rdf:Description rdf:about=""
     xmlns:digiKam="http://www.digikam.org/ns/1.0/"
     xmlns:xmp="http://ns.adobe.com/xap/1.0/"...

5. Remove/delete the first line, <?xml version... etc

6. Replace by the following :
<?xpacket begin='' id='W5M0MpCehiHzreSzNTczkc9d'?>

NB: replace *exactly*, you can copy/paste from this e-mail.

7. Go to the very end of you file (after the </x:xmpmeta> markup)
and add the following line :
<?xpacket end='w'?>

8. Save and quit the editor.

9. Supposing your image file (related to the sidecar file) is
IMAGE.JPG, run the following command line :
     exiftool '-xmp<=test.xmp' IMAGE.JPG

Warning: the quotes '...' are required to prevent the shell to
interpret the '<' symbol as a redirection. It's a redirection, yes,
but for exiftool not for bash.

10. You have now an image file with an embedded XMP metadata
section, containing all what was in your sidecar file. Restart
Digikam, select the image and, from the 'Edit caption/tags' folder,
at the bottom right, click on the 'More' menu and select "Read
metadata from file to database". Your caption should appear...


Sorry if all the above appears to be a dirty hack. I don't know
other ways to recover lost metadata. Of course, if you have
thousands of files, it's not considerable to do hand text editing
for images, one by one. If you have some skills at shell scripting,
writing a few lines script to do this is really simple, rebuild a
xpacket from the sidecar file and issue the exiftool command.
And, after tests, a script accepting one image/sidecar filename as
argument can be used to process all your collection with tools as
find :
     find /collection-base -iname \*.jpg -exec my-script {} \;
etc.
(If you're not script writer, feel free to contact me on my private
e-mail address, I could send you some stuff.)

Hope this (possibly) could help.
Regards,
Jean-François

PS: GIMP WARNING !!!
This is off topic, yes, but I'd like to warn all GIMP users,
occasionnaly or on a regular basis, that have a recent version of
The Gimp. (And OpenSuse 12.2 comes with a recent version, Gimp 2.8.0.)
Theses versions, 2.8.0 and newer 2.8.2 have a totally *broken* XMP
support.
(When I say totally, it's not once among many times, it's always.
The program can't even re-read what it wrote and starts with error
messages saying that the XMP section is corrupted.)

If you are DK users that refuse to write metadata into the files,
and use only sidecar files or nothing (just the database),
forget this warning.

If you save into the images files and you edit your image with Gimp
2.8+, your metadata will be LOST !

The conclusion : no conclusion:-(
My personal opinion id that NO images management software can be
trusted, wrt metadata, because metadata is not seen as an important
issue (compared to graphical and "seenable" features, special
effects, dozens of plugins, etc.)

My personal workaround : as I remain stuck to the « metadata in
files » idea, and as I like Gimp (and need it for some features I
use regularly, e.g. layers, that few edition software offer aside
Photoshop), I've just implemented a somewhat paranoïd strategy,
keeping backups of my XMP data. Again with exiftool :
     exiftool -fast2 -xmp -b the-imagefilename > the-xmpsave-file
And after Gimp edition (and metadata destruction) I can restore with
the proper command (as show above in item 9.).
In that case, the saved file is ready to be use, it's xpacket format,
not xml format.

Life isn't easy for metadata users... :-)
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Graham Dicker]

On Sunday 03 Feb 2013 16:19:51 Jean-François Rabasse wrote:
>
> Back to your problem. If DK can't read back its sidecar files,
> the solution could be to move metadata into the images files.
> Phil Harvey's exiftool can do such things but it's not
> straightforward at all. Here are some guidelines :

I was using the sidecar files because most of my photos are in a proprietary
raw format and I was unwilling to risk Digikam messing them up.

Thank you for your offer of help but I think I can extract the captions from
the old database and then transfer to another product, even if I have to write
it myself (I don't need anything complicated, just reliable).

Thanks again

Graham Dicker


_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Christoph Siedentop-2]

In response to Jean-François. I also have a workflow on getting meta
data from the xmp file into the image. I think it simpler.

First, Digikam appends the name .xmp after the .jpg. We'll change
that. Put the following in a file and run as a script:

for f in *.jpg.xmp; do
  mv "$f" "${f/.jpg/}"
done

Secondly, insert the data into the image:
exiv2 insert -i X *
So, this does insert (obviously), -i X means, use xmp. * means run on
all images in the folder.

No need to manually change the xmp files.

I hope this helps at least one person.

Regards,
Christoph

On Sun, Feb 3, 2013 at 5:06 PM, Graham Dicker <[hidden email]> wrote:

> On Sunday 03 Feb 2013 16:19:51 Jean-François Rabasse wrote:
>>
>> Back to your problem. If DK can't read back its sidecar files,
>> the solution could be to move metadata into the images files.
>> Phil Harvey's exiftool can do such things but it's not
>> straightforward at all. Here are some guidelines :
>
> I was using the sidecar files because most of my photos are in a proprietary
> raw format and I was unwilling to risk Digikam messing them up.
>
> Thank you for your offer of help but I think I can extract the captions from
> the old database and then transfer to another product, even if I have to write
> it myself (I don't need anything complicated, just reliable).
>
> Thanks again
>
> Graham Dicker
>
>
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

--
Christoph Siedentop
0176-399-422-45
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

Hello Jean-François,

thanks for the warning!
(Gentoo stable GIMP is currently 2.6.12 ;)

My workflow (to keep atleast the important metadata):

  1. Import JPG to DigiKam
  2. Tag / Rate / Assign Caption
    (-> stored in file and DigiKam DB)
  3. Edit with GIMP
    (tested: explicit Saving without Metadata)
  4. all Metadata in the file is gone :(
  5. in DigiKam click "Write Metadata to Image"

This way, at least the metadata (most important to me) is
kept: Tags / Ratings / Captions

But you would loose: Exposure, Focal, Aperture, ...
(Although this data is imported to the DigiKam DB, it seems
to not being written back to the image).

So best way, of course, would be to try not loosing the EXIF
data in the image. ;)

(I use DigiKam 2.9.0 with KDE 4.9.5)

Regards,
        Peter

On 03.02.2013 16:19, Jean-François Rabasse wrote:
> PS: GIMP WARNING !!!
> This is off topic, yes, but I'd like to warn all GIMP users,
> occasionnaly or on a regular basis, that have a recent
> version of
> The Gimp. (And OpenSuse 12.2 comes with a recent version,
> Gimp 2.8.0.)
> Theses versions, 2.8.0 and newer 2.8.2 have a totally
> *broken* XMP
> support.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Volker Henn]

Dear Peter,
do you really overwrite the original JPG file?

Volker  



Am Sonntag, 3. Februar 2013, 20:58:32 schrieb Peter Albrecht:

> Hello Jean-François,
>
> thanks for the warning!
> (Gentoo stable GIMP is currently 2.6.12 ;)
>
> My workflow (to keep atleast the important metadata):
>
>   1. Import JPG to DigiKam
>   2. Tag / Rate / Assign Caption
>     (-> stored in file and DigiKam DB)
>   3. Edit with GIMP

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Sun, 3 Feb 2013, Christoph Siedentop wrote:

> Secondly, insert the data into the image:
> exiv2 insert -i X *
> So, this does insert (obviously), -i X means, use xmp. * means
> run on all images in the folder.
>
> No need to manually change the xmp files.
>
> I hope this helps at least one person.

Hello Christoph, thanks for the hint.

I'm not used to command line exiv2 because of serious problems in the
past (support for Nikon makernotes awfully broken), so I gave the exiftool
variant. But yes, any metadata management program should be able to do
that of processing.
(And recognizing either xml or xpacket format is a plus, indeed.)
And I agree with you, this kind of trick could help, probably more than
one person, for one day or the other we have to face related problems.




On Sun, 3 Feb 2013, Peter Albrecht wrote:

> Hello Jean-François,
>
> thanks for the warning!
> (Gentoo stable GIMP is currently 2.6.12 ; )

Good, but beware of your next O.S. update :-)
I was using GIMP 2.6 too, until recently, and I've to admit the 2.8.2
is great. Except for that silly metadata problem.
(Version 2.8.2 is great, the 2.8.0 coming with my OpenSuse 12.2 is
buggy, I've recompiled 2.8.2 from sources package.)

> My workflow (to keep atleast the important metadata):
>
>    1. Import JPG to DigiKami
>    2. Tag / Rate / Assign Caption (-> stored in file and DigiKam DB)
>    3. Edit with GIMP (tested: explicit Saving without Metadata)
>    4. all Metadata in the file is gone : (
>    5. in DigiKam click "Write Metadata to Image"
>
> This way, at least the metadata (most important to me) is kept:
> Tags / Ratings / Captions
>
> But you would loose: Exposure, Focal, Aperture,... (Although
> this data is imported to the DigiKam DB, it seems to not being
> written back to the image).

Be aware that if you uncheck the « Save metadata », GIMP will update
neither the XMP section nor the Exif section, so you loose all.
Probably it's better to Save with metadata, at least you'll get a valid
Exif section (with the image thumbnail) and a broken XMP section.
And you can then, at your convenience, re write metadata from DK, or
from a saved XMP record, exiftool or exiv2 as Christoph said above.

> So best way, of course, would be to try not loosing the EXIF
> data in the image. ; )

Sounds wise :-)
And probably (or certainly), the safest is to export all and archive
metadata, Exif, XMP et al., along with the image file.
I use exiftool again :
  exiftool -X Image-file
The -X option exports all, JFIF, Exif, XMP, Composite, etc., and even
non standard XMP schemas (as I use non standard metadata, I have
personal XMP namespaces) under a XML format.
I like XML because it's a rock solid data format and I bet such files
will still be useable in 10 or 20 years from now. At least it would ever
be possible to custom any XML parser and recover data.
This, of course, in case of accident.

Also, note that you should consider using Digikam in both writing modes,
« Save metadata to image file AND sidecar file »
The sidecar files format is really complete and you have your XMP data
but also a subset of the most important Exif data, Exposure, Aperture,
Focal length, Creation date, ISO settings, GPS info if any, Camera maker
and model, etc.
So the sidecar file is also a good candidate for data archiving for
a long lifetime.
(And it's 3 to 4 Kbytes, so compared to the size of images files, it
won't explode your archives drives :-)



On Sun, 3 Feb 2013, Volker Henn wrote:

> Dear Peter,
> do you really overwrite the original JPG file?

Hem, shall I dare say I do :-)
(At least my working copy, the original as out of the SD is on a USB drive
and will be removed months later, when I archive final images folders.)

Regards,
Jean-François

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

On 03.02.2013 21:12, Volker Henn wrote:
> Dear Peter,
> do you really overwrite the original JPG file?

Even worse:
After rating all images, I start a batch job, which resizes
all JPGs with "star rating < 4" to 1500 pixel (at the longer
side) and 85 JPG-quality. This way most fotos do take about
200 KByte on my harddisc. ;)
Since I mostly want to "document" events, this is sufficient
for me.

The images with "rating >= 4", are kept in original size.
But I do brightness/color adjustment and other things on the
original files. So the real original JPG is gone in the end.

Of course I keep a few backups along this process!

Regards,
        Peter
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

> On Sun, 3 Feb 2013, Peter Albrecht wrote:
>>
>> thanks for the warning!
>> (Gentoo stable GIMP is currently 2.6.12 ; )
>
> (Version 2.8.2 is great, the 2.8.0 coming with my OpenSuse
> 12.2 is buggy, I've recompiled 2.8.2 from sources package.)

So I hope, it is an "OpenSuse-Bug". But I keep my eyes open.

>> My workflow (to keep atleast the important metadata):
>>
>>    1. Import JPG to DigiKami
>>    2. Tag / Rate / Assign Caption (-> stored in file and
>> DigiKam DB)
>>    3. Edit with GIMP (tested: explicit Saving without
>> Metadata)
>>    4. all Metadata in the file is gone : (
>>    5. in DigiKam click "Write Metadata to Image"
> Be aware that if you uncheck the « Save metadata », GIMP
> will update
> neither the XMP section nor the Exif section, so you loose all.

I used "Saving without Metadata" just to simulate the buggy
GIMP behaviour. In my real workflow, I keep "Save metadata"
checked.


> And probably (or certainly), the safest is to export all and
> archive
> metadata, Exif, XMP et al., along with the image file.
> I use exiftool again :
>  exiftool -X Image-file
> The -X option exports all, JFIF, Exif, XMP, Composite, etc.,
> and even
> non standard XMP schemas (as I use non standard metadata, I
> have personal XMP namespaces) under a XML format.

Thanks for the tip didn't know about the "-X" option.


> I like XML because it's a rock solid data format and I bet
> such files
> will still be useable in 10 or 20 years from now. At least
> it would ever
> be possible to custom any XML parser and recover data.
> This, of course, in case of accident.

This is, way I love XML, too.
But I hope, I never have to recover data with a self
programmed XML parser. ;)

Regards,
        Peter
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users