digiKam › digikam-users

Digikam XMP schema ?

Classic

List

Threaded

23 messages Options

Jean-François Rabasse

Digikam XMP schema ?

Hello,

I've a question, probably to the Digikam developpers but maybe
someone else knows...

I have a problem with Digikam XMP metadata, specially with the
tags list.
The libexiv2 (used by DK) processes this property as an ordered
list, i.e. in RDF representation a <rdf:Seq>
Other software (exiftool, the Gimp) process as an unordered list,
a <rdf:Bag>.

I don't know which is right or wrong (even if imho it should be
unordered, as does LightRoom for the HierarchicalSubject list)
but when an image file has « travelled » between different software,
the XMP metadata becomes a bit weird and can contain duplicates.
E.g. :
<digiKam:TagsList>
<rdf:Seq>
Here the list of my tags in <rdf:li> ... </rdf:li>
</rdf:Seq>
</digiKam:TagsList>
<digiKam:TagsList>
<rdf:Bag>
Here exactly the same list as above.
</rdf:Bag>
</digiKam:TagsList>

And with that, interoperability is really broken. Libexiv2 and thus
applications based on libexiv2 as Digikam, rejects input because of the
duplication.
(And other applications, e.g. exiftool, don't reject but concatenate the
two properties, leading to all list items being redoubled.)

** My question : where can I find an official, authoritative definition
of Digikam XMP schema ? (In order to know on which forum to post the
problem, exiv2 ? exiftool ? other ?)

(Tried the declared URI xmlns:digiKam="http://www.digikam.org/ns/1.0/"
but I get a « Page not found » message, instead of a possible and
expected documentation.)

Thanks in advance,
Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Gilles Caulier-4

Re: Digikam XMP schema ?

Hi Jean Francois,

The schema is defined as well in Exiv2. I do it personalty.

An ordered list sound logical for me. Why un-ordered one ?

The digiKam schema is a private stuff. If a 3rd party don't play properly with it, this is not digiKam fault. As you said : "Other software (exiftool, the Gimp) process as an unordered list, a <rdf:Bag>."

Why this software do not respect how digiKam store metadata ?

Gilles Caulier

2013/2/12 Jean-François Rabasse <[hidden email]>

Hello,

I've a question, probably to the Digikam developpers but maybe
someone else knows...

I have a problem with Digikam XMP metadata, specially with the
tags list.
The libexiv2 (used by DK) processes this property as an ordered
list, i.e. in RDF representation a <rdf:Seq>
Other software (exiftool, the Gimp) process as an unordered list,
a <rdf:Bag>.

I don't know which is right or wrong (even if imho it should be
unordered, as does LightRoom for the HierarchicalSubject list)
but when an image file has « travelled » between different software,
the XMP metadata becomes a bit weird and can contain duplicates.
E.g. :
<digiKam:TagsList>
<rdf:Seq>
Here the list of my tags in <rdf:li> ... </rdf:li>
</rdf:Seq>
</digiKam:TagsList>
<digiKam:TagsList>
<rdf:Bag>
Here exactly the same list as above.
</rdf:Bag>
</digiKam:TagsList>

And with that, interoperability is really broken. Libexiv2 and thus applications based on libexiv2 as Digikam, rejects input because of the duplication.
(And other applications, e.g. exiftool, don't reject but concatenate the two properties, leading to all list items being redoubled.)

** My question : where can I find an official, authoritative definition
of Digikam XMP schema ? (In order to know on which forum to post the
problem, exiv2 ? exiftool ? other ?)

(Tried the declared URI xmlns:digiKam="http://www.digikam.org/ns/1.0/"
but I get a « Page not found » message, instead of a possible and expected documentation.)

Thanks in advance,
Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Jean-François Rabasse

Re: Digikam XMP schema ?

On Tue, 12 Feb 2013, Gilles Caulier wrote:

> Hi Jean Francois,
> The schema is defined as well in Exiv2. I do it personalty.

Hi Gilles, thanks for the answer.
Ok, so you are the « connecting link » between Digikam and exiv2 :-)
I didn't know, and the problem I raised seems to indicate other
people didn't know (Phil Harvey et al.)

But, ok for rdf:Seq

> An ordered list sound logical for me. Why un-ordered one ?

It's a matter of taste. The RDF standards tend to indicate that ordered
arrays should wrap array items with some hierarchical order, the 1st
item, then the 2nd, the 3rd, etc. (As is done for the DCMI dc:creator
with 1st item is the major creator, then other contributors.)
For images tags, it's not obvious to settle a hierarchy.
But that's another issue, the reference is the schema designer(s).

> The digiKam schema is a private stuff. If a 3rd party don't play properly
> with it, this is not digiKam fault. As you said : "Other software (exiftool,
> the Gimp) process as an unordered list, a <rdf:Bag>."

I never said it was Digikam fault :-)
For me the problem was different interpretations between various metadata
handlers, libexiv2, exiftool et al. And that's why I was looking for THE
reference. You gave it above.

> Why this software do not respect how digiKam store metadata ?

Perhaps because the official reference isn't easy to find. It's obvious
for you 'cause you work on both exiv2 and DK, but it's not that obvious
for other people (beginning with me).
I think people from other software may have reproduced the LightRoom
HierachicalSubjet because the semantics are really close, aprt from the
tree components separator, '/' or '|'.
Perhaps (a suggestion) the exiv2 web site should indicate that « This is
the official Digikam schema », and/or the DK web site could have a link
to the related exiv2 page.

You're right, private schemas are private stuff, but for interoperability
reasons, metadata handling implementations want to be compatible with the
most possible data.
And even Digikam, when writing via libexiv2, stores the tagslists in
digiKam:Tagslist, yes, but also in lr:hierarchicalSubject (private stuff
from Adobe) and also MicrosoftPhot:LastKeywordXMP (private stuff from
the firm in Seattle).
Compatibility is a great thing and should be pushed as far as possible
and it's important that developpers could have the authoritative information
under hand.

(I'm in that case, writing personal tools to manage my metadata for
special needs, and before reading your answer, above, I was unable to
choose between rdf:Seg/rdf:Bag implementation.
This when producing XMP streams; reading should not be a problem if the
reader is prepare to acommodate known variants.)

Thanks for the answer anyway,
Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

tosca

Re: Digikam XMP schema ?

In reply to this post by Gilles Caulier-4

Could this ordered/unordered thing be related to the fact that tags are a complete mess (in fact, I don't manage tags anymore, waiting for a solution for that problem) when one tries to reorganize the tags hierarchy?

Marie-Noëlle

2013/2/12 Gilles Caulier <[hidden email]>

Hi Jean Francois,

An ordered list sound logical for me. Why un-ordered one ?

Gilles Caulier

2013/2/12 Jean-François Rabasse <[hidden email]>

Hello,

I don't know which is right or wrong (even if imho it should be
unordered, as does LightRoom for the HierarchicalSubject list)

Jean-François

Mes dernières photos sont dans ma galerie.
Connaissez-vous Image Fixe, le photo-club de Saint Jean du Gard ?
Et parcourez les Cévennes à ma façon avec Cévennes Plurielles,

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

jdd@dodin.org

Re: Digikam XMP schema ?

Le 17/02/2013 09:24, Marie-Noëlle Augendre a écrit :
> Could this ordered/unordered thing be related to the fact that tags
> are a complete mess (in fact, I don't manage tags anymore, waiting for
> a solution for that problem) when one tries to reorganize the tags
> hierarchy?

I use tags, but no hierachy :-)

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

tosca

Re: Digikam XMP schema ?

I need the hierarchy to separate different kinds of tags (who/where/ and the like) and to structure a great number of tags, especially those related to location that require 5 levels or more, depending of the area.

Marie-Noëlle

2013/2/17 jdd <[hidden email]>

I use tags, but no hierachy :-)

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

jdd@dodin.org

Re: Digikam XMP schema ?

Le 17/02/2013 09:36, Marie-Noëlle Augendre a écrit :
> I need the hierarchy to separate different kinds of tags (who/where/
> and the like) and to structure a great number of tags, especially
> those related to location that require 5 levels or more, depending of
> the area.

I agree that hierarchy would be great, but normalisation is mandatory,
hierarchy have to but managed the same way by all software.

It's the same for sidecars, if the file format is not the same for
all, it's useless.

For example, when piwigo changes a tag, the change is *not* written to
the image (nor the sidecar), so digikam can't recover it.

I'm pretty sure a script could recover one software data and convert
it to the other, but I'm a bit short of time and knowledge to write it
myself :-(

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

tosca

Re: Digikam XMP schema ?

Says who? ;-)

I only use Digikam to manage the pictures, so I don't mind if it has a proprietary model.
I use Darktable to "develop" the RAW pictures, but never use its management functions; my only purpose is to produce the .TIFF pictures that will go back in Digikam.

My local/own Digikam database must remain the "master", and every other storage is to be a "salve"; so I might get rid of tags that are for my own usage and I don't want to appear either in Piwigo/Picasa/whatever, but this mustn't modify whatever tag I use in Digikam.

Marie-Noëlle

2013/2/17 jdd <[hidden email]>

I agree that hierarchy would be great, but normalisation is mandatory, hierarchy have to but managed the same way by all software.

jdd@dodin.org

Re: Digikam XMP schema ?

Le 17/02/2013 09:55, Marie-Noëlle Augendre a écrit :

> My local/own Digikam database must remain the "master", and every
> other storage is to be a "salve"; so I might get rid of tags that are
> for my own usage and I don't want to appear either in
> Piwigo/Picasa/whatever, but this mustn't modify whatever tag I use in
> Digikam.

well, you can't know what you will do in the future, and yes for me
too Digikam have to be the master, but when I make a local change
(digikam), I would like to have it copied online (and vice-versa),
without having to copy back and forth the whole images.

that don't mean digikam don't have to be first consistent with himself
:-), but sicdecar files is a very convenient way to share data.

databases are good for speed, but a nightmare to backup (and restore!)

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Jean-François Rabasse

Re: Digikam XMP schema ?

In reply to this post by tosca

Hello Marie-Noëlle,

On Sun, 17 Feb 2013, Marie-Noëlle Augendre wrote:

> Could this ordered/unordered thing be related to the fact that tags are a
> complete mess (in fact, I don't manage tags anymore, waiting for a solution
> for that problem) when one tries to reorganize the tags hierarchy?

Not at all. This distinction has no technical implications, it's only a
matter of semantics when using metadata values.
Metadata properties may be of simple type, one name + one value, or list
type, one name + one or more values. And in the XMP standard, lists are
implemented exactly the same way. It's us to users and applications to
consider the values as unordered (a « bag » in RDF terminology), or to
consider ordered values (a « seq » in RDF terminology) with the 1st value
being more important than the second, the second value more important
than the third, etc.

And managing ordered lists from within an application would imply the
application interface offers a way to indicate how a newly added value
should be sorted among existing values, « add this in first place, or at
the end, etc. » And this is not what seems to be done with DK tagslists.

As for me, if I have two tags on an image, one to indicate the place,
e.g. Location/France/Paris, and one to indicate an image context,
e.g. Context/Urban/Street Art, I don't see reasons - for me - to say
the first tag is more/less important than the second.

As for the reorganisation problems you point out, my opinion is that it
comes from the way DK writes/read back tags between the images and the
database. When writing, all tags associated to the image (in the
database) are written. When reading, all tags found in the image are
added to the associated tags in the database. But existing tags (in DB)
are kept even if they no longer are in the image.
The process isn't symmetric, and that's why reorganisation is painful.

Probably the good approach is to reorganize from outside Digikam, with
some metadata edition tools, then delete all tags in Digikam and
re-read metadata from all images. This guarantees the DB will reflect
the images content.

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

jdd@dodin.org

Re: Digikam XMP schema ?

Le 17/02/2013 10:34, Jean-François Rabasse a écrit :

> Probably the good approach is to reorganize from outside Digikam, with
> some metadata edition tools, then delete all tags in Digikam and
> re-read metadata from all images. This guarantees the DB will reflect
> the images content.

there is an other way, remove the database entirely and build a new one.

but this implies *all* the metadata are written to the image or the an
xml file

what I would like is the following:

write *all* the digikam image data to xml sidecar files, this being
the safest way to archive them (can you archive the database?)

remove entirely the database

remove is necessary images that don't have to be online (only archived)

rebuild a new data base from the xml files

by the way, what is faster, sqllite (old database format) or mysql?

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

tosca

Re: Digikam XMP schema ?

In reply to this post by Jean-François Rabasse

Thanks for your explanations.

As for the reorganization, the way I see things is not only one huge operation. Every time I enter Digikam, I would like to be able to do small editing; for example, if a tag created previously is affected to too many items, that means I could need to refine the analyze and add one more level to get more details (especially for localization where I typically have a 'rest of the world' tag, but it goes to the village or even the hamlet in the area I live).

My patience is not yet exhausted, I'm still waiting for the developers to fix this issue before going for 'the' reorganization; then, I should be able to manage the tags on a day-to-day basis. As for now, I don't bother with the tags: at least, I'm not adding to the existing mess...

Marie-Noëlle

2013/2/17 Jean-François Rabasse <[hidden email]>

Probably the good approach is to reorganize from outside Digikam, with
some metadata edition tools, then delete all tags in Digikam and
re-read metadata from all images. This guarantees the DB will reflect
the images content.

Regards,
Jean-François

Volker Henn

Re: Digikam XMP schema ?

In reply to this post by jdd@dodin.org

>
> by the way, what is faster, sqllite (old database format) or mysql?
>
> jdd

mysql is much faster, especially when moving files from one directory to
another.

Volker
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

tosca

Re: Digikam XMP schema ?

But is it as well supported as SQLite3 by Digikam? I'm less concerned with speed than reliability and 'comfort of use' (read: I'm here to work on my pictures, not to administrate a database and fix new problems).

Marie-Noëlle

2013/2/17 Volker Henn <[hidden email]>

>
> by the way, what is faster, sqllite (old database format) or mysql?
>
> jdd

Jean-François Rabasse

Re: Digikam XMP schema ?

In reply to this post by jdd@dodin.org

On Sun, 17 Feb 2013, jdd wrote:

>> Probably the good approach is to reorganize from outside Digikam, with
>> some metadata edition tools, then delete all tags in Digikam and
>> re-read metadata from all images. This guarantees the DB will reflect
>> the images content.
>
>
> there is an other way, remove the database entirely and build a new one.
>
> but this implies *all* the metadata are written to the image or the an xml
> file

Yes :-)
Not all Digikam versions seem to read the sidecar files (mine, 2.6.0,
doesn't). But it's possible to write metadata (sidecar files content) into
the image file, for Dk to read it.

This has been discussed here recently, cf. :

> Date: Sun, 3 Feb 2013 19:14:17 +0100
> From: Christoph Siedentop <[hidden email]>
> Subject: Re: [Digikam-users] Transferring images with XMP sidecar files
>
> In response to Jean-François. I also have a workflow on getting meta
> data from the xmp file into the image. I think it simpler.
>
> First, Digikam appends the name .xmp after the .jpg. We'll change
> that. Put the following in a file and run as a script:
>
> for f in *.jpg.xmp; do
> mv "$f" "${f/.jpg/}"
> done
>
> Secondly, insert the data into the image:
> exiv2 insert -i X *
> So, this does insert (obviously), -i X means, use xmp. * means run on
> all images in the folder.

> write *all* the digikam image data to xml sidecar files, this being the
> safest way to archive them (can you archive the database?)

I agree. Sidecar files are under a standard XML format and will remain
readable (or translatable) forever. Applications specific databases
formats have no guarantee to be usable in 10 or 20 years from now.
Another argument in favor of archiving metadata as XML files is to have
a protection against buggy applications that corrupt metadata.
(I mentionned some versions of The Gimp). So, sidecar files is both a
long lifetime archiving system and a security backup for images metadata.

> by the way, what is faster, sqllite (old database format) or mysql?

Probably not a software issue but an installation issue. Applications
using databases run faster when the DB server is on a separate machine
connected via a high speed network (Gigabit ethernet or IP on Infiniband)
because the CPU running the application is/are relaxed from the SLQ
request (heavy) processing.
But if your MySQL server runs on the same computer, the difference with
SQLlite will be small. (And in favor of SQLite because it features less
things, so does a lighter processing.)

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Volker Henn

Re: Digikam XMP schema ?

Am Sonntag, 17. Februar 2013, 11:06:51 schrieb Jean-François Rabasse:

> > by the way, what is faster, sqllite (old database format) or mysql?
>
> Probably not a software issue but an installation issue. Applications
> using databases run faster when the DB server is on a separate machine
> connected via a high speed network (Gigabit ethernet or IP on Infiniband)
> because the CPU running the application is/are relaxed from the SLQ
> request (heavy) processing.
> But if your MySQL server runs on the same computer, the difference with
> SQLlite will be small. (And in favor of SQLite because it features less
> things, so does a lighter processing.)
>
> Regards,
> Jean-François

Hi, mysql is on the same machine.

With sqlite I obseverd that obviously (no guarantee) when a single change is
made, the whole database is written to hard disk. Then read again, next change
made, completely written again. This take lot of time, since I organize my
pictures in directories instead with tags, so I always shift a lot of files.
Mysql seems to have a better chache strategy. Nevertheless there are problems
with DK / mysql and the database user must be root.

Regards Volker
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

jdd@dodin.org

Re: Digikam XMP schema ?

In reply to this post by Volker Henn

Le 17/02/2013 11:01, Volker Henn a écrit :
>
>>
>> by the way, what is faster, sqllite (old database format) or mysql?
>>
>> jdd
>
>
> mysql is much faster, especially when moving files from one directory to
> another.

thanks
jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

jdd@dodin.org

Re: Digikam XMP schema ?

In reply to this post by tosca

Le 17/02/2013 10:51, Marie-Noëlle Augendre a écrit :
> Thanks for your explanations.
>
> As for the reorganization, the way I see things is not only one huge
> operation. Every time I enter Digikam, I would like to be able to do
> small editing; for example, if a tag created previously is affected to
> too many items, that means I could need to refine the analyze and add
> one more level to get more details (especially for localization where
> I typically have a 'rest of the world' tag, but it goes to the village
> or even the hamlet in the area I live).

yes

what I do now is using only one level and longer tag names. Not
optimal :-(

but I use mostly tags from piwigo online

http://dodin.org/piwigo/tags.php

jdd

--
http://www.dodin.org
http://jddtube.dodin.org/20120616-52-highway_v1115
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Jean-François Rabasse

Re: Digikam XMP schema ?

In reply to this post by Volker Henn

Hi Volker,

On Sun, 17 Feb 2013, Volker Henn wrote:

> Hi, mysql is on the same machine.
>
> With sqlite I obseverd that obviously (no guarantee) when a single change is
> made, the whole database is written to hard disk. Then read again, next change
> made, completely written again. This take lot of time, since I organize my
> pictures in directories instead with tags, so I always shift a lot of files.
> Mysql seems to have a better chache strategy. Nevertheless there are problems
> with DK / mysql and the database user must be root.

Well, in that case you're right. Doing important reorganisations leads to
important database activity. And SQLite3 is a light weight library without
all caching ans requests journaling strategies real RDBMS have.
In my previous comment, I intended mostly small actions (rating or tagging
an image, etc.)

For large operations (folders reorganisation, rereading metadata for many
images, etc.) it's slow and I also happened to have problems, crashes or
so. Not to be nervous, I took the habit to do that on a copy of database,
and I/O access to the SQLite file can be enhanced when using a ramdisk.
Linux supports that very well, so :
- backup /.../digikam4.db to ... digikam3.db.backup
- copy the file to ramdisk, /dev/shm/digikam4.db
- start DK with option : --database-directory /dev/shm
- work...
- when all is ok, satisfying results, no corruptions, no crashes, copy
the ramdisk file back to original place /.../digikam4.db
(Before shutting down the computer:-)
In case of problem, just forget, nothing is lost.

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Volker Henn

Re: Digikam XMP schema ?

Hi Jean-François
that's a good idea. I will keep that in mind when upgrading to Suse 12.3 (what
is almost the most critical thing for the picture database)

Regards Volker

Am Sonntag, 17. Februar 2013, 11:56:20 schrieb Jean-François Rabasse:

> Hi Volker,
>
> On Sun, 17 Feb 2013, Volker Henn wrote:
> > Hi, mysql is on the same machine.
> >
> > With sqlite I obseverd that obviously (no guarantee) when a single change
> > is made, the whole database is written to hard disk. Then read again,
> > next change made, completely written again. This take lot of time, since
> > I organize my pictures in directories instead with tags, so I always
> > shift a lot of files. Mysql seems to have a better chache strategy.
> > Nevertheless there are problems with DK / mysql and the database user
> > must be root.
>
> Well, in that case you're right. Doing important reorganisations leads to
> important database activity. And SQLite3 is a light weight library without
> all caching ans requests journaling strategies real RDBMS have.
> In my previous comment, I intended mostly small actions (rating or tagging
> an image, etc.)
>
> For large operations (folders reorganisation, rereading metadata for many
> images, etc.) it's slow and I also happened to have problems, crashes or
> so. Not to be nervous, I took the habit to do that on a copy of database,
> and I/O access to the SQLite file can be enhanced when using a ramdisk.
> Linux supports that very well, so :
> - backup /.../digikam4.db to ... digikam3.db.backup
> - copy the file to ramdisk, /dev/shm/digikam4.db
> - start DK with option : --database-directory /dev/shm
> - work...
> - when all is ok, satisfying results, no corruptions, no crashes, copy
> the ramdisk file back to original place /.../digikam4.db
> (Before shutting down the computer:-)
> In case of problem, just forget, nothing is lost.
>
> Regards,
> Jean-François

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users