digiKam › digikam-users

[digiKam-users] Keyword (metadata) consistency between programs

Classic

List

Threaded

15 messages Options

AlanH

[digiKam-users] Keyword (metadata) consistency between programs

I would very much like to use digiKam as a DAM for my photo collection. However, I'm finding problems with the interchange and display of metadata between processing software used in the past and digiKam (and going forward as I curate a growing collection). Looking at just keywords, I thought my problems would be largely answered when I found the configuration option to "Use a compatible file name for writing sidecar files". In the case of On1 this produces the On1 sidecar format of a ".on1" sidecar (supposedly for jpg photos, but see below) and ".xmp"sidecar (supposedly for raw photos, but not apparent in observations below).

If I am doing something incorrectly please let me know. If this is a program issue, is there anything that can be done to fix it (and when)?

Create sidecars in On1 (Windows 2020.1)
1)Adding a keyword to a jpg photo in On1 creates an .on1 sidecar. Appears to have both a jpg and raw (rw2) tag block (is this correct term?). Keyword only added to jpg block. On1 display jpg photo keyword correctly.
2)Adding a keyword to a raw photo in On1 creates a .xmp sidecar. Has 2 keyword tag blocks: lr:hierarchicalSubject & dc:subject, both containing the keyword. Also adds keyword to raw block in .on1 sidecar. On1 displays raw photo keyword correctly.
3)digiKam, after "Reread Metadata From File" separately for both jpg and raw photos, displays raw photo keyword for both the jpg and raw (rw2) photos.

Create sidecars in digiKam (Windows 6.4.0). (Config: write to XMP sidecar only; Use compatible filenames)
1)Adding a keyword to a jpg photo in dK creates an .on1 sidecar and .xmp simultaneously. The jpg photo keyword is added to both the jpg and raw (rw2) tag block in the .on1 sidecar as well as the .xmp sidecar. The .xmp sidecar has 6 keyword tag blocks: Acdsee, MicrosoftPhoto:LastKeywordXMP, digiKam:TagsList, lr:hierarchicalSubject, mediapro:CatalogSets & dc:subject, all containing the keyword. !!Adds a jpg photo keyword to a .xmp sidecar used (I think) by On1 for only raw photo metadata. On1 2020 displays the jpg photo for both the jpg and raw (rw2) photos.
2)Adding a keyword to a raw photo in dK (after .on1 & .xmp created above) -- the .on1 sidecar still has the original jpg keyword in both the jpg and raw blocks. In the .xmp sidecar the jpg photo keyword is replaced by the raw photo keyword in all 6 blocks. On1 2020 now displays the raw photo keyword for both the jpg and raw photo.

This behavior becomes more scrambled and confusing with repeated keyword additions when switching between one program and the other (digiKam and On1 in this case). In particular hierarchical keywords are parsed into separate component keywords as processed (I think; at least as visibly evident) in the dc:subject block.

Thanks for your assistance with this inquiry.

Alan

woenx

Re: Keyword (metadata) consistency between programs

Ok, this is a bit complex, so I am not sure if I'll miss something.

Basically you want to be able to interoperate between ON1 photo editor, and
Digikam, and only using sidecar files.

As far as I know, .on1 sidecars are exclusive to the ON1 photo editor, so I
don't think Digikam is able to read or produce them. Apparently, .on1 files
not only include metadata (like keywords and dates) but also information
about the editions done on the photo. So if you edit some keywords in
digikam, store them in a .xmp, and then ON1 reads only from the .on1 file,
it won't see any changes.

I am not sure about the correct blocks that jpg and RAW photos should or
should not. I know that digikam uses a bunch of redundant keyword tags in
order to be compatible with lots of other picture managers, and usually that
shouldn't be a problem, so I wouldn't be worried if you see a JPG and RW2
tag block (as long as there isn't contradictory information).

Moreover, the information of the XMP sidecar can also be embedded in the JPG
picture itself (but not the RAW, usually), so digikam and other softwares
can read that information without the need of a xmp sidecar. If there is
contradictory information between both, I am not sure to which one Digikam
will give priority.

Can you attach us an example of a picture and its sidecars (.xmp and .on1)
so I can see and try for myself?

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

AlanH

Re: Keyword (metadata) consistency between programs

I’ll create a Dropbox link with 3 sets of files and some explanation and send the link via your email.

Thanks

Sent from Mail for Windows 10

From: [hidden email]
Sent: Tuesday, April 28, 2020 12:43 PM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

AlanH

Re: Keyword (metadata) consistency between programs

I was thinking that given the configuration option “use a compatible file name for writing sidecar files”, and digiKam creating files using the extensions (.on1 & .xmp) that ON1 uses, that the sidecars would be interchangeable. The internal structure of the files even appears to be similar between what digiKam creates and ON1 creates (to an end-user, perhaps not a developer/programmer).

Unfortunately it seems that contradictory metadata is written. As you may see in the Dropbox files jpeg metadata is overwritten by raw metadata in what are suppose to be jpeg blocks. Or, metadata is not updated without applying a very unintuitive process. Raw photo metadata can therefore be embedded in jpeg photos.

Perhaps there could be more detailed explanation by digiKam what was meant by “Use a compatible file name….” since the way I thought it worked does not appear to be correct, or I’m not using it correctly. Or maybe ON1 has advanced the structure of their sidecar files beyond what digiKam has programmed.

Any insight appreciated. Thanks.

Sent from Mail for Windows 10

From: [hidden email]
Sent: Tuesday, April 28, 2020 1:59 PM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

I’ll create a Dropbox link with 3 sets of files and some explanation and send the link via your email.

Thanks

Sent from Mail for Windows 10

From: [hidden email]
Sent: Tuesday, April 28, 2020 12:43 PM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

Remco Viëtor

Re: Keyword (metadata) consistency between programs

On mardi 28 avril 2020 23:45:41 CEST Alan H wrote:
> I was thinking that given the configuration option “use a compatible file
> name for writing sidecar files”, and digiKam creating files using the
> extensions (.on1 & .xmp) that ON1 uses, that the sidecars would be
> interchangeable. The internal structure of the files even appears to be
> similar between what digiKam creates and ON1 creates (to an end-user,
> perhaps not a developer/programmer).

> Unfortunately it seems that contradictory metadata is written. As you may
> see in the Dropbox files jpeg metadata is overwritten by raw metadata in
> what are suppose to be jpeg blocks. Or, metadata is not updated without
> applying a very unintuitive process. Raw photo metadata can therefore be
> embedded in jpeg photos.
>
There's syntax (which XML fields are written) and semantics (what's the exact
meaning of the contents). The syntax is declared in the namespace definitions,
not the semantics. So if ON1 uses jpeg blocks and raw blocks in one file,
digikam may not realise those can legitimately hold different info.

> Perhaps there could be more detailed explanation by digiKam what was meant
> by “Use a compatible file name….” since the way I thought it worked does
> not appear to be correct, or I’m not using it correctly. Or maybe ON1 has
> advanced the structure of their sidecar files beyond what digiKam has
> programmed.

"Compatible file name for sidecar files" means basically "use this extra
extension for XMP files". The internal structure will be the same for all
sidecars (it's derived from XML). Available tags may differ, and on reading
digikam should ignore tags it doesn't know about.

Remco

P.S. Referring here to files you apparently want to keep private isn't
particularly helpful

AlanH

Re: Keyword (metadata) consistency between programs

Remco, and other viewers, apologies for not including the link to the files. Nothing that cannot be shared. Don’t know what I was thinking sending it just to Woenx. The more that can think about this the better. Here’s the link:

https://www.dropbox.com/sh/9x7heiuo6h492xb/AAB8tvDi6Or0_XLqcIK-ggCpa?dl=0

Thanks for the explanation. I can probably come up with some sort of kludgy solution to work around this since I really want to use digiKam as my DAM. There are times though that the processing tools of other apps suit my needs and familiarity better, such as the brush and masking tools in ON1. I’m about to go ahead with curating 30,000+ digitized slides and it would be very helpful to be able to search on ratings, color codes, keywords and other metadata added or changed (eg date taken) in digiKam reliably when in other apps and I want to find and process a particular photo.

As a curiosity, though, would it be a complex and/or a lengthy task for digiKam to create sidecars completely compatible with sidecars used by other apps? (possibly an industry first, aside from the snakepit of which apps to choose). I only ask because to my untrained eye the ON1 sidecar (“filename.on1”) created by digiKam appears to be very similar in format to the sidecar that ON1 creates, when “Use a compatible file name…” is checked. It is quite distinct from the (normal looking?) .xmp sidecar that digiKam (and most other apps) creates when that option is unchecked. From my view of the sidecars, the problem appears to be an inconsistency/confusion in whether the metadata tag belongs to a jpg or raw file and then writing it into the correct block. The blocks seem to be there, just not properly interpreted and applied. Could digiKam be programmed (and with what effort) to realize that the blocks legitimately hold different info?

Alan

Sent from Mail for Windows 10

From: [hidden email]
Sent: Wednesday, April 29, 2020 12:14 AM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

woenx

Re: Keyword (metadata) consistency between programs

Hi Alan,

I wanted to try ON1 to see if I can replicate your experience. In their
website they have several software packages to download and try:

INTEGRATED WORKFLOW
ON1 360°
ON1 Photo RAW 2020.1
ON1 Photo Mobile 2020

INDIVIDUAL PRODUCTS
ON1 HDR 2020
ON1 Resize 2020
ON1 Effects 2020
ON1 Video 2020

MEMBERSHIPS
ON1 Plus

Which one do you use?

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

AlanH

Re: Keyword (metadata) consistency between programs

Hi,

Thanks for taking a look. I’m using ON1 Photo RAW 2020.1 (Windows).

Sent from Mail for Windows 10

From: [hidden email]
Sent: Saturday, May 2, 2020 11:08 AM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

woenx

Re: Keyword (metadata) consistency between programs

Ok, so I installed both (On1 and Digikam) in a windows machine. I told
digikam to read and save all data to .xmp sidecars. I also told ON1 not to
save metadata and edits to .on1 files.

I first wanted to test the most basic assumption, that both can write and
read from sidecar files, so they can exchange information.

For that purpose, I created a directory which will be the picture library
for both Softwares, and I put a simple JPG picture I got online (with no
metadata).

If I edit that picture in digikam and add a keyword, I can see it generates
a .xmp file and the keyword is there, under a number of sections
<digiKam:TagsList>, <MicrosoftPhoto:LastKeywordXMP>,
<lr:hierarchicalSubject>, <mediapro:CatalogSets> and <dc:subject>.

However, if I take that same picture (the original one without metadata) and
I add a keyword, no .xmp sidecar is generated (and the metadata is not
embedded in the picture either). Moreover, it seems that the next time you
start ON1, it activates the saving to .on1 files again automatically.

If I add a keyword from ON1, digikam does not seem to see it. And
vice-versa.

So, how do you tell ON1 to store the metadata in a .xmp sidecar file?

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

AlanH

Re: Keyword (metadata) consistency between programs

Just for a bit of clarity with respect to my experience…

-the 4 th paragraph (If I edit that picture in digicam…): did digicam create the .xmp file in the format “filename.jpg.xmp”?

-the 5^th paragraph (However, if I…): were you in ON1 when adding the keyword? Agreed on the second statement. ON1 appears to write JPG metadata (your last question) to its proprietary “filename.on1” sidecar file and creates these files automatically for all JPGs the moment any file in a directory is opened.

When I saw the configuration in digikiam to “Use a compatible file name for writing to sidecar files” and it appeared to create a sidecar in a format very similar to ON1’s, I thought I had at last found two apps for which the metadata sidecars would be interchangeable. It seems though that there is still some confusion about how they process each others sidecars.

It gets more confusing when .raw files come into the mix as I tried to show in previous posts. On1 writes a raw file’s metadata to a .raw sidecar but also seems to include (some?) raw metadata in the .on1 sidecar file, along with all of the parametric commands for whatever non-destructive edits are done. Again, in my experience, if the .on1 sidecar was, in particular, created by digikam both apps have difficulty writing/reading/parsing the JPG and RAW blocks appropriately.

Regarding the last question, I have opened a ticket with ON1 somewhat on that subject – how to get ON1 to read digikam’s filename.jpg.xmp and filename.raw.xmp files. Tech support got back to me and said they had to pass it up to their developers since they did not have an answer. I’ll try to steer the inquiry to writing to “proper” xmp sidecars as well. I’ll pass that info along when (hopefully) I have it.

This isn’t just a digicam vs ON1 issue. I have found sidecar compatibility issues between other popular photo processors – Luminar, DXO PL, Exposure, Affinity – that are trying to provide DAMs (or even just non-destructive editing sidecars). It is such a contentious issue in DXO that the user forum regularly advocates staying away from the DAM game and stick to what they do best.

Sent from Mail for Windows 10

From: [hidden email]
Sent: Tuesday, May 5, 2020 10:10 AM
To: [hidden email]
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

woenx

Re: Keyword (metadata) consistency between programs

- If you check the "sidecar file name compatible with commercial software"
option, the sidecar filename is "filename.xmp". Otherwise is
"filename.jpg.xmp".

- Yes, I was referring to ON1. I couldn't make it to create a XMP sidecar,
so Digikam can read it. Maybe there's an option, but I didn't find it
quickly.

Regarding RAW files, I guess it's normal, since they are not meant to be
edited, so all changes will be saved in sidecar files (I believe this is how
digikam works for RAW files). I still have not tried if ON1 can generate XMP
files for sidecars, and, if that's the case, they can be read by digikam.

There's also the question if ON1 can also read keywords saved directly
inside the picture file (that's how I do it, under the risk of overwriting
the original picture file every time). In that case, maybe digikam and ON1
could interoperate.

I hope things will become more standard in the future. For what I see,
Digikam tries to be as compatible as possible including a number of tags
supported by other software (Adobe Lightroom, Microsoft photos, etc.), but
it also requires that other software embrace that compatibility.

The problem with all that is that these software suites won't exist forever.
At some point, they'll stop being maintained (like what already happened
with Picasa), and I want to be able to switch to another program to manage
my library without losing any data. Even if some software does everything
you need, you still need to think forward when that software does not work
anymore. I am currently using digikam because I believe it's the most
standard one (and also has lots of features) and I'll probably don't lose my
data when it stops being maintained.

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

Remco Viëtor

Re: Keyword (metadata) consistency between programs

OK, so I had a look at those ON1 files.
First, those are *not* XML files, but look like json files.

Second, while keywords are stored, I'm not sure the hierarchy is preserved
(hard to see, as there's either no tree, or the tree has only one branch).

So, if I understand correctly what you wrote in the accompanying text, the
_1030666 sidecars were written by ON1. While the .ON1 file has two sections,
the .XMP only has one section (no separate "raw" and "jpeg" sections).

So from what you describe, digikam does *not* read the .ON1 files, which is
understandable, as they are not XML data. I don't think digikam reads json
format...

It does read the ON1-generated XMP file, but that one is the same for both the
jpeg and the raw file, so keywords will be the same for both.

Remco

Gilles Caulier-4

Re: Keyword (metadata) consistency between programs

Le mer. 6 mai 2020 à 08:49, Remco Viëtor <[hidden email]> a écrit :

>
> OK, so I had a look at those ON1 files.
> First, those are *not* XML files, but look like json files.
>
> Second, while keywords are stored, I'm not sure the hierarchy is preserved
> (hard to see, as there's either no tree, or the tree has only one branch).
>
> So, if I understand correctly what you wrote in the accompanying text, the
> _1030666 sidecars were written by ON1. While the .ON1 file has two sections,
> the .XMP only has one section (no separate "raw" and "jpeg" sections).
>
> So from what you describe, digikam does *not* read the .ON1 files, which is
> understandable, as they are not XML data. I don't think digikam reads json
> format...

Yes, digiKam support standard. XMP sidecar are the standard, defined
by Metadata Working Group (including Adobe). XMP use XML format, not
JSON.

https://en.wikipedia.org/wiki/Extensible_Metadata_Platform

There is no plan to support exotic sidecar file, as ON1 do. ON1 must
use standards to improve interoperability, and not to re-invent the
wheel...

Best

Gilles Caulier

>
> It does read the ON1-generated XMP file, but that one is the same for both the
> jpeg and the raw file, so keywords will be the same for both.
>
> Remco
>
>
>
>
>

AlanH

Re: Keyword (metadata) consistency between programs

Thanks all for having a look at this. My OP was due to a concern about metadata (focusing on keywords) inconsistency between digikam and ON1. I didn't understand what was happening and was looking for a solution/explanation, thinking that both should be using industry standards for creating sidecars (ON1, as it appears, not). I thought that if in ON1, and while using one their editing tools I also wanted to modify/add/delete a keyword, description, rating, date ,etc, I could do that while in ON1. Likewise for digikam. That wasn't working as I expected, getting a confusing output of incorrect ratings, missing/changed keywords, keywords entered in a raw photo showing up differently/not all in its jpg, keyword hierarchies being parsed into component words, etc. And, in my experience, embedding/updating metadata to/from a jpg only served to transfer this mess from the sidecars to the jpg file.

I now have a much better sense of the situation. Also, this is not just a digikam/ON1 problem. I have found sidecar/metadata management incompatibilities in other popular processing apps - DXO PL, Luminar, Exposure, to name a few. Through all of this though, digikam has performed as the best DAM (speed, stability, interface, GPS/map functions, and yes, standards) for me, and I imagine for many others.

Now that I have a better understanding of what is happening I have changed my workflow to use only digikam for metadata management, including creating only the sidecars filename.jpg.xmp and filename.raw.xmp (filename.rw2.xmp in my case). I will not being using ON1 any more to update these sidecars nor to create original sidecars, nor use digikam to modify sidecars created by ON1. Not a difficult decision since ON1's DAM capabilities, although greatly advanced over the past few years, are still (IHMO) quite buggy - incorrect search results in keyword list, not finding keywords listed/visible for a photo, keywords deleted from keyword list (and photo, and sidecars) showing up in list again, etc.

Since any photos processed in ON1 were saved as JPGs and/or TIFFs, and I'm not one to ever re-edit them (or would rather start from scratch if I did), I have no problem deleting whatever sidecars ON1 created in the past (using CMD prompt DEL+argument) to keep conflicting .xmp and .on1 sidecars out of the directories. If I have a particular problem with a jpg photo's metadata being scrambled through previous joint access by digikam and ON1 I'm using exiftool to remove offending/all XMP and IPTC tags. Going forward ON1 will still create its own .on1 and .xmp sidecars for metadata and parametric editing info. I will simply not do anything as a user to modify these files.

Regards,

Alan

From: Digikam-users <[hidden email]> on behalf of Gilles Caulier <[hidden email]>
Sent: Wednesday, May 6, 2020 1:06 AM
To: digiKam - Home Manage your photographs as a professional with the power of open source <[hidden email]>
Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs

Le mer. 6 mai 2020 à 08:49, Remco Viëtor <[hidden email]> a écrit :
>
> OK, so I had a look at those ON1 files.
> First, those are *not* XML files, but look like json files.
>
> Second, while keywords are stored, I'm not sure the hierarchy is preserved
> (hard to see, as there's either no tree, or the tree has only one branch).
>
> So, if I understand correctly what you wrote in the accompanying text, the
> _1030666 sidecars were written by ON1. While the .ON1 file has two sections,
> the .XMP only has one section (no separate "raw" and "jpeg" sections).
>
> So from what you describe, digikam does *not* read the .ON1 files, which is
> understandable, as they are not XML data. I don't think digikam reads json
> format...

Yes, digiKam support standard. XMP sidecar are the standard, defined
by Metadata Working Group (including Adobe). XMP use XML format, not
JSON.

https://en.wikipedia.org/wiki/Extensible_Metadata_Platform

There is no plan to support exotic sidecar file, as ON1 do. ON1 must
use standards to improve interoperability, and not to re-invent the
wheel...

Best

Gilles Caulier

>
> It does read the ON1-generated XMP file, but that one is the same for both the
> jpeg and the raw file, so keywords will be the same for both.
>
> Remco
>
>
>
>
>

Alyssa3217

Re: [digiKam-users] Keyword (metadata) consistency between programs

In reply to this post by AlanH