[digiKam-users] Keyword (metadata) consistency between programs
Locked
 
|
I would very much like to use digiKam as a DAM for my photo collection. However, I'm finding problems with the interchange and display of metadata between processing software used in the past and digiKam (and going forward as I curate a growing collection).
Looking at just keywords, I thought my problems would be largely answered when I found the configuration option to "Use a compatible file name for writing sidecar files". In the case of On1 this produces the On1 sidecar format of a ".on1" sidecar (supposedly
for jpg photos, but see below) and ".xmp"sidecar (supposedly for raw photos, but not apparent in observations below).
If I am doing something incorrectly please let me know. If this is a program issue, is there anything that can be done to fix it (and when)? Create sidecars in On1 (Windows 2020.1) 1)Adding a keyword to a jpg photo in On1 creates an .on1 sidecar. Appears to have both a jpg and raw (rw2) tag block (is this correct term?). Keyword only added to jpg block. On1 display jpg photo keyword correctly. 2)Adding a keyword to a raw photo in On1 creates a .xmp sidecar. Has 2 keyword tag blocks: lr:hierarchicalSubject & dc:subject, both containing the keyword. Also adds keyword to raw block in .on1 sidecar. On1 displays raw photo keyword correctly. 3)digiKam, after "Reread Metadata From File" separately for both jpg and raw photos, displays raw photo keyword for both the jpg and raw (rw2) photos. Create sidecars in digiKam (Windows 6.4.0). (Config: write to XMP sidecar only; Use compatible filenames) 1)Adding a keyword to a jpg photo in dK creates an .on1 sidecar and .xmp simultaneously. The jpg photo keyword is added to both the jpg and raw (rw2) tag block in the .on1 sidecar as well as the .xmp sidecar. The .xmp sidecar has 6 keyword tag blocks: Acdsee, MicrosoftPhoto:LastKeywordXMP, digiKam:TagsList, lr:hierarchicalSubject, mediapro:CatalogSets & dc:subject, all containing the keyword. !!Adds a jpg photo keyword to a .xmp sidecar used (I think) by On1 for only raw photo metadata. On1 2020 displays the jpg photo for both the jpg and raw (rw2) photos. 2)Adding a keyword to a raw photo in dK (after .on1 & .xmp created above) -- the .on1 sidecar still has the original jpg keyword in both the jpg and raw blocks. In the .xmp sidecar the jpg photo keyword is replaced by the raw photo keyword in all 6 blocks. On1 2020 now displays the raw photo keyword for both the jpg and raw photo.
This behavior becomes more scrambled and confusing with repeated keyword additions when switching between one program and the other (digiKam and On1 in this case). In particular hierarchical keywords are parsed into separate component keywords as processed
(I think; at least as visibly evident) in the dc:subject block.
Thanks for your assistance with this inquiry.
Alan |
|
|
Ok, this is a bit complex, so I am not sure if I'll miss something.
Basically you want to be able to interoperate between ON1 photo editor, and Digikam, and only using sidecar files. As far as I know, .on1 sidecars are exclusive to the ON1 photo editor, so I don't think Digikam is able to read or produce them. Apparently, .on1 files not only include metadata (like keywords and dates) but also information about the editions done on the photo. So if you edit some keywords in digikam, store them in a .xmp, and then ON1 reads only from the .on1 file, it won't see any changes. I am not sure about the correct blocks that jpg and RAW photos should or should not. I know that digikam uses a bunch of redundant keyword tags in order to be compatible with lots of other picture managers, and usually that shouldn't be a problem, so I wouldn't be worried if you see a JPG and RW2 tag block (as long as there isn't contradictory information). Moreover, the information of the XMP sidecar can also be embedded in the JPG picture itself (but not the RAW, usually), so digikam and other softwares can read that information without the need of a xmp sidecar. If there is contradictory information between both, I am not sure to which one Digikam will give priority. Can you attach us an example of a picture and its sidecars (.xmp and .on1) so I can see and try for myself? -- Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html |
|
|
I’ll create a Dropbox link with 3 sets of files and some explanation and send the link via your email. Thanks Sent from Mail for Windows 10 From: [hidden email] Ok, this is a bit complex, so I am not sure if I'll miss something. |
|
|
I was thinking that given the configuration option “use a compatible file name for writing sidecar files”, and digiKam creating files using the extensions (.on1 & .xmp) that ON1 uses, that the sidecars would be interchangeable. The internal structure of the files even appears to be similar between what digiKam creates and ON1 creates (to an end-user, perhaps not a developer/programmer). Unfortunately it seems that contradictory metadata is written. As you may see in the Dropbox files jpeg metadata is overwritten by raw metadata in what are suppose to be jpeg blocks. Or, metadata is not updated without applying a very unintuitive process. Raw photo metadata can therefore be embedded in jpeg photos. Perhaps there could be more detailed explanation by digiKam what was meant by “Use a compatible file name….” since the way I thought it worked does not appear to be correct, or I’m not using it correctly. Or maybe ON1 has advanced the structure of their sidecar files beyond what digiKam has programmed. Any insight appreciated. Thanks. Sent from Mail for Windows 10 From: [hidden email] I’ll create a Dropbox link with 3 sets of files and some explanation and send the link via your email. Thanks Sent from
Mail for Windows 10 From: [hidden email] Ok, this is a bit complex, so I am not sure if I'll miss something. |
Re: Keyword (metadata) consistency between programs
|
|
On mardi 28 avril 2020 23:45:41 CEST Alan H wrote:
> I was thinking that given the configuration option “use a compatible file > name for writing sidecar files”, and digiKam creating files using the > extensions (.on1 & .xmp) that ON1 uses, that the sidecars would be > interchangeable. The internal structure of the files even appears to be > similar between what digiKam creates and ON1 creates (to an end-user, > perhaps not a developer/programmer). > Unfortunately it seems that contradictory metadata is written. As you may > see in the Dropbox files jpeg metadata is overwritten by raw metadata in > what are suppose to be jpeg blocks. Or, metadata is not updated without > applying a very unintuitive process. Raw photo metadata can therefore be > embedded in jpeg photos. > There's syntax (which XML fields are written) and semantics (what's the exact meaning of the contents). The syntax is declared in the namespace definitions, not the semantics. So if ON1 uses jpeg blocks and raw blocks in one file, digikam may not realise those can legitimately hold different info. > Perhaps there could be more detailed explanation by digiKam what was meant > by “Use a compatible file name….” since the way I thought it worked does > not appear to be correct, or I’m not using it correctly. Or maybe ON1 has > advanced the structure of their sidecar files beyond what digiKam has > programmed. "Compatible file name for sidecar files" means basically "use this extra extension for XMP files". The internal structure will be the same for all sidecars (it's derived from XML). Available tags may differ, and on reading digikam should ignore tags it doesn't know about. Remco P.S. Referring here to files you apparently want to keep private isn't particularly helpful |
|
|
Remco, and other viewers, apologies for not including the link to the files. Nothing that cannot be shared. Don’t know what I was thinking sending it just to Woenx. The more that can think about this the better. Here’s the link: https://www.dropbox.com/sh/9x7heiuo6h492xb/AAB8tvDi6Or0_XLqcIK-ggCpa?dl=0 Thanks for the explanation. I can probably come up with some sort of kludgy solution to work around this since I really want to use digiKam as my DAM. There are times though that the processing tools of other apps suit my needs and familiarity better, such as the brush and masking tools in ON1. I’m about to go ahead with curating 30,000+ digitized slides and it would be very helpful to be able to search on ratings, color codes, keywords and other metadata added or changed (eg date taken) in digiKam reliably when in other apps and I want to find and process a particular photo. As a curiosity, though, would it be a complex and/or a lengthy task for digiKam to create sidecars completely compatible with sidecars used by other apps? (possibly an industry first, aside from the snakepit of which apps to choose). I
only ask because to my untrained eye the ON1 sidecar (“filename.on1”) created by digiKam appears to be very similar in format to the sidecar that ON1 creates, when “Use a compatible file name…” is checked. It is quite distinct from the (normal looking?) .xmp
sidecar that digiKam (and most other apps) creates when that option is unchecked. From my view of the sidecars, the problem appears to be an inconsistency/confusion in whether the metadata tag belongs to a jpg or raw file and then writing it into the correct
block. The blocks seem to be there, just not properly interpreted and applied. Could digiKam be programmed (and with what effort) to realize that the blocks legitimately hold different info? Alan Sent from Mail for Windows 10 From: [hidden email] On mardi 28 avril 2020 23:45:41 CEST Alan H wrote: |
|
|
Hi Alan,
I wanted to try ON1 to see if I can replicate your experience. In their website they have several software packages to download and try: INTEGRATED WORKFLOW ON1 360° ON1 Photo RAW 2020.1 ON1 Photo Mobile 2020 INDIVIDUAL PRODUCTS ON1 HDR 2020 ON1 Resize 2020 ON1 Effects 2020 ON1 Video 2020 MEMBERSHIPS ON1 Plus Which one do you use? -- Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html |
|
|
Hi, Thanks for taking a look. I’m using ON1 Photo RAW 2020.1 (Windows). Sent from Mail for Windows 10 From: [hidden email] Hi Alan, |
|
|
Ok, so I installed both (On1 and Digikam) in a windows machine. I told
digikam to read and save all data to .xmp sidecars. I also told ON1 not to save metadata and edits to .on1 files. I first wanted to test the most basic assumption, that both can write and read from sidecar files, so they can exchange information. For that purpose, I created a directory which will be the picture library for both Softwares, and I put a simple JPG picture I got online (with no metadata). If I edit that picture in digikam and add a keyword, I can see it generates a .xmp file and the keyword is there, under a number of sections <digiKam:TagsList>, <MicrosoftPhoto:LastKeywordXMP>, <lr:hierarchicalSubject>, <mediapro:CatalogSets> and <dc:subject>. However, if I take that same picture (the original one without metadata) and I add a keyword, no .xmp sidecar is generated (and the metadata is not embedded in the picture either). Moreover, it seems that the next time you start ON1, it activates the saving to .on1 files again automatically. If I add a keyword from ON1, digikam does not seem to see it. And vice-versa. So, how do you tell ON1 to store the metadata in a .xmp sidecar file? -- Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html |
|
|
Just for a bit of clarity with respect to my experience… -the 4 th paragraph (If I edit that picture in digicam…): did digicam create the .xmp file in the format “filename.jpg.xmp”?
-the 5th paragraph (However, if I…): were you in ON1 when adding the keyword? Agreed on the second statement. ON1 appears to write JPG metadata (your last question) to its proprietary “filename.on1” sidecar file and creates these
files automatically for all JPGs the moment any file in a directory is opened. When I saw the configuration in digikiam to “Use a compatible file name for writing to sidecar files” and it appeared to create a sidecar in a format very similar to ON1’s, I thought I had at last found two apps for which the metadata sidecars
would be interchangeable. It seems though that there is still some confusion about how they process each others sidecars. It gets more confusing when .raw files come into the mix as I tried to show in previous posts. On1 writes a raw file’s metadata to a .raw sidecar but also seems to include (some?) raw metadata in the .on1 sidecar file, along with all of
the parametric commands for whatever non-destructive edits are done. Again, in my experience, if the .on1 sidecar was, in particular, created by digikam both apps have difficulty writing/reading/parsing the JPG and RAW blocks appropriately.
Regarding the last question, I have opened a ticket with ON1 somewhat on that subject – how to get ON1 to read digikam’s filename.jpg.xmp and filename.raw.xmp files. Tech support got back to me and said they had to pass it up to their developers
since they did not have an answer. I’ll try to steer the inquiry to writing to “proper” xmp sidecars as well. I’ll pass that info along when (hopefully) I have it. This isn’t just a digicam vs ON1 issue. I have found sidecar compatibility issues between other popular photo processors – Luminar, DXO PL, Exposure, Affinity – that are trying to provide DAMs (or even just non-destructive editing sidecars).
It is such a contentious issue in DXO that the user forum regularly advocates staying away from the DAM game and stick to what they do best.
Sent from Mail for Windows 10 From: [hidden email] Ok, so I installed both (On1 and Digikam) in a windows machine. I told |
|
|
- If you check the "sidecar file name compatible with commercial software"
option, the sidecar filename is "filename.xmp". Otherwise is "filename.jpg.xmp". - Yes, I was referring to ON1. I couldn't make it to create a XMP sidecar, so Digikam can read it. Maybe there's an option, but I didn't find it quickly. Regarding RAW files, I guess it's normal, since they are not meant to be edited, so all changes will be saved in sidecar files (I believe this is how digikam works for RAW files). I still have not tried if ON1 can generate XMP files for sidecars, and, if that's the case, they can be read by digikam. There's also the question if ON1 can also read keywords saved directly inside the picture file (that's how I do it, under the risk of overwriting the original picture file every time). In that case, maybe digikam and ON1 could interoperate. I hope things will become more standard in the future. For what I see, Digikam tries to be as compatible as possible including a number of tags supported by other software (Adobe Lightroom, Microsoft photos, etc.), but it also requires that other software embrace that compatibility. The problem with all that is that these software suites won't exist forever. At some point, they'll stop being maintained (like what already happened with Picasa), and I want to be able to switch to another program to manage my library without losing any data. Even if some software does everything you need, you still need to think forward when that software does not work anymore. I am currently using digikam because I believe it's the most standard one (and also has lots of features) and I'll probably don't lose my data when it stops being maintained. -- Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html |
Re: Keyword (metadata) consistency between programs
|
|
OK, so I had a look at those ON1 files.
First, those are *not* XML files, but look like json files. Second, while keywords are stored, I'm not sure the hierarchy is preserved (hard to see, as there's either no tree, or the tree has only one branch). So, if I understand correctly what you wrote in the accompanying text, the _1030666 sidecars were written by ON1. While the .ON1 file has two sections, the .XMP only has one section (no separate "raw" and "jpeg" sections). So from what you describe, digikam does *not* read the .ON1 files, which is understandable, as they are not XML data. I don't think digikam reads json format... It does read the ON1-generated XMP file, but that one is the same for both the jpeg and the raw file, so keywords will be the same for both. Remco |
Re: Keyword (metadata) consistency between programs
|
|
Le mer. 6 mai 2020 à 08:49, Remco Viëtor <[hidden email]> a écrit :
> > OK, so I had a look at those ON1 files. > First, those are *not* XML files, but look like json files. > > Second, while keywords are stored, I'm not sure the hierarchy is preserved > (hard to see, as there's either no tree, or the tree has only one branch). > > So, if I understand correctly what you wrote in the accompanying text, the > _1030666 sidecars were written by ON1. While the .ON1 file has two sections, > the .XMP only has one section (no separate "raw" and "jpeg" sections). > > So from what you describe, digikam does *not* read the .ON1 files, which is > understandable, as they are not XML data. I don't think digikam reads json > format... Yes, digiKam support standard. XMP sidecar are the standard, defined by Metadata Working Group (including Adobe). XMP use XML format, not JSON. https://en.wikipedia.org/wiki/Extensible_Metadata_Platform There is no plan to support exotic sidecar file, as ON1 do. ON1 must use standards to improve interoperability, and not to re-invent the wheel... Best Gilles Caulier > > It does read the ON1-generated XMP file, but that one is the same for both the > jpeg and the raw file, so keywords will be the same for both. > > Remco > > > > > |
|
|
Thanks all for having a look at this. My OP was due to a concern about metadata (focusing on keywords) inconsistency between digikam and ON1. I didn't understand what was happening and was looking for a solution/explanation, thinking that both should be using
industry standards for creating sidecars (ON1, as it appears, not). I thought that if in ON1, and while using one their editing tools I also wanted to modify/add/delete a keyword, description, rating, date ,etc, I could do that while in ON1. Likewise for digikam.
That wasn't working as I expected, getting a confusing output of incorrect ratings, missing/changed keywords, keywords entered in a raw photo showing up differently/not all in its jpg, keyword hierarchies being parsed into component words, etc. And, in my
experience, embedding/updating metadata to/from a jpg only served to transfer this mess from the sidecars to the jpg file.
I now have a much better sense of the situation. Also, this is not just a digikam/ON1 problem. I have found sidecar/metadata management incompatibilities in other popular processing apps - DXO PL, Luminar, Exposure, to name a few. Through all of this though,
digikam has performed as the best DAM (speed, stability, interface, GPS/map functions, and yes, standards) for me, and I imagine for many others.
Now that I have a better understanding of what is happening I have changed my workflow to use only digikam for metadata management, including creating only the sidecars filename.jpg.xmp and filename.raw.xmp (filename.rw2.xmp in my case). I will not being using
ON1 any more to update these sidecars nor to create original sidecars, nor use digikam to modify sidecars created by ON1. Not a difficult decision since ON1's DAM capabilities, although greatly advanced over the past few years, are still (IHMO) quite buggy
- incorrect search results in keyword list, not finding keywords listed/visible for a photo, keywords deleted from keyword list (and photo, and sidecars) showing up in list again, etc.
Since any photos processed in ON1 were saved as JPGs and/or TIFFs, and I'm not one to ever re-edit them (or would rather start from scratch if I did), I have no problem deleting whatever sidecars ON1 created in the past (using CMD prompt DEL+argument) to keep
conflicting .xmp and .on1 sidecars out of the directories. If I have a particular problem with a jpg photo's metadata being scrambled through previous joint access by digikam and ON1 I'm using exiftool to remove offending/all XMP and IPTC tags. Going forward
ON1 will still create its own .on1 and .xmp sidecars for metadata and parametric editing info. I will simply not do anything as a user to modify these files.
Regards,
Alan
From: Digikam-users <[hidden email]> on behalf of Gilles Caulier <[hidden email]>
Sent: Wednesday, May 6, 2020 1:06 AM To: digiKam - Home Manage your photographs as a professional with the power of open source <[hidden email]> Subject: Re: [digiKam-users] Keyword (metadata) consistency between programs Le mer. 6 mai 2020 à 08:49, Remco Viëtor <[hidden email]> a écrit :
> > OK, so I had a look at those ON1 files. > First, those are *not* XML files, but look like json files. > > Second, while keywords are stored, I'm not sure the hierarchy is preserved > (hard to see, as there's either no tree, or the tree has only one branch). > > So, if I understand correctly what you wrote in the accompanying text, the > _1030666 sidecars were written by ON1. While the .ON1 file has two sections, > the .XMP only has one section (no separate "raw" and "jpeg" sections). > > So from what you describe, digikam does *not* read the .ON1 files, which is > understandable, as they are not XML data. I don't think digikam reads json > format... Yes, digiKam support standard. XMP sidecar are the standard, defined by Metadata Working Group (including Adobe). XMP use XML format, not JSON. https://en.wikipedia.org/wiki/Extensible_Metadata_Platform There is no plan to support exotic sidecar file, as ON1 do. ON1 must use standards to improve interoperability, and not to re-invent the wheel... Best Gilles Caulier > > It does read the ON1-generated XMP file, but that one is the same for both the > jpeg and the raw file, so keywords will be the same for both. > > Remco > > > > > |
| Free forum by Nabble | Edit this page |