[digikam] [Bug 337688] New: Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
     Ever confirmed|0                           |1
         Resolution|FIXED                       |---

--- Comment #20 from Veaceslav Munteanu <[hidden email]> ---
Still fixing, digiKam still do not overwrite the old format...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #21 from Christian <[hidden email]> ---
(In reply to Veaceslav Munteanu from comment #17)
> that host is so slow, it takes me 8 hours to download it. Use google drive
> or dropbox for faster speeds.

thank you : )

Sorry, a download within 2 minutes is available again after rebooting my
server. There is a degradation of performance after several months uptime of my
apache 2.2. I found no time to find out what causes this - no hints in my logs
so far.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #22 from Veaceslav Munteanu <[hidden email]> ---
https://www.dropbox.com/s/axtmrkmu27nkyxi/tags_clean.png

This is tag tree after importing your mega-pack :D

I guess it's pretty clean. Also I have no idea how could you make two
_Digikam_root_tag_ on the same level, it is almost impossible, there are
duplicate checks everywhere...

Also, now digiKam is able to clean-up your metadata when you write it back, so
your images have brand new, clean metadata in them.

Still need to check few of your test cases, but... after lunch :)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #23 from Christian <[hidden email]> ---
(In reply to Veaceslav Munteanu from comment #22)
> https://www.dropbox.com/s/axtmrkmu27nkyxi/tags_clean.png
>
> This is tag tree after importing your mega-pack :D
>
> I guess it's pretty clean. Also I have no idea how could you make two
> _Digikam_root_tag_ on the same level, it is almost impossible, there are
> duplicate checks everywhere...

Wow - that would be great. So I reread metadata from all images and write
metadata to all images using tag manager to clean up?

Question:
Will your fixes apply
-to tag manager tools only, or only
-to explicit calls of "read metadata" / "write metadata" for files and albums,
-or to both ?
How about the inital reading of a collection? Same code?

> _Digikam_root_tag_ on the same level

This happended only once with the root tag, when I read metadata from three
files at once, two with inconsistent IPTC and XMP into a messed up tree. Maybe
a display error.

Duplication of other tags on the same level is very common - see eg my latest
attachment. I copied 5 files of the testcase to my big collection and used tag
manager to read metadata of all images to find out if the same bugs apply. They
do.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #24 from Christian <[hidden email]> ---
Created attachment 87932
  --> https://bugs.kde.org/attachment.cgi?id=87932&action=edit
Duplication of tags on same level - a few files can mess up whole tag tree

Tag duplication on the same level in big collection with 5 inconsistent files:

I copied 5 files of the testcase to my big collection and used tag manager to
read metadata of all images to find out if the same bugs apply. They do.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|REOPENED                    |RESOLVED
      Latest Commit|http://commits.kde.org/digi |http://commits.kde.org/digi
                   |kam/86d06f51a3d391fd243ad82 |kam/992a219433264daa00c77f3
                   |983e532e12171b6b5           |c6ad27a92705d6900

--- Comment #25 from Veaceslav Munteanu <[hidden email]> ---
Git commit 992a219433264daa00c77f3c6ad27a92705d6900 by Veaceslav Munteanu.
Committed on 24/07/2014 at 13:08.
Pushed by munteanu into branch 'master'.

M  +65   -74   digikam/fileaction/metadatahub.cpp
M  +9    -0    digikam/fileaction/metadatahub.h

http://commits.kde.org/digikam/992a219433264daa00c77f3c6ad27a92705d6900

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |---

--- Comment #26 from Veaceslav Munteanu <[hidden email]> ---
The original problems are from _Digikam_root_tag_, which I added extra checks.

every occurrence of it will be deleted, so it doesn't matter if it is one
_Digikam_root_tag_ or 5 of them.

The nested root tags I discovered in your metadata, some images contained:

_Digikam_root_tag_/_Digikam_root_tag_, probably old bugs...

The check I applied to both parts where digiKam read tags and where digiKam
write tags, clean-up can be done using any metadata writing option:

1. Write metadata
2. Maintenance tool
3. Tags Manager sync export options

Please note: if your tags database is empty and you trigger write, all metadata
from images will be cleared. Do not forget to read them before writing.

Also, about duplicate tags on the same level, digiKam do not allow me to make
them, even re-read do not reveal anything suspicious.

Only a corrupt database can contain this, use new one(when testing my fixes).

Also I tested the tag move to different sub-tree and it works, all tags are
correctly written and read.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #27 from Christian <[hidden email]> ---
(In reply to Veaceslav Munteanu from comment #26)

Wow, this was really quick : )
Thank you for the quick response and the time invested.

Yes, I expect that all tags are removed if nothing was read before.
Remark: I used a completely empty mysql database for the test. The three root
tags showed up on the second level - the position were the gui often got out of
sync. They might have been fakes disappearing after closing and opening. I do
not remember.

How do I get the fix?

I am from the Java side, so I don't know exactly how I can build something from
the updated master branch that runs in my distro (OpenSuse) with mysql.

Do you recommend the procedere on https://www.digikam.org/download/GIT ?
Or should I contact guys from the suse factory? Easier ways like a fix?
I guess I have to install the dependency packages for kipi and digikam?

Christian

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|REOPENED                    |RESOLVED
      Latest Commit|http://commits.kde.org/digi |http://commits.kde.org/digi
                   |kam/992a219433264daa00c77f3 |kam/5cc7125ba8ec452b9d4f956
                   |c6ad27a92705d6900           |87f35e7071bbd9b55

--- Comment #28 from Veaceslav Munteanu <[hidden email]> ---
Git commit 5cc7125ba8ec452b9d4f95687f35e7071bbd9b55 by Veaceslav Munteanu.
Committed on 24/07/2014 at 14:34.
Pushed by munteanu into branch 'master'.

M  +25   -16   digikam/fileaction/metadatahub.cpp
M  +7    -0    digikam/fileaction/metadatahub.h
M  +3    -2    libs/database/imagescanner.cpp

http://commits.kde.org/digikam/5cc7125ba8ec452b9d4f95687f35e7071bbd9b55

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #29 from Veaceslav Munteanu <[hidden email]> ---
I have no experience with packaging and the fix is only available in git
repository. You can try and build digiKam from git sources or to wait until
digiKam 4.2 will be available ( Release date: 2014-08-31)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Gilles Caulier-4
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Version Fixed In|                            |4.2.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Christian <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |---

--- Comment #30 from Christian <[hidden email]> ---
Reason why I want to reopen this bug: The last fix improved tagging a lot, but
there is still an issue related to "zombie tags" that cannot be deleted in
digikam 4.2.

----------------------------------------------------------------------------

This testcase demonstrates a bug (or several bugs) related to "zombie tags",
that cannot be removed with any of the tools in digikam 4.2 (linux: openSuse).
Even it looks like they are gone, they will come back after reading metadata.
Some only show up again if you excplicitely "read metadata" from the files.

Download Link:
-----------------------------------
http://buitk.at/download/digikam42_zombie_tag_testcase.zip

Pre condition:
-----------------------------------
I cleaned all tags in the files and in the database with dk R4.2 for openSuse
first. Tags were written two times to all files.

Result: About 30.000 tagged images seem to be clean now, apr. 14.000 are still
infected with old stuff that was not deleted as excpected.
When reading metadata a second time a lot of old tags came to live again. I
removed them again, but finally some show up during writing again.
Only these cases are documented here.

I install Digikam 4.2 with an empty sqlite database to do the tests described
below.

Testcase and reproducable symptoms:
-----------------------------------

I install Digikam 4.2 with an empty sqlite database and import tags from the
file "00_all_keywords_buitk.jpg".
To do so you have to activate "read metadata from sidecar files". I use "write
tags to files" settings for metadata.

Then I add all images of the folders below to the collection:
  "cannot_remove_tags_from_different_subtree_with_same_leaves"
  "cannot_remove_tags_from_wrong_position"

Then I try to remove the tags that are mentioned in the name of the jpg images.
Example: File
"dsc01931_remove_Zeit-bChr004_Neuzeit_and_1400Spaetmittelalter.jpg"
 ... refers to <toplevel>Zeit/bChr004_Neuzeit/...   and all subsequent tags
 ... refers to <toplevel>1400Spaetmittelalter

Symptom: Deleted "Zombie Tags" come back
------------------------------------------

Digikam 4.2 can read and write all tagges files now. So the changes are always
written to the images without any error messages.
But in many cases (14.000) the removed tags still remain somewhere in the
written metadata - and not in the database. I have checked this by starting
with an empty database again.
The zombie-tags will show up again if you manually select a single file to read
metadata. Sometimes these zombie tags are created with several copies, if you
select more than one file at a time to read metadata.

Analysis: some hints to track down "why?"
-----------------------------------------
When looking at the cases that still cause troubles in dk4.2 it becomes obvious
that these zombie tags are related to tag-names that are used on more than one
position in the tag tree.
There are several reasons why tag-names are used twice many times in practice:

1. They come from moving tags to another position e.g. if a wrong geographic
classification is corrected, e.g. location is moved to another parent node.
Note: I will never drag the tag to the new position. A new is created, the old
is deleted.

2. Historical reasons, eg. there was an old subbranch of the time-categories:
Zeit/bChr004_Neuzeit/* with the same leaves like the new one
Zeit/bChr006_Neuzeit/*
Since 2009 I try to get rid of the old Zeit/bChr004_Neuzeit/ branch - with no
luck till today.

3. Many orphan tag nodes come from the "Digikam root tag" bug in older
versions, that caused the loss of one or more levels of nesting.

My guess is that there is a bug in writing, when a tag with the same tag-name
should be removed from one position, but still has to remain in another
position.
NOTE: If you remove all tags manually, all tags will be gone, also the zombies.
These zombies only persist if there is at least another tag around !

Final Note: I also deleted a lot of files with bad tags to get rid of the mess.
Important ones were cleaned using Gimp. But I cannot clean up 14.000 images
this way.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Christian <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|4.1.0                       |4.2.0
           Severity|grave                       |major

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Christian <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Version Fixed In|4.2.0                       |

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Christian <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #87875|0                           |1
        is obsolete|                            |
  Attachment #87882|0                           |1
        is obsolete|                            |
  Attachment #87885|0                           |1
        is obsolete|                            |
  Attachment #87925|0                           |1
        is obsolete|                            |
  Attachment #87932|0                           |1
        is obsolete|                            |

--- Comment #31 from Christian <[hidden email]> ---
Created attachment 88295
  --> https://bugs.kde.org/attachment.cgi?id=88295&action=edit
Testcase for digikam 4.2 tags that cannot be removed

http://buitk.at/download/digikam42_zombie_tag_testcase.zip

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Christian-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #32 from Christian <[hidden email]> ---
Created attachment 88296
  --> https://bugs.kde.org/attachment.cgi?id=88296&action=edit
because of undeletable tags, duplication still occurs

The bug described in the testcase might also cause duplication of tags when
selecting more than one file to reread metadata or when reading/writing all
images. It does not occur, if single files are selected to read metadata one
after another.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #33 from Veaceslav Munteanu <[hidden email]> ---
:( Yes, I can reproduce some problems, such as can't remove the same tag from
different sub-trees... I'm a little busy now to finish the work on other part
of digiKam, might take some time...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Gilles Caulier-4
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

--- Comment #34 from Gilles Caulier <[hidden email]> ---
Veaceslav,

Any progress here before 4.3.0 release ?

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Latest Commit|http://commits.kde.org/digi |http://commits.kde.org/digi
                   |kam/5cc7125ba8ec452b9d4f956 |kam/2719c59891e2a72a15e47f5
                   |87f35e7071bbd9b55           |39c26b2721e049a24
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #35 from Veaceslav Munteanu <[hidden email]> ---
Git commit 2719c59891e2a72a15e47f539c26b2721e049a24 by Veaceslav Munteanu.
Committed on 01/09/2014 at 17:52.
Pushed by munteanu into branch 'development/balooport'.

M  +0    -1    digikam/fileaction/metadatahub.cpp
M  +5    -1    utilities/baloo/baloowrap.cpp

http://commits.kde.org/digikam/2719c59891e2a72a15e47f539c26b2721e049a24

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 337688] Reading/writing of keyword-tags to jpg and xmp corrupts tag hierarchy, duplicate root tag

Veaceslav Munteanu-2
In reply to this post by Christian-2
https://bugs.kde.org/show_bug.cgi?id=337688

Veaceslav Munteanu <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |---

--- Comment #36 from Veaceslav Munteanu <[hidden email]> ---
Still working... :D

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
123