What is being/will be done about the tags management problem?

[tosca]

The tags synchronization issue has been poisoning the life (OK, only the photography workflow ;-) ) of a number of Digikam users for several months/years and frequently re-emerge as a subject in this mailing-list.
As for myself (and I think for some others, too) I've gave up long ago to try to manage my tags, and I am patiently (more or less) waiting for the situation getting better before doing a huge organization of my whole photobase.

Could the developers team tell us how soon the users can expect a solution to this blocking problem.

Thanks,
Marie-Noëlle


--


Mes dernières photos sont dans ma galerie.
Connaissez-vous Image Fixe, le photo-club de Saint Jean du Gard ?
Et parcourez les Cévennes à ma façon avec Cévennes Plurielles,

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Elle Stone]

Seconding the request! Just knowing there is a light at the end of the
tunnel would be nice.

--
http://ninedegreesbelow.com - articles on open source digital photography
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Gilles Caulier-4]

The light is here. I'm aware about this problem. As we entry in a new students period, i will try to find people to investigate and fix it.

Gilles

2013/2/19 Elle Stone <[hidden email]>

Seconding the request! Just knowing there is a light at the end of the
tunnel would be nice.

--
http://ninedegreesbelow.com - articles on open source digital photography
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Elle Stone]

Gilles, thanks!

--
http://ninedegreesbelow.com - articles on open source digital photography
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[M. Fioretti]

On Tue, February 19, 2013 11:16 am, Marie-Noëlle Augendre wrote:
> The tags synchronization issue has been poisoning the life

Sorry, I am not sure I understand what kind of synchronization you are
referring right now. SYnchronization between in-picture metadata and
database, or something else? May Iask you to point to the threads where
the specific problem you refer to has been already discussed?

TIA,
Marco
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Tue, 19 Feb 2013, Marie-Noëlle Augendre wrote:

> The tags synchronization issue has been poisoning the life (OK,
> only the photography workflow ; -) ) of a number of Digikam users
> for several months/years and frequently re-emerge as a subject in
> this mailing-list. As for myself (and I think for some others,
> too) I've gave up long ago to try to manage my tags, and I am
> patiently (more or less) waiting for the situation getting better
> before doing a huge organization of my whole photobase.
>
> Could the developers team tell us how soon the users can expect a
> solution to this blocking problem.

On Tue, 19 Feb 2013, Gilles Caulier wrote:

> The light is here. I'm aware about this problem. As we entry in a
> new students period, i will try to find people to investigate and
> fix it.

Hello,

a few comments about this problem which is - as Marie-Noëlle said -
reccurent on this list.

From my humble opinion, it's not just a fix to software, it's a
software design strategy. I've been concerned too, for a long time,
with this issue and I had a look at the source code.
The code seems corrects, some design choices may appear more
questionnable.

1. Synchronization of tags between database and images :
When Digikam writes metadata to images (or sidecar files),
the written tagslist is the current database state,
so « export » synchronization is done.

When Digikam re-reads metadata, the tagslist is merged with the
current tags associated to the image. (This is in source file
metadatahub.cpp, in method MetadataHub::loadTags, it's an explicit
merge, new tags are added to the current tags from DB, not a delete
current and replace by new).
And there, « import » synchronization is not effective, the database
doesn't reflect what is into the images XMP sections.

I don't see very well in which use cases this merging is useful,
but why not, if it has been coded that way, it probably is.
But what should be choosen, replace or merge ?

At least, a simple fix could be an option in the metadata folder,
« upon read concat new tags vs. replace all current by new ones ».
And in the code, just a "tagList = loadedTagPaths" instead of merge.
This would guarantee the database reflects correctly what is found
from images and enforce reversible behaviour between « Write metadata »
and « Reread Metadata » .

2. Metadata interchange with other applications : This is another
tags related problems, due to the different tags storing formats.
When Digikam writes metadata, tagslist are saved in several possible
formats, the Digikam format, digiKam:TagsList, the Adobe LightRoom
format, lr:hierarchicalSubject, the Microsoft Photo format,
MicrosoftPhoto.LastKeywordXMP. So, the great thing is that other
applications will get the tags.

When Digikam re-reads metadata, reading is done with a priority
order. Look for Digikam tags, if found keep that. If not found,
look for Microsoft tags. If found keep that. If not found look for
LightRooom tags, etc.
(From source module dmetadata.cpp, method DMetadata::getImageTagsPath)

But this means that if someone tags an image with Digikam, then uses
later another application, e.g. LightRoom, and edit the tags,
when back to Digikam the older tags in Digikam format will be
reloaded, not the newer LightRoom tags.

I just signal that, but it's very difficult to fix at software
level. The only solution I can imagine would be to add into the
digiKam namespace a kind of modification date. Upon read, comparing
that date with the standard xmp:MetadataDate would allow to detect
that another application has been used and that it could be better
to load LightRoom tags first. But it's probably too rare a use case
to be worth the work.
Software can't be magic and letting users know what they do, and
asuming what they do, is probably a simpler strategy.


In conclusion : I really think that tags problem should probably
require more accurate design definitions. What is expected, what do
people do with tags, what should be the expected behaviour, do a
majority of users work with different software and how should be the
interoperability strategy, etc. But clearly, it's not a bug, not at
all. It's design choice, and on the software planet, design changes
should come from requirements changes. IMHO.

Regards,
Jean-François

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Tue, 19 Feb 2013, M. Fioretti wrote:

> On Tue, February 19, 2013 11:16 am, Marie-Noëlle Augendre wrote:
>> The tags synchronization issue has been poisoning the life
>
> Sorry, I am not sure I understand what kind of synchronization you are
> referring right now. SYnchronization between in-picture metadata and
> database, or something else? May Iask you to point to the threads where
> the specific problem you refer to has been already discussed?

Hello Marco,

Synchronization, as for this tags issue, means comparing metadata in
the image (XMP section of the image file) to metadata associated to
that image in the Digikam database.

If you have, say three tags for a Digikam image, Writing metadata to
image will copy exactly these tags to the XMP section of the file.

If now, « from outside » (i.e. using some other software than DK),
you remove these three tags and put two differents ones, your image
file reflect the two new tags only.
And when, in Digikam, you Re-read metadata from image, the image
will have five associated tags ! The two read from the file, plus the
three previous obsolete ones. And thus your database state is « out of
synchronization » wrt. to image tags.


Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Le 19/02/2013 16:00, Jean-François Rabasse a écrit :

> a few comments about this problem which is - as Marie-Noëlle said -
> reccurent on this list.

very good work, thanks

> I don't see very well in which use cases this merging is useful,

anyway, there are two ways of solving this issue, one easy, the other
difficult :-)

* easy mode: when allowing write metadata to images, ask for an option:
image metadata replace database metadata ("priority to the image
metadata") or database metadata replace image metadata ("priority to the
database metadata"). The rationale is that an image can be changed from
the operating system (think at the reloading of backup).

* difficult mode: merge the metadata in the two ways, but then I not
even imagine how one can sort the thing

anyway at the end the metadata should be the same everywhere!!

we could also have two more options:

* read meta data from image, write to database
* read meta data from database, write to image

jdd


--
http://dodin.org
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[tosca]

I'm afraid I've no better way than you to search the archives of a mailing-list... searching a forum would be easier.
I 'know' because I've been here long enough. But you can start by having a look at this bug report: https://bugs.kde.org/show_bug.cgi?id=268688

Marie-Noëlle

2013/2/19 M. Fioretti <[hidden email]>

 May Iask you to point to the threads where
the specific problem you refer to has been already discussed?

TIA,
Marco

--


Mes dernières photos sont dans ma galerie.
Connaissez-vous Image Fixe, le photo-club de Saint Jean du Gard ?
Et parcourez les Cévennes à ma façon avec Cévennes Plurielles,

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[tosca]

2013/2/19 jdd <[hidden email]>

anyway at the end the metadata should be the same everywhere!!

+1
 

we could also have two more options:

* read meta data from image, write to database
* read meta data from database, write to image

jdd

This could be, at least, a kind of rebuild procedure in case anything goes wrong.

As for now, in 2.9.0 version (Fedora 17 have no 3.0 yet, because of a KDE dependency, I understood), Digikam still crashes regularly (or sometimes I have to kill it) during the update file process; so this is still one more reason for metadata to become de-synchronized, even if things were OK at the beginning of a working session.


Marie-Noëlle

--


Mes dernières photos sont dans ma galerie.
Connaissez-vous Image Fixe, le photo-club de Saint Jean du Gard ?
Et parcourez les Cévennes à ma façon avec Cévennes Plurielles,

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Gilles Caulier-4]

Jean François,

Please report this investigation in this bugzilla file :

https://bugs.kde.org/show_bug.cgi?id=268688

Gilles

2013/2/19 Jean-François Rabasse <[hidden email]>

On Tue, 19 Feb 2013, Marie-Noëlle Augendre wrote:

The tags synchronization issue has been poisoning the life (OK,
only the photography workflow ; -) ) of a number of Digikam users
for several months/years and frequently re-emerge as a subject in
this mailing-list. As for myself (and I think for some others,
too) I've gave up long ago to try to manage my tags, and I am
patiently (more or less) waiting for the situation getting better
before doing a huge organization of my whole photobase.

Could the developers team tell us how soon the users can expect a
solution to this blocking problem.

On Tue, 19 Feb 2013, Gilles Caulier wrote:

The light is here. I'm aware about this problem. As we entry in a
new students period, i will try to find people to investigate and
fix it.

Hello,

a few comments about this problem which is - as Marie-Noëlle said -
reccurent on this list.

From my humble opinion, it's not just a fix to software, it's a
software design strategy. I've been concerned too, for a long time,
with this issue and I had a look at the source code.
The code seems corrects, some design choices may appear more
questionnable.

1. Synchronization of tags between database and images :
When Digikam writes metadata to images (or sidecar files),
the written tagslist is the current database state,
so « export » synchronization is done.

When Digikam re-reads metadata, the tagslist is merged with the
current tags associated to the image. (This is in source file
metadatahub.cpp, in method MetadataHub::loadTags, it's an explicit
merge, new tags are added to the current tags from DB, not a delete
current and replace by new).
And there, « import » synchronization is not effective, the database
doesn't reflect what is into the images XMP sections.

I don't see very well in which use cases this merging is useful,
but why not, if it has been coded that way, it probably is.
But what should be choosen, replace or merge ?

At least, a simple fix could be an option in the metadata folder,
« upon read concat new tags vs. replace all current by new ones ».
And in the code, just a "tagList = loadedTagPaths" instead of merge.
This would guarantee the database reflects correctly what is found
from images and enforce reversible behaviour between « Write metadata »
and « Reread Metadata » .

2. Metadata interchange with other applications : This is another
tags related problems, due to the different tags storing formats.
When Digikam writes metadata, tagslist are saved in several possible
formats, the Digikam format, digiKam:TagsList, the Adobe LightRoom
format, lr:hierarchicalSubject, the Microsoft Photo format,
MicrosoftPhoto.LastKeywordXMP. So, the great thing is that other
applications will get the tags.

When Digikam re-reads metadata, reading is done with a priority
order. Look for Digikam tags, if found keep that. If not found,
look for Microsoft tags. If found keep that. If not found look for
LightRooom tags, etc.
(From source module dmetadata.cpp, method DMetadata::getImageTagsPath)

But this means that if someone tags an image with Digikam, then uses
later another application, e.g. LightRoom, and edit the tags,
when back to Digikam the older tags in Digikam format will be
reloaded, not the newer LightRoom tags.

I just signal that, but it's very difficult to fix at software
level. The only solution I can imagine would be to add into the
digiKam namespace a kind of modification date. Upon read, comparing
that date with the standard xmp:MetadataDate would allow to detect
that another application has been used and that it could be better
to load LightRoom tags first. But it's probably too rare a use case
to be worth the work. Software can't be magic and letting users know what they do, and
asuming what they do, is probably a simpler strategy.


In conclusion : I really think that tags problem should probably
require more accurate design definitions. What is expected, what do
people do with tags, what should be the expected behaviour, do a
majority of users work with different software and how should be the
interoperability strategy, etc. But clearly, it's not a bug, not at
all. It's design choice, and on the software planet, design changes
should come from requirements changes. IMHO.

Regards,
Jean-François

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Tue, 19 Feb 2013, jdd wrote:

> anyway, there are two ways of solving this issue, one easy, the other
> difficult :-)
>
> * easy mode: when allowing write metadata to images, ask for an option: image
> metadata replace database metadata ("priority to the image metadata") or
> database metadata replace image metadata ("priority to the database
> metadata"). The rationale is that an image can be changed from the operating
> system (think at the reloading of backup).
>
> * difficult mode: merge the metadata in the two ways, but then I not even
> imagine how one can sort the thing

No way !
Because of lack of information. Synchronization issues require that a
« reference » could be set, and a « outdated » be set too.
And a synchronization software needs an information to decide. In most
case a timestamp is used, as do files synchronization tools such as
rsync. The oldest stuff is considered as the outdated wrt. the newest
and will be replace by the newest.

But with XMP metadata, there's no such information. The XMP schema has
a dedicated field, xmp:MetadataDate, that should be the last time
metadata has been updated. Ideally, this field should be set
automaticaly when writing metadata, but very few software do that.
At least, the libexiv2 doesn't, too bad :-(

So, only the user can decide which is the reference and which is the
outdated, and in which way to do the sync copy.

Anyway, with Digikam there's the « heavy » solution, when one wants the
database to be in sync with images : just delete all tags, in DK, all
tags trees, then reload metadata from images. Tags are rebuilt on
the fly and reflect exactly the images data.
That's my paleolithic technique, rought but efficient. :-)

Jean-François

NB: this is not just a tags issue, same problems occurs with other
metadata, e.g. title and caption. If you remove a title from an image,
outside Digikam, then re-read metadata, the DK database still contains
the old title, that must be removed by hand.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Elle Stone]

There is also the problem of keeping the database synchronized with
the sidecar metadata, if someone chooses to only write metadata to a
sidecar rather than risk writing to the image metadata:

https://bugs.kde.org/show_bug.cgi?id=309058




--
http://ninedegreesbelow.com - articles on open source digital photography
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Brian Morrison]

On Tue, 19 Feb 2013 12:25:08 -0500
Elle Stone wrote:

> There is also the problem of keeping the database synchronized with
> the sidecar metadata, if someone chooses to only write metadata to a
> sidecar rather than risk writing to the image metadata:
>
> https://bugs.kde.org/show_bug.cgi?id=309058

I don't mean to sink any ships here, but isn't this problem related to
which programs support which metadata?

There are various types, metadata which is widely known about and
supported for editing in multiple programs and there is private metadata
which is only understood in one program (or maybe two). Unless there is
a spec somewhere that defines the 'shared' metadata so that developers
can implement support for it in their software then this circle can't be
squared.

Isn't this exactly why sidecar files came about? A way of keeping data
associated with a file in such a way that programs that don't know
about it don't mess with it?

--

Brian Morrison
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Gilles Caulier-4]

Please,

If someone can synthesis all reference to bugzilla entries and all step to reproduce all problem, this will help me to delegate job to a student. Why not to use KDE wiki and create a resume page of this "Tag Metadata Management" project :

http://community.kde.org/Digikam/GSoC2013#Small_Projects_to_Introduce_Students

This can be a good introduction project for new students who want to contribute this summer on GoSC 2013...

Note; i'm currently busy with new libkface implementation done by an Indian student about Face Recognition, with algorithm rewritten from scratch...

Gilles Caulier

2013/2/19 Elle Stone <[hidden email]>

There is also the problem of keeping the database synchronized with
the sidecar metadata, if someone chooses to only write metadata to a
sidecar rather than risk writing to the image metadata:

https://bugs.kde.org/show_bug.cgi?id=309058

--
http://ninedegreesbelow.com - articles on open source digital photography

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Le 19/02/2013 18:02, Jean-François Rabasse a écrit :

> That's my paleolithic technique, rought but efficient. :-)

yes. With sqlite, remove the database and rebuild :-)

jdd

--
http://dodin.org
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Le 19/02/2013 18:38, Brian Morrison a écrit :

> Isn't this exactly why sidecar files came about? A way of keeping data
> associated with a file in such a way that programs that don't know
> about it don't mess with it?
>
yes, it's a good way, but when copying, it's at risk to be lost

jdd

--
http://dodin.org
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Tue, 19 Feb 2013, Gilles Caulier wrote:

> Jean François,
>
> Please report this investigation in this bugzilla file :
>
> https://bugs.kde.org/show_bug.cgi?id=268688

Gilles,
it's not a bug. It's a *too* complicated problem with *too* many
possible choices. And whatever fix is done, some users may agree
and some other users disagree, depending on who does what.

The actuel « merge tags » choice is a problem for reorganization
but may be useful in some occasions : a user tags an image with
Digikam and writes metadata to files. Then the image goes into
LightRoom and the user adds other tags.
Back to DK, re-read should, for this user, also merge the new
lr:hierarchicalSubject.
But this would be possible only if the DK tags are removed from the
file metadata, if not DK will not even read LightRoom data.
And it doesn't seem wise to provide also options to select in what
order data should be read, the Metadata settings folder would no
longer fit on a screen :-)

That's why the problem is complicated. Providing all possible
options and choices at application level would generate a monster.
(In French a « usine à gaz » :-)

My opinion : you, Digikam developpers, do a great work but you can't
be in all users minds. So we, Digikam users, should probably help in
analysis, collect use cases, collect problems and why, in order to
produce useable feedback and wishlists.
My previous analysis reflects only what I was looking for, regarding
my personal usage of DK. And it's useless because it only states
facts, no guidelines.

But if interested and concerned users are ok to help that way,
I'm ok to get feedback (use cases, problems, wishes) on my private
e-mail, not to spam this list too much.
Then try to do a synthesis and give you feedback.

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Gilles Caulier-4]

2013/2/19 Jean-François Rabasse <[hidden email]>

On Tue, 19 Feb 2013, Gilles Caulier wrote:

Jean François,

Please report this investigation in this bugzilla file :

https://bugs.kde.org/show_bug.cgi?id=268688

Gilles,
it's not a bug. It's a *too* complicated problem with *too* many possible choices. And whatever fix is done, some users may agree
and some other users disagree, depending on who does what.

The actuel « merge tags » choice is a problem for reorganization
but may be useful in some occasions : a user tags an image with
Digikam and writes metadata to files. Then the image goes into
LightRoom and the user adds other tags.
Back to DK, re-read should, for this user, also merge the new
lr:hierarchicalSubject.
But this would be possible only if the DK tags are removed from the
file metadata, if not DK will not even read LightRoom data.
And it doesn't seem wise to provide also options to select in what
order data should be read, the Metadata settings folder would no
longer fit on a screen :-)

That's why the problem is complicated. Providing all possible
options and choices at application level would generate a monster.
(In French a « usine à gaz » :-)

This will not be a "Usine à gaz" (GPL of course) if a clean metadata control panel is done about this subject. I already talk about this topic with Marcel in the pass. The goal is to check which option must be turned on by default to satisfy a lots of users by default. 

Secondary, the big question to know is how main Photo management program work with tags workflow in metadata. When i said main applications, i said real pro photograph application, as Aperture, NX, LR, etc...

And definitively, this cannot be an introduce small project for students, but a real GoSC 2013 project, because it's more complex that i supposed in my previous mail.

Gilles

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Le 19/02/2013 18:46, Gilles Caulier a écrit :

> And definitively, this cannot be an introduce small project for
> students, but a real GoSC 2013 project, because it's more complex that
> i supposed in my previous mail.

are there on this list people that needs merging of the tags in the
database?

if not it could be very easy to stop messing tags (the solution was
given in the JFR mail).

that's not to say that other problems have to be ignored, but building a
gscoc is often a way to forget a task, not all students are serious (I
have experienced that)

thanks
jdd
--
http://dodin.org
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users