[digiKam-users] Tags manage - performance

[ceinmart]

Hi, 

I'm new to Digikam , using version 6.4.

I imported my photo library which is ~70k photos.

I'm using SQLite over an SSD hard disk.

Before Digikam I use the Picasa Desktop to manage my photos where I always keep my organization and tags.

I also set geotag using a program called geosetter which is very handy to set all the geolocation of my photos (since my camera is old and don't have a GPS embedded into).

This program set for each photo the tags "geo:lat=#####" and "geo:lon=#####" .

I have this only in ~5% of my photos, which gives ~8k tags (based into this SQL: select count(*) from tags where name like 'geo:%'; ).

Now I'm trying to organize my tags inside of digicam. 

I don't want to remove theses tags, I want to use the tree set resource for better organization. 

However, at the "Tags Manager" moving theses tags inside of a parent tag (called geotagged) is very, very, very slow. 

Tooks ~3 secs to move each tag inside them. 

I check this monitoring the table TagsTree (with dbeaver sqlclient : select count(*) from TagsTree tt ;). 

I already added an index over this table and Tags table to see if help, but appear to have no effect. 

My guess is the slowness is into the client tree view object where is affecting the update process.

I don't know if affect this, but I have my metadata configuration set to "lazy sync" 

Any tips?

[ceinmart]

If I close the DK application and update the table tags and insert manually into tagstree, will work? 

Since there is no trigger to update tags only I think I will need to manually create the record into tagstree table. 

I'm pretty sure that is not recommendable but this behavior is very annoying and it took too long. 

Em ter., 7 de jul. de 2020 às 10:52, Cesar Inacio Martins <[hidden email]> escreveu:

Hi, 

I'm new to Digikam , using version 6.4.

I imported my photo library which is ~70k photos.

I'm using SQLite over an SSD hard disk.

Before Digikam I use the Picasa Desktop to manage my photos where I always keep my organization and tags.

I also set geotag using a program called geosetter which is very handy to set all the geolocation of my photos (since my camera is old and don't have a GPS embedded into).

This program set for each photo the tags "geo:lat=#####" and "geo:lon=#####" .

I have this only in ~5% of my photos, which gives ~8k tags (based into this SQL: select count(*) from tags where name like 'geo:%'; ).

Now I'm trying to organize my tags inside of digicam. 

I don't want to remove theses tags, I want to use the tree set resource for better organization. 

However, at the "Tags Manager" moving theses tags inside of a parent tag (called geotagged) is very, very, very slow. 

Tooks ~3 secs to move each tag inside them. 

I check this monitoring the table TagsTree (with dbeaver sqlclient : select count(*) from TagsTree tt ;). 

I already added an index over this table and Tags table to see if help, but appear to have no effect. 

My guess is the slowness is into the client tree view object where is affecting the update process.

I don't know if affect this, but I have my metadata configuration set to "lazy sync" 

Any tips?

[woenx]

I also migrated from Picasa to Digikam a while ago.


Ok, so first of all, version 6.4 is a bit old now, and lots of bugs have
been fixed in the new 7.0 release (the final version will be released in
a few days, I think).

The good thing with digikam, is that it respects the previous metadata
made by other software, but also saves the new metadata into the file or
sidecar so it's permanent. That means that, moving a tag around a tree
will trigger writing that change to all pictures that share the tags
that have been affected by the move/merge. Basically digikam has to read
and then write each one of the files. So for a large library, it can
take a while. But the idea is that you have a previous idea of the
hierarchy you want to implement, and minimize the number of "moves" you
want to make in the tree. (e.g. I have several "root" tags, the main
ones being People and Places, so moving a branch of places outside of it
can affect thousands of pictures).

El 7/7/20 a les 15:52, Cesar Inacio Martins ha escrit:

> Hi,
>
> I'm new to Digikam , using version 6.4.
> I imported my photo library which is ~70k photos.
> I'm using SQLite over an SSD hard disk.
>
> Before Digikam I use the Picasa Desktop to manage my photos where I
> always keep my organization and tags.
> I also set geotag using a program called geosetter which is very handy
> to set all the geolocation of my photos (since my camera is old and
> don't have a GPS embedded into).
> This program set for each photo the tags "geo:lat=#####" and
> "geo:lon=#####" .
> I have this only in ~5% of my photos, which gives ~8k tags (based into
> this SQL: select count(*) from tags where name like 'geo:%'; ).
>
> Now I'm trying to organize my tags inside of digicam.
> I don't want to remove theses tags, I want to use the tree set
> resource for better organization.
> However, at the "Tags Manager" moving theses tags inside of a parent
> tag (called geotagged) is very, very, very slow.
> Tooks ~3 secs to move each tag inside them.
> I check this monitoring the table TagsTree (with dbeaver sqlclient :
> select count(*) from TagsTree tt ;).
> I already added an index over this table and Tags table to see if
> help, but appear to have no effect.
> My guess is the slowness is into the client tree view object where is
> affecting the update process.
>
> I don't know if affect this, but I have my metadata configuration set
> to "lazy sync"
>
> Any tips?
>
>

[ceinmart]

Hi Marc, 

Thanks for your answers. 

So, about version 6.4 , I choose it because is the last official stable version. 

I really don't like to put in risk my photos with a not stable program.

(yes, I have backup and etc, but who want to recover from backup.. very annoying right)

I hope the version 7 should be stable enough, then I already will upgrade to it.

About the tag performance, what happened: 

1) the moving of theses geotags took ~3 hours.

  During this, the application don't consume more than 15% of CPU and no disk write. 

  Apparently it worked only into the metadata database on SQLite. 

 2) Then I click on Album -> Write metadata to file 

  Remember that my configurations are set to lazy synchronization.

  Apparently they update few photos files, but I have no sure what they did because there is no log in the application to check.

  They were very fast, took ~1 minute. 

   I believe they update other changes I was made before and not that 8k tags.

   Also because probably all photos with tag "geo:*" already have the "geotagged" tag, so no changes need. 

 3) now I also see the tree structure I did into the DK don't go to the photo file, only the individual tags.

That said, for me, the application still have a logic to do this movement of tags where don't get into consideration a high volume of tags or was never tested with this objective. 

Em ter., 7 de jul. de 2020 às 16:25, <[hidden email]> escreveu:

I also migrated from Picasa to Digikam a while ago.


Ok, so first of all, version 6.4 is a bit old now, and lots of bugs have
been fixed in the new 7.0 release (the final version will be released in
a few days, I think).

The good thing with digikam, is that it respects the previous metadata
made by other software, but also saves the new metadata into the file or
sidecar so it's permanent. That means that, moving a tag around a tree
will trigger writing that change to all pictures that share the tags
that have been affected by the move/merge. Basically digikam has to read
and then write each one of the files. So for a large library, it can
take a while. But the idea is that you have a previous idea of the
hierarchy you want to implement, and minimize the number of "moves" you
want to make in the tree. (e.g. I have several "root" tags, the main
ones being People and Places, so moving a branch of places outside of it
can affect thousands of pictures).

El 7/7/20 a les 15:52, Cesar Inacio Martins ha escrit:
> Hi,
>
> I'm new to Digikam , using version 6.4.
> I imported my photo library which is ~70k photos.
> I'm using SQLite over an SSD hard disk.
>
> Before Digikam I use the Picasa Desktop to manage my photos where I
> always keep my organization and tags.
> I also set geotag using a program called geosetter which is very handy
> to set all the geolocation of my photos (since my camera is old and
> don't have a GPS embedded into).
> This program set for each photo the tags "geo:lat=#####" and
> "geo:lon=#####" .
> I have this only in ~5% of my photos, which gives ~8k tags (based into
> this SQL: select count(*) from tags where name like 'geo:%'; ).
>
> Now I'm trying to organize my tags inside of digicam.
> I don't want to remove theses tags, I want to use the tree set
> resource for better organization.
> However, at the "Tags Manager" moving theses tags inside of a parent
> tag (called geotagged) is very, very, very slow.
> Tooks ~3 secs to move each tag inside them.
> I check this monitoring the table TagsTree (with dbeaver sqlclient :
> select count(*) from TagsTree tt ;).
> I already added an index over this table and Tags table to see if
> help, but appear to have no effect.
> My guess is the slowness is into the client tree view object where is
> affecting the update process.
>
> I don't know if affect this, but I have my metadata configuration set
> to "lazy sync"
>
> Any tips?
>
>

[woenx]

From 6.4 to 7.0 is mostly bugfixes (some related to writing to the tag tree), so I don't think you'll ruin anything by trying a beta version. In any case, I've heard (somewhere in this mailing list) that the final 7.0 version will be released on the 10th of july.

So, what you say is quite weird. Usually what takes more times is writing to the files, but writing the changes to the database is quite fast. How many pictures are we talking about?

I also like to recreate my database from time to time. I just delete digikam's database, and let it rebuild it from the metadata in the pictures. It might take a few hours, but I do it every several months and it's a way to be sure that your database<->metadata changes are synchronized. So I am not really worried about losing the database.

El 7/7/20 a les 21:43, Cesar Inacio Martins ha escrit:

Hi Marc, 

Thanks for your answers. 

So, about version 6.4 , I choose it because is the last official stable version. 

I really don't like to put in risk my photos with a not stable program.

(yes, I have backup and etc, but who want to recover from backup.. very annoying right)

I hope the version 7 should be stable enough, then I already will upgrade to it.

About the tag performance, what happened: 

1) the moving of theses geotags took ~3 hours.

  During this, the application don't consume more than 15% of CPU and no disk write. 

  Apparently it worked only into the metadata database on SQLite. 

 2) Then I click on Album -> Write metadata to file 

  Remember that my configurations are set to lazy synchronization.

  Apparently they update few photos files, but I have no sure what they did because there is no log in the application to check.

  They were very fast, took ~1 minute. 

   I believe they update other changes I was made before and not that 8k tags.

   Also because probably all photos with tag "geo:*" already have the "geotagged" tag, so no changes need. 

 3) now I also see the tree structure I did into the DK don't go to the photo file, only the individual tags.

That said, for me, the application still have a logic to do this movement of tags where don't get into consideration a high volume of tags or was never tested with this objective. 

Em ter., 7 de jul. de 2020 às 16:25, <[hidden email]> escreveu:

I also migrated from Picasa to Digikam a while ago.


Ok, so first of all, version 6.4 is a bit old now, and lots of bugs have
been fixed in the new 7.0 release (the final version will be released in
a few days, I think).

The good thing with digikam, is that it respects the previous metadata
made by other software, but also saves the new metadata into the file or
sidecar so it's permanent. That means that, moving a tag around a tree
will trigger writing that change to all pictures that share the tags
that have been affected by the move/merge. Basically digikam has to read
and then write each one of the files. So for a large library, it can
take a while. But the idea is that you have a previous idea of the
hierarchy you want to implement, and minimize the number of "moves" you
want to make in the tree. (e.g. I have several "root" tags, the main
ones being People and Places, so moving a branch of places outside of it
can affect thousands of pictures).

El 7/7/20 a les 15:52, Cesar Inacio Martins ha escrit:
> Hi,
>
> I'm new to Digikam , using version 6.4.
> I imported my photo library which is ~70k photos.
> I'm using SQLite over an SSD hard disk.
>
> Before Digikam I use the Picasa Desktop to manage my photos where I
> always keep my organization and tags.
> I also set geotag using a program called geosetter which is very handy
> to set all the geolocation of my photos (since my camera is old and
> don't have a GPS embedded into).
> This program set for each photo the tags "geo:lat=#####" and
> "geo:lon=#####" .
> I have this only in ~5% of my photos, which gives ~8k tags (based into
> this SQL: select count(*) from tags where name like 'geo:%'; ).
>
> Now I'm trying to organize my tags inside of digicam.
> I don't want to remove theses tags, I want to use the tree set
> resource for better organization.
> However, at the "Tags Manager" moving theses tags inside of a parent
> tag (called geotagged) is very, very, very slow.
> Tooks ~3 secs to move each tag inside them.
> I check this monitoring the table TagsTree (with dbeaver sqlclient :
> select count(*) from TagsTree tt ;).
> I already added an index over this table and Tags table to see if
> help, but appear to have no effect.
> My guess is the slowness is into the client tree view object where is
> affecting the update process.
>
> I don't know if affect this, but I have my metadata configuration set
> to "lazy sync"
>
> Any tips?
>
>

[ceinmart]

> From 6.4 to 7.0 is mostly bugfixes (some related to writing to the tag tree), so
> I don't think you'll ruin anything by trying a beta version. In any case, I've heard
> (somewhere in this mailing list) that the final 7.0 version will be released on the 10th of July.

Nice! for sure I will install it as soon they be released. 
( 18 July : http://digikam.1695700.n4.nabble.com/digiKam-users-7RC-Slide-Show-tp4713033p4713037.html )

> So, what you say is quite weird. Usually what takes more times is writing to the files,
> but writing the changes to the database is quite fast. How many pictures are we talking about?

Is 4143 pictures. 
I understand the tag tree configuration isn't forward to the picture metadata. 
If I configure the tree: geotagged/geo:lat=1234 and the picture don't have the tag "geotagged" they don't add it into the picture. 

So, no changes in the picture file are made.

Em ter., 7 de jul. de 2020 às 16:51, <[hidden email]> escreveu:

From 6.4 to 7.0 is mostly bugfixes (some related to writing to the tag tree), so I don't think you'll ruin anything by trying a beta version. In any case, I've heard (somewhere in this mailing list) that the final 7.0 version will be released on the 10th of july.

So, what you say is quite weird. Usually what takes more times is writing to the files, but writing the changes to the database is quite fast. How many pictures are we talking about?

I also like to recreate my database from time to time. I just delete digikam's database, and let it rebuild it from the metadata in the pictures. It might take a few hours, but I do it every several months and it's a way to be sure that your database<->metadata changes are synchronized. So I am not really worried about losing the database.

El 7/7/20 a les 21:43, Cesar Inacio Martins ha escrit:

Hi Marc, 

Thanks for your answers. 

So, about version 6.4 , I choose it because is the last official stable version. 

I really don't like to put in risk my photos with a not stable program.

(yes, I have backup and etc, but who want to recover from backup.. very annoying right)

I hope the version 7 should be stable enough, then I already will upgrade to it.

About the tag performance, what happened: 

1) the moving of theses geotags took ~3 hours.

  During this, the application don't consume more than 15% of CPU and no disk write. 

  Apparently it worked only into the metadata database on SQLite. 

 2) Then I click on Album -> Write metadata to file 

  Remember that my configurations are set to lazy synchronization.

  Apparently they update few photos files, but I have no sure what they did because there is no log in the application to check.

  They were very fast, took ~1 minute. 

   I believe they update other changes I was made before and not that 8k tags.

   Also because probably all photos with tag "geo:*" already have the "geotagged" tag, so no changes need. 

 3) now I also see the tree structure I did into the DK don't go to the photo file, only the individual tags.

That said, for me, the application still have a logic to do this movement of tags where don't get into consideration a high volume of tags or was never tested with this objective. 

Em ter., 7 de jul. de 2020 às 16:25, <[hidden email]> escreveu:

I also migrated from Picasa to Digikam a while ago.


Ok, so first of all, version 6.4 is a bit old now, and lots of bugs have
been fixed in the new 7.0 release (the final version will be released in
a few days, I think).

The good thing with digikam, is that it respects the previous metadata
made by other software, but also saves the new metadata into the file or
sidecar so it's permanent. That means that, moving a tag around a tree
will trigger writing that change to all pictures that share the tags
that have been affected by the move/merge. Basically digikam has to read
and then write each one of the files. So for a large library, it can
take a while. But the idea is that you have a previous idea of the
hierarchy you want to implement, and minimize the number of "moves" you
want to make in the tree. (e.g. I have several "root" tags, the main
ones being People and Places, so moving a branch of places outside of it
can affect thousands of pictures).

El 7/7/20 a les 15:52, Cesar Inacio Martins ha escrit:
> Hi,
>
> I'm new to Digikam , using version 6.4.
> I imported my photo library which is ~70k photos.
> I'm using SQLite over an SSD hard disk.
>
> Before Digikam I use the Picasa Desktop to manage my photos where I
> always keep my organization and tags.
> I also set geotag using a program called geosetter which is very handy
> to set all the geolocation of my photos (since my camera is old and
> don't have a GPS embedded into).
> This program set for each photo the tags "geo:lat=#####" and
> "geo:lon=#####" .
> I have this only in ~5% of my photos, which gives ~8k tags (based into
> this SQL: select count(*) from tags where name like 'geo:%'; ).
>
> Now I'm trying to organize my tags inside of digicam.
> I don't want to remove theses tags, I want to use the tree set
> resource for better organization.
> However, at the "Tags Manager" moving theses tags inside of a parent
> tag (called geotagged) is very, very, very slow.
> Tooks ~3 secs to move each tag inside them.
> I check this monitoring the table TagsTree (with dbeaver sqlclient :
> select count(*) from TagsTree tt ;).
> I already added an index over this table and Tags table to see if
> help, but appear to have no effect.
> My guess is the slowness is into the client tree view object where is
> affecting the update process.
>
> I don't know if affect this, but I have my metadata configuration set
> to "lazy sync"
>
> Any tips?
>
>

[ceinmart]

Marc, 

Now I see why you found the behave I said strange.  

I just start to organize my tags in the tag manager.

I move a tag inside other, now where they affect ~800 pictures. 

I created a tag called "0_object" and put a tag called "bicicleta" inside.

This time the DK ask about make the update of the pictures in front or background. 

Before confirm I saved the metadata , apply the changes and get the metadata to compare. 

Here is what they modify into my pictures : https://imgur.com/a/NORTy1T 

This behave doesn't happen when I move the geotags, perhaps some bug (!?) .

Then I renamed the tag "0_object"  to only "object" and they don't apply to the pictures , another bug!?

Well, I will avoid this organization of my tags now and wait for version 7 where I hope they solve these issues. 

Em ter., 7 de jul. de 2020 às 17:35, Cesar Inacio Martins <[hidden email]> escreveu:

> From 6.4 to 7.0 is mostly bugfixes (some related to writing to the tag tree), so
> I don't think you'll ruin anything by trying a beta version. In any case, I've heard
> (somewhere in this mailing list) that the final 7.0 version will be released on the 10th of July.

Nice! for sure I will install it as soon they be released. 
( 18 July : http://digikam.1695700.n4.nabble.com/digiKam-users-7RC-Slide-Show-tp4713033p4713037.html )

> So, what you say is quite weird. Usually what takes more times is writing to the files,
> but writing the changes to the database is quite fast. How many pictures are we talking about?

Is 4143 pictures. 
I understand the tag tree configuration isn't forward to the picture metadata. 
If I configure the tree: geotagged/geo:lat=1234 and the picture don't have the tag "geotagged" they don't add it into the picture. 

So, no changes in the picture file are made.

Em ter., 7 de jul. de 2020 às 16:51, <[hidden email]> escreveu:

From 6.4 to 7.0 is mostly bugfixes (some related to writing to the tag tree), so I don't think you'll ruin anything by trying a beta version. In any case, I've heard (somewhere in this mailing list) that the final 7.0 version will be released on the 10th of july.

So, what you say is quite weird. Usually what takes more times is writing to the files, but writing the changes to the database is quite fast. How many pictures are we talking about?

I also like to recreate my database from time to time. I just delete digikam's database, and let it rebuild it from the metadata in the pictures. It might take a few hours, but I do it every several months and it's a way to be sure that your database<->metadata changes are synchronized. So I am not really worried about losing the database.

El 7/7/20 a les 21:43, Cesar Inacio Martins ha escrit:

Hi Marc, 

Thanks for your answers. 

So, about version 6.4 , I choose it because is the last official stable version. 

I really don't like to put in risk my photos with a not stable program.

(yes, I have backup and etc, but who want to recover from backup.. very annoying right)

I hope the version 7 should be stable enough, then I already will upgrade to it.

About the tag performance, what happened: 

1) the moving of theses geotags took ~3 hours.

  During this, the application don't consume more than 15% of CPU and no disk write. 

  Apparently it worked only into the metadata database on SQLite. 

 2) Then I click on Album -> Write metadata to file 

  Remember that my configurations are set to lazy synchronization.

  Apparently they update few photos files, but I have no sure what they did because there is no log in the application to check.

  They were very fast, took ~1 minute. 

   I believe they update other changes I was made before and not that 8k tags.

   Also because probably all photos with tag "geo:*" already have the "geotagged" tag, so no changes need. 

 3) now I also see the tree structure I did into the DK don't go to the photo file, only the individual tags.

That said, for me, the application still have a logic to do this movement of tags where don't get into consideration a high volume of tags or was never tested with this objective. 

Em ter., 7 de jul. de 2020 às 16:25, <[hidden email]> escreveu:

I also migrated from Picasa to Digikam a while ago.


Ok, so first of all, version 6.4 is a bit old now, and lots of bugs have
been fixed in the new 7.0 release (the final version will be released in
a few days, I think).

The good thing with digikam, is that it respects the previous metadata
made by other software, but also saves the new metadata into the file or
sidecar so it's permanent. That means that, moving a tag around a tree
will trigger writing that change to all pictures that share the tags
that have been affected by the move/merge. Basically digikam has to read
and then write each one of the files. So for a large library, it can
take a while. But the idea is that you have a previous idea of the
hierarchy you want to implement, and minimize the number of "moves" you
want to make in the tree. (e.g. I have several "root" tags, the main
ones being People and Places, so moving a branch of places outside of it
can affect thousands of pictures).

El 7/7/20 a les 15:52, Cesar Inacio Martins ha escrit:
> Hi,
>
> I'm new to Digikam , using version 6.4.
> I imported my photo library which is ~70k photos.
> I'm using SQLite over an SSD hard disk.
>
> Before Digikam I use the Picasa Desktop to manage my photos where I
> always keep my organization and tags.
> I also set geotag using a program called geosetter which is very handy
> to set all the geolocation of my photos (since my camera is old and
> don't have a GPS embedded into).
> This program set for each photo the tags "geo:lat=#####" and
> "geo:lon=#####" .
> I have this only in ~5% of my photos, which gives ~8k tags (based into
> this SQL: select count(*) from tags where name like 'geo:%'; ).
>
> Now I'm trying to organize my tags inside of digicam.
> I don't want to remove theses tags, I want to use the tree set
> resource for better organization.
> However, at the "Tags Manager" moving theses tags inside of a parent
> tag (called geotagged) is very, very, very slow.
> Tooks ~3 secs to move each tag inside them.
> I check this monitoring the table TagsTree (with dbeaver sqlclient :
> select count(*) from TagsTree tt ;).
> I already added an index over this table and Tags table to see if
> help, but appear to have no effect.
> My guess is the slowness is into the client tree view object where is
> affecting the update process.
>
> I don't know if affect this, but I have my metadata configuration set
> to "lazy sync"
>
> Any tips?
>
>