[digikam] [Bug 362023] New: Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

            Bug ID: 362023
           Summary: Extremely slow metadata writing via maintenance
           Product: digikam
           Version: 5.0.0
          Platform: Debian unstable
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: Metadata
          Assignee: [hidden email]
          Reporter: [hidden email]

I am writing tags that previously only existed in (sqlite) database to image
metadata via the maintenance tool. This takes an enormous amount of time, I am
currently at 10% after 3 days continuously running. The bottleneck is
(obviously) disk io. Still this should take much less time (exiftool took about
3h to delete old tags of the same collection). The collection contains 100'000
items and probably about half of them are to be tagged.
What looks odd to me is, that throughout the process digikam.general reports
that QFileSystemWatcher detected change in the folder that is currently written
to by the metadata write. The occurrence of this is also not regular and at a
fixed position within the log entries from writing metadata to tags, which
suggest to me that the two things are separate.

As it is digikam that is writing to these folders and not an external programs,
are these scans really necessary?
And if they are, can they be delayed till the maintenance tool finished writing
to a directory.

I do not see an option to attach something. I will get a partial log into this
bug report as soon as I found out how (or use a pastebin).

Reproducible: Always

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #1 from Simon <[hidden email]> ---
Created attachment 98490
  --> https://bugs.kde.org/attachment.cgi?id=98490&action=edit
Command line output of writing metadata to files.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #2 from Maik Qualmann <[hidden email]> ---
Git commit 773f9361e5df5904c938ad9ee4cbb19acd1aa1f6 by Maik Qualmann.
Committed on 21/04/2016 at 19:03.
Pushed by mqualmann into branch 'master'.

fix absolute file path without symbolic links

M  +2    -2    libs/dmetadata/metaengine.cpp

http://commits.kde.org/digikam/773f9361e5df5904c938ad9ee4cbb19acd1aa1f6

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

Maik Qualmann <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from Maik Qualmann <[hidden email]> ---
This commit fixes only the writing of metadata for images which are linked via
a symbolic link.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #4 from Maik Qualmann <[hidden email]> ---
Created attachment 98521
  --> https://bugs.kde.org/attachment.cgi?id=98521&action=edit
scancontroller.patch

Can you try this test patch? And report how digiKam now behaves.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #5 from Simon <[hidden email]> ---
Thanks for looking into this. I applied your patch. The results seems to be the
same (maybe somewhat less frequent rescans). I attached the initial part of the
log after startup, where redundant stuff is excluded (marked by [...]). The
actual scanning starts at line 400. A second command line output is from later
on during the scan.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #6 from Simon <[hidden email]> ---
Created attachment 98527
  --> https://bugs.kde.org/attachment.cgi?id=98527&action=edit
Startup of digikam and start of writing metadata with  scancontroller.patch

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #7 from Simon <[hidden email]> ---
Created attachment 98528
  --> https://bugs.kde.org/attachment.cgi?id=98528&action=edit
Command line output in the middle of writing metadat to files with
scancontroller.patch

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Simon Frei]

And the scan is now going clearly slower than before the patch. I am now
running it almost two days and its at 3% only.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #8 from Simon <[hidden email]> ---
And the scan is now going clearly slower than before the patch. I am now
running it almost two days and its at 3% only.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

Maik Qualmann <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #98521|0                           |1
        is obsolete|                            |

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #9 from Maik Qualmann <[hidden email]> ---
Created attachment 98651
  --> https://bugs.kde.org/attachment.cgi?id=98651&action=edit
scancontroller2.patch

That it is now slowly working because now images are processed with symbolic
links.
Please try this patch. He also adds a time measurement.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #10 from Simon <[hidden email]> ---
Do I apply this patch on top of the current HEAD or on top of your
previous patch? I guess the first, but just to be sure.

On 27/04/16 21:14, Maik Qualmann via KDE Bugzilla wrote:

> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #9 from Maik Qualmann <[hidden email]> ---
> Created attachment 98651
>    --> https://bugs.kde.org/attachment.cgi?id=98651&action=edit
> scancontroller2.patch
>
> That it is now slowly working because now images are processed with symbolic
> links.
> Please try this patch. He also adds a time measurement.
>
> Maik
>

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Simon Frei]

Do I apply this patch on top of the current HEAD or on top of your
previous patch? I guess the first, but just to be sure.

On 27/04/16 21:14, Maik Qualmann via KDE Bugzilla wrote:

> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #9 from Maik Qualmann <[hidden email]> ---
> Created attachment 98651
>    --> https://bugs.kde.org/attachment.cgi?id=98651&action=edit
> scancontroller2.patch
>
> That it is now slowly working because now images are processed with symbolic
> links.
> Please try this patch. He also adds a time measurement.
>
> Maik
>

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #11 from Simon <[hidden email]> ---
Created attachment 98664
  --> https://bugs.kde.org/attachment.cgi?id=98664&action=edit
Startup of digikam and start of writing metadata with scancontroller2.patch

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #12 from Simon <[hidden email]> ---
Created attachment 98665
  --> https://bugs.kde.org/attachment.cgi?id=98665&action=edit
Command line output in the middle of writing metadat to files with
scancontroller2.patch

I applied the patch and added the command line output in the same fashion as
before.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #13 from Maik Qualmann <[hidden email]> ---
This is are long waiting times, up to 5 seconds until a scan is completed for
one image. Disabling the scanning does not help, he would be rescheduled in any
case. Modification date or file size have changed and need to be updated in the
DB. Writing to the SQLite DB is the time problem. The SQLite DB to put on an
SSD drive is strongly recommended. Here are a few measured values, writing of
one image information in the DB this include read new information from image
(images on HDD - EXT4):

HDD:
SQLite: 180-270ms
internal MySQL: 40-70ms

SSD:
SQLite: 30-60ms

Are the images on an NTFS partition? Is also here the SQLite DB?

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Simon Frei]

Indeed, my setup is far from optimal for disk io. I have both the
database and the images on a ntfs pratition of a hard disk on my laptop
(at least not system hd). I thought that the database would be
automatically cached in ram. I will look at it again some time.
Thanks again for your help.

On 02/05/16 21:14, Maik Qualmann via KDE Bugzilla wrote:

> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #13 from Maik Qualmann <[hidden email]> ---
> This is are long waiting times, up to 5 seconds until a scan is completed for
> one image. Disabling the scanning does not help, he would be rescheduled in any
> case. Modification date or file size have changed and need to be updated in the
> DB. Writing to the SQLite DB is the time problem. The SQLite DB to put on an
> SSD drive is strongly recommended. Here are a few measured values, writing of
> one image information in the DB this include read new information from image
> (images on HDD - EXT4):
>
> HDD:
> SQLite: 180-270ms
> internal MySQL: 40-70ms
>
> SSD:
> SQLite: 30-60ms
>
> Are the images on an NTFS partition? Is also here the SQLite DB?
>
> Maik
>

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #14 from Simon <[hidden email]> ---
Indeed, my setup is far from optimal for disk io. I have both the
database and the images on a ntfs pratition of a hard disk on my laptop
(at least not system hd). I thought that the database would be
automatically cached in ram. I will look at it again some time.
Thanks again for your help.

On 02/05/16 21:14, Maik Qualmann via KDE Bugzilla wrote:

> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #13 from Maik Qualmann <[hidden email]> ---
> This is are long waiting times, up to 5 seconds until a scan is completed for
> one image. Disabling the scanning does not help, he would be rescheduled in any
> case. Modification date or file size have changed and need to be updated in the
> DB. Writing to the SQLite DB is the time problem. The SQLite DB to put on an
> SSD drive is strongly recommended. Here are a few measured values, writing of
> one image information in the DB this include read new information from image
> (images on HDD - EXT4):
>
> HDD:
> SQLite: 180-270ms
> internal MySQL: 40-70ms
>
> SSD:
> SQLite: 30-60ms
>
> Are the images on an NTFS partition? Is also here the SQLite DB?
>
> Maik
>

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

Bizy <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #15 from Bizy <[hidden email]> ---
Same here (Ubuntu 16.4).

Already more than 4 hours to update (via 'Maintenance', tags database -->
images) a folder with some 4000 images.  Memory use more than 4 Gb...
Progression window still indicating 0%...
Guess that's not how it's supposed to be...

Workaround:  selecting all images and same command via 'Edit', takes 15
minutes...

PS:  If you want me to do something, please be very specific... (most of the
conversation above is beyond my comprehension...)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel