[digikam] [Bug 318516] New: Improve Digikam loading time.

[Yann Le Hir]

https://bugs.kde.org/show_bug.cgi?id=318516

            Bug ID: 318516
           Summary: Improve Digikam loading time.
    Classification: Unclassified
           Product: digikam
           Version: unspecified
          Platform: Compiled Sources
                OS: Linux
            Status: UNCONFIRMED
          Severity: wishlist
          Priority: NOR
         Component: Usability
          Assignee: [hidden email]
          Reporter: [hidden email]

Right now the loading time for digikam is very high.
I use digikam with something like 16k Albums and 150k pictures.
Loading digikam can take up to 3-4 minutes with a high end CPU.
Passing the fact that digikam only uses 1 core to load, it should still be much
faster.

I have two patches that improves loading time tremendously.

Reproducible: Always

Steps to Reproduce:
1. Start digikam
2.
3.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Yann Le Hir]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #1 from Yann Le Hir <[hidden email]> ---
Created attachment 78989
  --> https://bugs.kde.org/attachment.cgi?id=78989&action=edit
Change loading sequence.

This patch change the loading sequence, so digikam scan albums before loading
the GUI. This way, GUI Objects don't get all first loading SIGNAL that take a
lot of time to handle for nothing...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Yann Le Hir]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #2 from Yann Le Hir <[hidden email]> ---
Created attachment 78990
  --> https://bugs.kde.org/attachment.cgi?id=78990&action=edit
Patch that fixes initialisation of ImageAlbumFilterModel

This patch removes two useless loops.
Indeed the loop call a method with an Album in parameter, but that method
doesn't really depend on the Album but more on the AlbumManager.
So I splitting the method in two other methods to avoid code redundance, and
call them appropriately.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318516

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]
          Component|Usability                   |Database
            Version|unspecified                 |3.1.0
            Summary|Improve Digikam loading     |Improve digiKam loading
                   |time.                       |time [patch]

--- Comment #3 from Gilles Caulier <[hidden email]> ---
Thanks for your patches...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[yann LE HIR-2]

https://bugs.kde.org/show_bug.cgi?id=318516

yann LE HIR <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #4 from yann LE HIR <[hidden email]> ---
Unforseen consequence of the patches,
digikam doesn't load the last album shown when previously quit.
I'll take a look at this.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[yann LE HIR-2]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #5 from Yann LE HIR <[hidden email]> ---
Created attachment 78993
  --> https://bugs.kde.org/attachment.cgi?id=78993&action=edit
Restore the main view when restoreSession in called by main.

Ok, so the view was restored when the AlbumManger emited allAlbumLoaded.
Which is due to my patch never received by the main view.
I joined a patch that put the restore view mechanism when restoreSession is
called by main.

This is becoming more intrusive that I would like...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #6 from Marcel Wiesweg <[hidden email]> ---
Ok, with 16000 albums I never tested. I concede that some relevant stuff scales
linearly.
With you patches, you cut your loading time by...an order of magnitude? I mean,
this was the problem?
Then it's not that intrusive but a very relevant fix. We only need to test
thoroughly for regressions.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[yann LE HIR-2]

https://bugs.kde.org/show_bug.cgi?id=318516

Yann LE HIR <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #78993|0                           |1
        is obsolete|                            |

--- Comment #7 from Yann LE HIR <[hidden email]> ---
Created attachment 79013
  --> https://bugs.kde.org/attachment.cgi?id=79013&action=edit
Cleaner patch  to change the loading sequence and keep the session restore.

I joined a cleaner patch for the loading sequence.

On a clean branch, with digikam compiled in debug mode.
I get an average loading time of 33.36s (with 10 launches)
With the patches I get an average loading time of 2.85s
Almost 12x faster :)
It's near 10x faster in release mode.

I'm pretty sure it was the main issue. The fact that you try to reload 16k
times all the filter models is quite a performance killer. Not to mention the
AbstractAlbumModel that also took something like 32k events (AboutToBeAdded +
Added)...
Without the patches, profiling on the loading of digikam can show up to 103
million calls to Album::next() and a fair amount of Album::title()...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Yann Le Hir]

https://bugs.kde.org/show_bug.cgi?id=318516

Yann Le Hir <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #78989|0                           |1
        is obsolete|                            |

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Smit Mehta]

https://bugs.kde.org/show_bug.cgi?id=318516

Smit Mehta <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #8 from Smit Mehta <[hidden email]> ---
Hi

I tested it on ~10 albums, and obviously I am not getting that big a speedup.
@Marcel, @Gilles, do we need more testing, or shall I apply this to git? I have
tested this. It doesn't seem to break anything.

Smit

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[yann LE HIR-2]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #9 from Yann LE HIR <[hidden email]> ---
Please do not apply.
This patch is not fixing everything, I did more testing, the scanDalbum with
the delay via kioslave etc is still loading after the graphic elements and thus
still slowing down everthing.
I have some ideas for a more long term, elegant solution to this issue.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gowtham Ashok]

https://bugs.kde.org/show_bug.cgi?id=318516

Gowtham Ashok <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #10 from Gowtham Ashok <[hidden email]> ---
I applied this patch and found that when loading 1000+ pictures, digiKam loads
pictures even after the loading window closes, which slows down digiKam
considerably. This does not happen in the clean one. Is this the same bug you
are facing Yann Le Hir?

I saw that much time is spent on waiting for SQLite( it's in locked state). I
think loading would be faster, if, when loading >1000 files, ~5-10 images are
committed to the DB at a time, rather than one at a time.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #11 from Gilles Caulier <[hidden email]> ---
Yann

Can you give a feedback about comment #10 from Gowtham, please ?

Marcel,

There is any work from you about to review/test this entry recently ?

Best

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318516

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Improve digiKam loading     |SCAN : Improve digiKam
                   |time [patch]                |loading time [patch]

--- Comment #12 from Gilles Caulier <[hidden email]> ---
Marcel,

Whats new about this file ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318516

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Database                    |Database-Scan

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=318516

--- Comment #13 from [hidden email] ---
To Yann le Hir,

I would to see more investigations about your patches. Can you update these
file to be compatible with current implementation from git/master based on Qt5
?

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel