[digikam] [Bug 376891] New: Digikam becomes unusable with many Metatags

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

            Bug ID: 376891
           Summary: Digikam becomes unusable with many Metatags
           Product: digikam
           Version: 5.4.0
          Platform: Neon Packages
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: NOR
         Component: Metadata
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

Problem/Motivation
I tried Digikam as a replacement for my current Mac/Aperture setup. I tried on
my Linux Box first with some images and everything does well, nice :) So I
started the actual migration of around 28.000 images and let Digikam finish all
work. Some hours later all thumbs and other stuff were created okay.

The actual problem started when it comes to Metadata. Searching for a tag takes
ages, but when it comes to tagging, digikam totally failed :(
I have 5000+ tags in my list all at first level (no nested tags)
Opening the Tag Manager takes around 15seconds, adding a new tag or moving a
tag takes around 1 minute and digikam doesn't respond to anything during that.

My first idea: Okay there are missing database indexes, but that doesn't seem
the case.
Next idea, IO problem: I moved the digikam SQLiteDB to RAM, but no effect.
Next idea, SQLite problem: Even knowing SQLite is usually really fast I gave
MySQL a try - but as to expect - no effect.

So the problem is with digikam itself. As the CPU usage goes up to 100% I think
digikam stucks during tags qt-list-building/rendering or whatever.

Any ideas ;)

Thanks

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from [hidden email] ---
Where digiKam waste time at startup ? Can you investigate using kcallgrind tool
? This will indicate which code will be called a lots of time in this
situation...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #2 from [hidden email] ---
Created attachment 104212
  --> https://bugs.kde.org/attachment.cgi?id=104212&action=edit
Valgrind log

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #3 from [hidden email] ---
(In reply to caulier.gilles from comment #1)
> Where digiKam waste time at startup ? Can you investigate using kcallgrind
> tool ? This will indicate which code will be called a lots of time in this
> situation...
>
> Gilles Caulier

Hi Gilles,

Digikam starts within a minute or two, so that is no problem for me as I
usually will start it once a day. The problem comes up when I try to
add/edit/delete a tag, Digikam will freeze for a whole minute until done.

I just installed valgrind and started digikam like this:
valgrind --tool=callgrind digikam

I get lots of segment overflows like :
==5290== brk segment overflow in thread #1: can't grow to 0x4a38000

and I am waiting now 20 minutes for Digikam to start up ;)

I found some info on stackoverflow:

http://stackoverflow.com/questions/35129135/valgrind-reporting-a-segment-overflow

still not sure if that is a problem at all. However, seems I can't even get
Digikam to start up running valgrind. I killed the process and attaching the
output till here, maybe it's already usefull for debugging.

Thanks!

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #4 from [hidden email] ---
new 5.6.0 pre-release as bundle is available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Please check if this problem still reproducible with these versions.

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #5 from [hidden email] ---
Thanks Gilles for looking into that.

Unfortunately I think it didn't change anything. I'm now using Digikam on my
more powerful MacBook Pro with SSD but still have the problem with tags
management. Just to make sure I stopped some times to have something to compare
if you find something other to try :)
===
- Startup time digikam: 3min 20 seconds (mostly "Reading Database...")
- Startup time "Open Tag Manager" 23 seconds (scrolling through the list of
tags is fast BTW)
- Click on "+" to add a new tag, 5 seconds till window open.
- Entering a new tag in that window and click save makes digikam become
unresponsive for 1min and 33 seconds, doh! (e.g. digikam completely hangs)
- Deleting the newly created tag takes nearly the same time, 1min and 36s
===

Here are some stats from my database:

digikam version 5.6.0
Images:
JPG: 32227
total: 32227
:
Videos:
AVI: 1
total: 1
:
Total Items: 32228
Albums: 422
Tags: 5124
:
Database backend: QSQLITE

Thanks again!
cheers bzrudi

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #6 from [hidden email] ---
ok.

Just to be sure :

1/ backup your sqlite database files.
2/ rename it as .old or delete files.
3/ restart digiKam and repopulate the collections from scratch.
4/ Check if problem remain.

Notes :

- here i use more than 150000 items with sqlite and a lots of tags without any
special time latency.

- 5.6.0 has a database garbage collector tool in maintenance. Thi scan be a
solution to cleanup your original DB.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #7 from [hidden email] ---
Okay Gilles, did some more testing.
Started from scratch and sadly no change, problem exists. So I checked database
and started to simply delete some tags from the Tags relation. E.g. as i have
more than 5000 tags I started with:

sqlite> delete from Tags where id > 4500;
no real change...
sqlite> delete from Tags where id > 4000;
no real change...
sqlite> delete from Tags where id > 3000;
better, tag manager now starts within 7 seconds...
sqlite> delete from Tags where id > 1000;
nice, tag manager starts within 1-2 seconds. I would say things work within an
acceptable time.
going even further and just leave 500 tags within database things rock!

===
- Startup time digikam: 5000+ tags 3min 20 seconds vs. 500 tags 13 seconds!
- Startup time "Open Tag Manager": 5000+ 23 seconds vs. 500 tags 1 second!
- Click on "+" to add a new tag: 5000+ 5 seconds vs. 500 tags realtime
- Entering a new tag in that window and click save makes digikam become
unresponsive for 1min and 33 seconds, doh! vs. 500 tags 2 seconds
- Deleting the newly created tag takes nearly the same time, 1min and 36s vs.
500 tags 2 seconds
===


So things get massiv slower when reaching 2000 or 3000 tags. Feels some indexes
are missing!? Maybe I have the time tomorrow for testing out.

cheers bzrudi

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #8 from [hidden email] ---
I think the problem can be the auto-completer for tag names in search
tree-view.

This one is populated with database contents and is updated at each changes.

This will explain why with a fresh DB, the problem remain...

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #9 from [hidden email] ---
digiKam 5.6.0 is now released and available as bundle for Linux, MacOS and
Windows.

https://www.digikam.org/news/2017-06-21-5.6.0-release-announcement/

Can you check if problem still exists with this version ?

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=376891

--- Comment #10 from [hidden email] ---
New digiKam 5.7.0 are built with current implementation as pre-release bundles:

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Problem still reproducible ?

--
You are receiving this mail because:
You are the assignee for the bug.