[Bug 273077] New: Generate Fingerprints gets slower and slower

https://bugs.kde.org/show_bug.cgi?id=273077

           Summary: Generate Fingerprints gets slower and slower
           Product: digikam
           Version: 2.0.0
          Platform: Compiled Sources
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: NOR
         Component: Fuzzy Searches
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]


Version:           2.0.0 (using KDE 4.6.2)
OS:                Linux

When I generate fingerprints for 50.000 images, it is really fast in the
beginning, but it gets slower and slower during calculation.
One image takes around 0.5 seconds to calculate, but after a while it takes 2
seconds and more.
If I cancel the operation, restart digiKam and scan for missing fingerprints,
the same files that took over 2 seconds to calculate are handled in 0.5 seconds
again.

So it must have something to do with the amount of images that have been
already scanned / calculated. Maybe we are using a cache somewhere that gets
bigger and bigger and therefore slower to scan?
After a digiKam restart it is fast again, so it could be related to a hashmap
or vector that is growing fast.

Reproducible: Didn't try




OS: Linux (i686) release 2.6.38-ARCH
Compiler: gcc

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #1 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-05-12 22:32:19 ---
Do you see substantial growth in memory usage? Then massif can give us a hint.

My first suspect was pixmap generation and the progress dialog, but I think
this problem was fixed already. To rule out a problem here, disable the
relevant lines 183 and 184.

There is only a QStringList in the fingerprints generator, I doubt that is the
problem.

Second step is the HaarIface. To rule out any problems here, you could try to
use a fresh HaarIface object any time.

If all does not help - though it takes longer - we'd need to run digikam under
callgrind, and switch on instrumentation as soon the presumed bug's calculation
time is significant, like > 50%.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #2 from Andi Clemens <andi clemens gmx net>  2011-05-22 18:19:24 ---
I tried to profile this, but somehow valgrind doesn't seem to get information
about running kioslaves.
Any tip how to profile an kioslave?

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #3 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-05-23 20:20:31 ---
There is a hack to run an ioslave under gdb, but it's so complicated I never
tried.
Then, which ioslave is eating CPU? There only so much tasks done in an ioslave.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Francesco Riosa-2]

https://bugs.kde.org/show_bug.cgi?id=273077


Francesco Riosa <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #4 from Francesco Riosa <francesco+kde pnpitalia it>  2011-05-24 22:19:40 ---
I was here today ;-)
http://techbase.kde.org/Development/Tutorials/Debugging/Debugging_IOSlaves

could work

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #5 from Andi Clemens <andi clemens gmx net>  2011-06-22 00:37:20 ---
(In reply to comment #1)

> Do you see substantial growth in memory usage? Then massif can give us a hint.
>
> My first suspect was pixmap generation and the progress dialog, but I think
> this problem was fixed already. To rule out a problem here, disable the
> relevant lines 183 and 184.
>
> There is only a QStringList in the fingerprints generator, I doubt that is the
> problem.
>
> Second step is the HaarIface. To rule out any problems here, you could try to
> use a fresh HaarIface object any time.
>
> If all does not help - though it takes longer - we'd need to run digikam under
> callgrind, and switch on instrumentation as soon the presumed bug's calculation
> time is significant, like > 50%.

Memory consumption raises continuously, I disabled the fingerprint generation
and the pixmap generation and only left the loadingdescriptionthread acive.
Still the same result.
So I guess something is wrong in there?
I ran it through massif for 2 hours now, but actually I don't quite understand
the results massif gives me.
I'll attach them to this thread, maybe someone can extract vital information.

Andi

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #6 from Andi Clemens <andi clemens gmx net>  2011-06-22 00:37:59 ---
Created an attachment (id=61227)
 --> (http://bugs.kde.org/attachment.cgi?id=61227)
massif #1

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #7 from Andi Clemens <andi clemens gmx net>  2011-06-22 00:38:21 ---
Created an attachment (id=61228)
 --> (http://bugs.kde.org/attachment.cgi?id=61228)
massif #2

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #8 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-06-23 13:19:09 ---
You will want to use "massif-visualizer" to analyse these logs.
The first log has a peak memory usage of 181,2 MB, the second ends at 6,8 MB.
Both is not much, with today's images of 15 MPx easily taking more than 100 MB
of memory.
If there is a memory leak, I would expect a steadily increasing memory usage,
but looking at the first log here, there is a plateau which usually rules out
any leak.
You wrote above memory usage was increasing - which number did the system
monitor give you? I think the KDE system monitor gives a pretty good number for
used memory (which is not trivial)

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #9 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-06-23 13:24:02 ---
There is another possibility: May be there is no memory leaks, but some
structure is increasing in size (still small memory-wise) but is hugging CPU.
To find that, "callgrind" will help. It is key to start callgrind without
measurements (digikam runs at almost normal speed), provoke the bug, and when
the CPU usage is high, enable instrumentation:
valgrind --tool=callgrind --instr-atstart=no digikam
(make CPU usage go up)
callgrind_control -i on
(collect one or two minutes of CPU)
callgrind_control -k
(kills digikam, dumps output)
Output is to be analysed with kcachegrind

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #10 from Andi Clemens <andi clemens gmx net>  2011-06-23 16:08:54 ---
The memory increased by 1MB each second, I watched it with the KDE system
monitor.
I will try callgrind again, maybe these outputs will be more helpful.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #11 from Andi Clemens <andi clemens gmx net>  2011-06-23 19:36:16 ---
The only thing I can see which takes a long time is the following function:
ClassicLoadingCacheFileWatch::slotUpdateDirWatch()

and here especially line 473.

I'll attach the callgrind files...

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #12 from Andi Clemens <andi clemens gmx net>  2011-06-23 19:38:35 ---
Created an attachment (id=61285)
 --> (http://bugs.kde.org/attachment.cgi?id=61285)
callgrind

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=273077


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #13 from Gilles Caulier <caulier gilles gmail com>  2011-06-23 19:49:38 ---
Andi,

I also see a lot of KDirWatch report on the console which slow down digiKAm on
my laptop.

Run kdebugdialog, trun on right kdelibs debug space, and run image editor.
Export image as a new file, especially a huge photo in PNG (large file). Now
look how KDirwatch fire changes in HDD when image is written on disk. For me,
it's abnormal. In this case, KDirwatch must be paused when file is written.

It's possible that DB file bring up KDirWhatch when fingerprint are registered.

What's debug trace give exactly about KDirWatch ?

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #14 from Gilles Caulier <caulier gilles gmail com>  2011-06-23 22:34:51 ---
Andi,

I just tested Fingerprint registration in DB under Windows. It work like a
charm. No crash, no memory leak. Windows memory control center from Win7 said
that memory allocation is stable to 67% (3Gb). 2250 RAW/JPG are managed.
digiKam is compiled with MSVC 2008.

The only thing to improve here is CPU load. Only 58% of CPU is used. It's a
double core computer. This want mean that only one CPU is used. I'm sure that
we can use OpenMP here to speed up computation (MSVC support it very well, as
GCC)

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #15 from Andi Clemens <andi clemens gmx net>  2011-06-24 08:07:54 ---
I just tested it again, it gets clearly slower after 2 minutes and memory usage
went from 180MB to 287MB.
There is something growing in memory, and it seems to stay in this way when I
stop the fingerprints generator and re-activate it again after 5 minutes. So
there is something else (not related to fingerprints) that stays in memory.
Only a digiKam restart fixes the memory and speed issue.

I still think it has something to do with this method:
ClassicLoadingCacheFileWatch::slotUpdateDirWatch()

There is a list that seems to grow all the time...?

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #16 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-07-14 16:54:31 ---
For Haar we load very small images if size 128 and put them in the cache. With
the cache size chosen on my machine, this means more than 4000 entries fit in
the cache - for normal-size images, it is only a few. Apparently KDirWatch, its
backends and/or our code for going through the lists get slow then.

One question that occurred to me is why dont we use the thumbnail loader, which
provides up to 256, for loading images for fingerprint generation?

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #17 from Gilles Caulier <caulier gilles gmail com>  2011-07-14 18:22:47 ---
Marcel,

no idea why thumbnail loader is not user here. I agree that it must do it.

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jaime Torres]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #18 from Jaime Torres <jtamate gmail com>  2011-07-25 18:25:09 ---
Created an attachment (id=62186)
 --> (http://bugs.kde.org/attachment.cgi?id=62186)
callgrind snapshot

I guess the problem is in the data structures.
I've run digikam under callgrind doing an update of the firgerprints, and you
can see where the time is spent in the screenshot: Searching in a probably
growing list.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Andi Clemens]

https://bugs.kde.org/show_bug.cgi?id=273077





--- Comment #19 from Andi Clemens <andi clemens googlemail com>  2011-07-25 21:30:47 ---
Git commit 9e0baf9f6216942c1664337a8d4b9f584f1b4989 by Andi Clemens.
Committed on 25/07/2011 at 23:29.
Pushed by aclemens into branch 'master'.

Possible fix: Use a QSet instead of a QStringList. This container provides a
much faster lookup, at least on my system it works quite fine.

CCBUG:273077

M  +3    -2    libs/threadimageio/loadingcache.h
M  +11   -11   libs/threadimageio/loadingcache.cpp

http://commits.kde.org/digikam/9e0baf9f6216942c1664337a8d4b9f584f1b4989

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel