[digikam] [Bug 330230] New: Memory leak in Image Quality Sorter

[Ryan Henderson]

https://bugs.kde.org/show_bug.cgi?id=330230

            Bug ID: 330230
           Summary: Memory leak in Image Quality Sorter
    Classification: Unclassified
           Product: digikam
           Version: 4.0.0-beta2
          Platform: Compiled Sources
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: NOR
         Component: Maintenance
          Assignee: [hidden email]
          Reporter: [hidden email]

- Large memory leak running maintenance image quality sorter.   (500Mb / 500
images)
- Tags are not applied.

There are several of these in the valigrind log, here's the largest:
==26674== 307,183,616 bytes in 107 blocks are possibly lost in loss record
42,037 of 42,037
==26674==    at 0x4C2AFE7: operator new[](unsigned long) (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26674==    by 0x73963F0: unsigned char*
Digikam::DImgLoader::new_failureTolerant<unsigned char>(unsigned long)
(dimgloader.h:183)
==26674==    by 0x737362B: Digikam::DImg::allocateData() (dimg.cpp:315)
==26674==    by 0x737616F: Digikam::DImg::DImg(Digikam::DImg const&, int, int)
(dimg.cpp:136)
==26674==    by 0x738C1C8: Digikam::DImg::smoothScaleClipped(int, int, int,
int, int, int) const (dimgscale.cpp:233)
==26674==    by 0x738C47B: Digikam::DImg::smoothScaleClipped(QSize const&,
QRect const&) const (dimgscale.cpp:194)
==26674==    by 0x738C514: Digikam::DImg::smoothScale(QSize const&,
Qt::AspectRatioMode) const (dimgscale.cpp:183)
==26674==    by 0x738C55F: Digikam::DImg::smoothScale(int, int,
Qt::AspectRatioMode) const (dimgscale.cpp:188)
==26674==    by 0x74D2D07: Digikam::PreviewLoadingTask::execute()
(previewtask.cpp:433)
==26674==    by 0x74D1783:
Digikam::PreviewLoadThread::loadSynchronously(Digikam::LoadingDescription
const&) (previewloadthread.cpp:114)
==26674==    by 0x50E734: Digikam::ImageQualityTask::run()
(imagequalitytask.cpp:96)

Also:
==26674==    at 0x4C2AFE7: operator new[](unsigned long) (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==26674==    by 0x73809DD: unsigned int*
Digikam::DImgLoader::new_failureTolerant<unsigned int>(unsigned long)
(dimgloader.h:183)
==26674==    by 0x7378128: Digikam::DImg::rotate(Digikam::DImg::ANGLE)
(dimg.cpp:2298)
==26674==    by 0x7378576: Digikam::DImg::rotateAndFlip(int) (dimg.cpp:2649)
==26674==    by 0x74C581D: Digikam::LoadSaveThread::exifRotate(Digikam::DImg&,
QString const&) (loadsavethread.cpp:335)
==26674==    by 0x74D2D47: Digikam::PreviewLoadingTask::execute()
(previewtask.cpp:439)
==26674==    by 0x74D1783:
Digikam::PreviewLoadThread::loadSynchronously(Digikam::LoadingDescription
const&) (previewloadthread.cpp:114)
==26674==    by 0x50E734: Digikam::ImageQualityTask::run()
(imagequalitytask.cpp:96)




Reproducible: Always

Steps to Reproduce:
1. Run image quality sorter with library > 500.
2. Mark maintenance option to clear db and re-run for all

Actual Results:  
2. 500Mb memory leak
3. Tags not applied

Expected Results:  
Digikam would validate my photography skills.  or at least not leak memory and
apply the good, bad, ok tags.

4.0.0-beta-3 built from git
gitup current as of Jan 20th 2014.
Ubuntu 13.10 (gcc 2.8.1)
libraw from git 0.16.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=330230

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Gilles Caulier <[hidden email]> ---
*** Bug 330227 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=330230

--- Comment #2 from Gilles Caulier <[hidden email]> ---
Which Image Quality Sorter settings do you use exactly ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=330230

--- Comment #3 from Gilles Caulier <[hidden email]> ---
Gowthan,

This evening from IRC channel, a thread about this entry has give more
important information:

[23:10] <rhenders> I'm seeing some pretty big memory leaks in Image Quality
sorter in 4.0.0-beta2:  https://gist.github.com/bitflippersanonymous/8530225
[23:11] <rhenders> Built from source on Ubuntu/AMD64
[23:11] <rhenders> I looked for existing bugs but couldn't find any, thought
I'd check here before filing
[23:12] <teprrr> uhu, that looks leaky indeed :o
[23:13] <teprrr> problem is here:            d->imgqsort = new ImgQSort(dimg,
d->quality, &pick);
[23:13] <rhenders> It grows to 100GB for my collection of 5k pictures.
[23:13] <rhenders> I could build debug and get line numbers, ...    what file
is that in?
[23:13] <teprrr> most likely. run() is being called multiple times, and that
new gets called all the time then
[23:14] <teprrr> digikam/utilities/maintenance/imagequalitytask.cpp
[23:14] <teprrr> line 110
[23:15] <teprrr> if that isn't destroyed somewhere in the result handler that's
most likely the leaker
[23:16] <teprrr> DImg there is the image data itself, which may be heavy :P
[23:16] <rhenders> There are several leaks in that valgrind log.  The last one
is the largest, but they all look similar
[23:16] <teprrr> but if you can, provide a valgrind log with the debug-info
enabled and report it to the bugs.kde.org :p
[23:16] <teprrr> hmm, related to maintenance tool?
[23:17] <rhenders> I also see leaks when rebuilding thumbnails, but this was
the largest.  I'll rebuild in debug and report it
[23:17] <rhenders> (What's the cmake flag to enable debug)
[23:18] <teprrr> -DCMAKE_BUILD_TYPE=Debug
[23:18] <teprrr> or DebugFull
[23:18] <rhenders> Great, thanks
[23:20] <teprrr> not a problem, I gotta go now away for a while. thanks for
noticing such a bug :)

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ryan Henderson]

https://bugs.kde.org/show_bug.cgi?id=330230

--- Comment #4 from Ryan Henderson <[hidden email]> ---
Just the defaults AFAIK

Enabled
Speed: 1
Detect Everything
Assign Everything
Reject threshold: 10
Pending threshold: 40
Accepted threshold: 60
Blur Weight: 100
Noise Weight: 100
Compress Weight: 100

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gowtham Ashok]

https://bugs.kde.org/show_bug.cgi?id=330230

--- Comment #5 from Gowtham Ashok <[hidden email]> ---
Thank you teppr and Ryan. You are right. Image quality sorter data was not
getting deleted after completion of task. Fixed it.

Please note that Image quality sorter is still under development. The goal is
to tag really bad pictures, not to assess the quality of photographs
subjectively.
As of now, we have to manually set the threshold values according to the type
of pictures processed.
Presently, we're trying to detect different cases of blurring in an image.
We're also trying to detect the type of pictures so that the threshold values
are automatically set.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=330230

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
   Version Fixed In|                            |4.0.0
         Resolution|---                         |FIXED
      Latest Commit|                            |http://commits.kde.org/digi
                   |                            |kam/df13317e5d9310f7d07943d
                   |                            |f8b1501d4706f9e17

--- Comment #6 from Gilles Caulier <[hidden email]> ---
Git commit df13317e5d9310f7d07943df8b1501d4706f9e17 by Gowtham Ashok.
Committed on 22/01/2014 at 01:46.
Pushed by gowthamashok into branch 'master'.

Fixed imgqsort huge memory leak

M  +6    -2    utilities/maintenance/imagequalitytask.cpp

http://commits.kde.org/digikam/df13317e5d9310f7d07943df8b1501d4706f9e17

diff --git a/utilities/maintenance/imagequalitytask.cpp
b/utilities/maintenance/imagequalitytask.cpp
index 1df7a23..eafa8d8 100644
--- a/utilities/maintenance/imagequalitytask.cpp
+++ b/utilities/maintenance/imagequalitytask.cpp
@@ -78,8 +78,11 @@ void ImageQualityTask::slotCancel()
 {
     d->cancel = true;

-    if (d->imgqsort)
+   //TODO: Investigate if commented code is necessary
+   /* if (d->imgqsort)
         d->imgqsort->cancelFilter();
+
+   */
 }

 void ImageQualityTask::run()
@@ -111,8 +114,9 @@ void ImageQualityTask::run()

             ImageInfo info(d->path);
             info.setPickLabel(pick);
+            if(d->imgqsort)
+                    delete d->imgqsort;                 //delete image data
after setting label
         }
-
         // Dispatch progress to Progress Manager
         QImage qimg = dimg.smoothScale(22, 22,
Qt::KeepAspectRatio).copyQImage();
         emit signalFinished(qimg);

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=330230

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|4.0.0-beta2                 |4.0.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel