[digikam] [Bug 326525] New: Severe memory leak during maintainance scan for rebuild thumbnails

[Ananta Palani]

https://bugs.kde.org/show_bug.cgi?id=326525

Ananta Palani <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #37 from Ananta Palani <[hidden email]> ---
(In reply to comment #28)
> Re PGF:
> Even if we switch thumbnail creation to another algorithm, we
> a) need to continue providing PGF read support for those gigabytes of
> thumbnails on our users machines
> b) we break backwards compatibility to use a new thumbnail db with an older
> version, so it should be done only with a major release

I agree with Marcel. Also, the WebP is inferior in some ways to PGF (vastly
reduced color depth support), but superior in others (animation, color
management). Might be nice to use WebP in certain cases, like to create
animated thumbnails for animated gifs or videos, but I suppose most users won't
want their thumbnail list to be animated. Perhaps it could be an option if the
user would like it.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #38 from Gilles Caulier <[hidden email]> ---
Hi Ananta,

I agree with you that PGF is superior than WebP.

But the topic here is not to use WebP to store photo, but only thumbnails in
database, compressed with Wavelets algorithm. In this case WebP can be
suitable. We don't need 16 bits color depth.

As you can see, i recieve PGF maintainer feedback and it's good for the future.
If PGF memory corruption/leak problems are fixed there is no reason to switch
to WebP.

There is also this problem with OpenMP support in PGF which don't work. I just
re-tested here and crash still here. I wait to see if PGF memory fixes improve
OPenMP support before look in-deep in this problem.

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #39 from Gilles Caulier <[hidden email]> ---
Ananta,

Just to talk more about wavelets image format, there is another challenger
named JPEG-XR done by ... Microsoft, and valided as a standard bu JPEG group to
include it in digital camera to replace usual JPEG !

This format is now available to developers though an "open-source like" library
named "jxlib"

https://jxrlib.codeplex.com/

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #40 from Gilles Caulier <[hidden email]> ---
Raphael,

Do you see my comment #33 ?

I can add you to digiKam Coverity Scan project. Like this you can analyse
in-deep all Coverity entries about libPGF. Reports are really detailled and
instructive...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Raphael Schweizer]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #41 from Raphael Schweizer <[hidden email]> ---
(In reply to comment #40)

Yes, I applied for access using a different mail address (minus' not allowed at
coverity).

Raphael

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #42 from Gilles Caulier <[hidden email]> ---
Raphael,

I send you invitation with your new email...

As i list in comment #16, Coverity Scan entries ID relevant to libPGF are :

- 981290
- 981172
- 981291
- 981292
- 981188
- 981293
- 981497
- 981294

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #43 from Gilles Caulier <[hidden email]> ---
*** Bug 327033 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #44 from Gilles Caulier <[hidden email]> ---
Raphael,

Any progress about libpgf memory leak fixes ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[eldk]

Hello,

I know the question is not for me.

But what I can say is that use of "internal libpgf" reduce most of the
memory leak. Before the use of memory was growing fast to 1 gb and go
away to crash. Now, for scanning the all collection (same condition and
same pictures) it's almost use 400 mo for digikam to the end of
thumbnail scan.

Is libpgf has something to do with face scan and/or recognition :

- thumbnails for faces storage in digikam thumb db?

- and other stuff in recognition,db ?

Greatings,

Eric




Le mercredi 06 novembre 2013 à 16:20 +0000, Gilles Caulier a écrit :
> https://bugs.kde.org/show_bug.cgi?id=326525
>
> --- Comment #44 from Gilles Caulier <[hidden email]> ---
> Raphael,
>
> Any progress about libpgf memory leak fixes ?
>
> Gilles Caulier
>


_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

Eric,

Please post your comment through bugzilla file, not directly on mailing-list

Gilles Caulier

2013/11/6 eric <[hidden email]>:

> Hello,
>
> I know the question is not for me.
>
> But what I can say is that use of "internal libpgf" reduce most of the
> memory leak. Before the use of memory was growing fast to 1 gb and go
> away to crash. Now, for scanning the all collection (same condition and
> same pictures) it's almost use 400 mo for digikam to the end of
> thumbnail scan.
>
> Is libpgf has something to do with face scan and/or recognition :
>
> - thumbnails for faces storage in digikam thumb db?
>
> - and other stuff in recognition,db ?
>
> Greatings,
>
> Eric
>
>
>
>
> Le mercredi 06 novembre 2013 à 16:20 +0000, Gilles Caulier a écrit :
>> https://bugs.kde.org/show_bug.cgi?id=326525
>>
>> --- Comment #44 from Gilles Caulier <[hidden email]> ---
>> Raphael,
>>
>> Any progress about libpgf memory leak fixes ?
>>
>> Gilles Caulier
>>
>
>
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Raphael Schweizer]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #45 from Raphael Schweizer <[hidden email]> ---
Gilles,

at last:
- regarding memory leak @CPGFImage::WriteHeader(CPGFStream*)
(PGFimage.cpp:924): we believe we only leaked memory if we eventually ran out
of it (fixed in latest sourceforfe repo version) (though I'm not sure how OMP
is related here)
- uninitialized values @CEncoder::WriteValue(CSubband*, int) (Encoder.cpp:327)
are false positives
- uninitialized values @CSubband::Quantize(int) (Subband.cpp:134) are also
believed to be false positives

We've just set up a new linux box for more testing (including your test from
comment #35)

- Raphael

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[eldk]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #46 from [hidden email] ---
Hello,

I know the question is not for me.

But what I can say is that use of "internal libpgf" reduce most of the
memory leak. Before the use of memory was growing fast to 1 gb and go
away to crash. Now, for scanning the all collection (same condition and
same pictures) it's almost use 400 mo for digikam to the end of
thumbnail scan.

Is libpgf has something to do with face scan and/or recognition :

- thumbnails for faces storage in digikam thumb db?

- and other stuff in recognition,db ?

Greatings,

Eric

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #47 from Gilles Caulier <[hidden email]> ---
Raphael,

I will backport last libpgf code into digiKam core and look valgrind report to
see if any improvements have been done about memory leak.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #48 from Gilles Caulier <[hidden email]> ---
Eric,

libpgf has nothing releated to face recognition memory leak. All problem with
face management are already reported in another report and problem is relevant
of OpenCV bugs.

https://bugs.kde.org/show_bug.cgi?id=323888

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Raphael Schweizer]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #49 from Raphael Schweizer <[hidden email]> ---
Gilles,

Does digiKam "internal libpgf" have other improvements (comment #5) besides
disabling OpenMP regarding better "memory performance"? I'd then patch our
sources before you do the work of backporting.
Also, I'd like to fix some "uninitialized values in constructor" issued
reported by coverity (though I don't think they actually have any effect).

- Raphael

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #50 from Gilles Caulier <[hidden email]> ---
Raphael,

No, as i remember, i don't patch internal libpgf code. I just only include a
cmake switch to enable/disable OpenMP support. That all...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[eldk]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #51 from [hidden email] ---
Created attachment 83399
  --> https://bugs.kde.org/attachment.cgi?id=83399&action=edit
Build traces for digikam with external libpgf

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[eldk]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #52 from [hidden email] ---
Created attachment 83400
  --> https://bugs.kde.org/attachment.cgi?id=83400&action=edit
Build traces for digikam with inrenal libpgf

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #53 from Gilles Caulier <[hidden email]> ---
Git commit 8c41f33ff8489e863130003a6f2bfea1bb6393e5 by Gilles Caulier.
Committed on 07/11/2013 at 14:02.
Pushed by cgilles into branch 'master'.

update internal libpgf source code to last revision #r121 from SF repository

M  +1    -1    libs/3rdparty/libpgf/Decoder.cpp
M  +1    -1    libs/3rdparty/libpgf/Decoder.h
M  +1    -1    libs/3rdparty/libpgf/Encoder.h
M  +26   -11   libs/3rdparty/libpgf/PGFimage.cpp
M  +4    -4    libs/3rdparty/libpgf/PGFplatform.h
M  +4    -4    libs/3rdparty/libpgf/PGFstream.h
M  +4    -4    libs/3rdparty/libpgf/PGFtypes.h
M  +2    -2    libs/3rdparty/libpgf/Subband.cpp
M  +56   -5    libs/3rdparty/libpgf/WaveletTransform.cpp

http://commits.kde.org/digikam/8c41f33ff8489e863130003a6f2bfea1bb6393e5

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=326525

--- Comment #54 from Gilles Caulier <[hidden email]> ---
Raphael,

As you can see in my last comment, i commited last code from libpgf repository
and recompiled digiKam. Running it through valgrind report again some issue
about memory problems. Below, it's not a memory leak :

=2438== Thread 21:
==2438== Conditional jump or move depends on uninitialised value(s)
==2438==    at 0x80BAA10: CEncoder::CMacroBlock::NumberOfBitplanes()
(Encoder.cpp:740)
==2438==    by 0x80BA233: CEncoder::CMacroBlock::BitplaneEncode()
(Encoder.cpp:495)
==2438==    by 0x80B9E7A: CEncoder::EncodeBuffer(ROIBlockHeader)
(Encoder.cpp:347)
==2438==    by 0x80B9D2B: CEncoder::Flush() (Encoder.cpp:313)
==2438==    by 0x80BE2CD: CPGFImage::WriteImage(CPGFStream*, bool (*)(double,
bool, void*), void*) (PGFimage.cpp:1128)
==2438==    by 0x80BE381: CPGFImage::Write(CPGFStream*, unsigned int*, bool
(*)(double, bool, void*), void*) (PGFimage.cpp:1163)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage
const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&,
QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279:
Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&,
Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect
const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const
(thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:172)
==2438==
==2438== Conditional jump or move depends on uninitialised value(s)
==2438==    at 0x80BAA41: CEncoder::CMacroBlock::NumberOfBitplanes()
(Encoder.cpp:741)
==2438==    by 0x80BA233: CEncoder::CMacroBlock::BitplaneEncode()
(Encoder.cpp:495)
==2438==    by 0x80B9E7A: CEncoder::EncodeBuffer(ROIBlockHeader)
(Encoder.cpp:347)
==2438==    by 0x80B9D2B: CEncoder::Flush() (Encoder.cpp:313)
==2438==    by 0x80BE2CD: CPGFImage::WriteImage(CPGFStream*, bool (*)(double,
bool, void*), void*) (PGFimage.cpp:1128)
==2438==    by 0x80BE381: CPGFImage::Write(CPGFStream*, unsigned int*, bool
(*)(double, bool, void*), void*) (PGFimage.cpp:1163)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage
const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&,
QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279:
Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&,
Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect
const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const
(thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:172)
==2438==

At end of application, valgrind report memory leak. It's not too huge, because
i stopped thumbnail maintenance task 30 secondes after to start (1%
completed)...

==2438== 9,200 bytes in 25 blocks are possibly lost in loss record 25,377 of
25,727
==2438==    at 0x4C26DFF: calloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==2438==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==2438==    by 0xFD028B8: pthread_create@@GLIBC_2.2.5 (in
/usr/lib64/libpthread-2.17.so)
==2438==    by 0x10AACCBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==2438==    by 0x80BD89F: CPGFImage::WriteHeader(CPGFStream*)
(PGFimage.cpp:924)
==2438==    by 0x80BE366: CPGFImage::Write(CPGFStream*, unsigned int*, bool
(*)(double, bool, void*), void*) (PGFimage.cpp:1160)
==2438==    by 0x80CA106: Digikam::PGFUtils::writePGFImageDataToStream(QImage
const&, CPGFStream&, int, unsigned int&, bool) (pgfutils.cpp:288)
==2438==    by 0x80C9562: Digikam::PGFUtils::writePGFImageData(QImage const&,
QByteArray&, int, bool) (pgfutils.cpp:184)
==2438==    by 0x80AA279:
Digikam::ThumbnailCreator::storeInDatabase(Digikam::ThumbnailInfo const&,
Digikam::ThumbnailImage const&) const (thumbnailcreator.cpp:695)
==2438==    by 0x80A7BF8: Digikam::ThumbnailCreator::load(QString const&, QRect
const&, bool) const (thumbnailcreator.cpp:267)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const
(thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:172)

==2438== 2,208 bytes in 6 blocks are possibly lost in loss record 24,976 of
25,727
==2438==    at 0x4C26DFF: calloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==2438==    by 0x4011008: _dl_allocate_tls (in /usr/lib64/ld-2.17.so)
==2438==    by 0xFD028B8: pthread_create@@GLIBC_2.2.5 (in
/usr/lib64/libpthread-2.17.so)
==2438==    by 0x10AACCBD: ??? (in /usr/lib64/libgomp.so.1.0.0)
==2438==    by 0x80BC372: CPGFImage::Read(int, bool (*)(double, bool, void*),
void*) (PGFimage.cpp:428)
==2438==    by 0x80C8C51: Digikam::PGFUtils::readPGFImageData(QByteArray
const&, QImage&, bool) (pgfutils.cpp:93)
==2438==    by 0x80AAE7E:
Digikam::ThumbnailCreator::loadFromDatabase(Digikam::ThumbnailInfo const&)
const (thumbnailcreator.cpp:886)
==2438==    by 0x80A7ACC: Digikam::ThumbnailCreator::load(QString const&, QRect
const&, bool) const (thumbnailcreator.cpp:248)
==2438==    by 0x80A77A5: Digikam::ThumbnailCreator::load(QString const&) const
(thumbnailcreator.cpp:199)
==2438==    by 0x80B6685: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:172)
==2438==    by 0x8091550: Digikam::LoadSaveThread::run()
(loadsavethread.cpp:136)
==2438==    by 0x80CF451: Digikam::DynamicThread::DynamicThreadPriv::run()
(dynamicthread.cpp:186)

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel