[digikam] [Bug 323888] New: Face recognition makes digikam fill all the available memory

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #59 from Gilles Caulier <[hidden email]> ---
The other part where memory issue can be relevant of this file is OpenCV shared
library. Please check which version is installed on your system....

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ritesh Raj Sarraf-3]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #60 from Ritesh Raj Sarraf <[hidden email]> ---
(In reply to comment #59)
> The other part where memory issue can be relevant of this file is OpenCV
> shared library. Please check which version is installed on your system....
>
> Gilles Caulier

The version I have is: 2.4.8+dfsg1-2

Does it have known issues ?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[eldk]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #61 from [hidden email] ---
Hello,

I don't know if openCV have an issue, but I was having a problem with it too.

Should you look here to check some openCV compilation feature and your hardware
(for me disable SSE3 feature - Intel pentium M):

https://bugs.kde.org/show_bug.cgi?id=325309

Greatings,

Eric

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Scott]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #62 from Scott <[hidden email]> ---
> But... The problem remains the same. Digikam still leaks memory. Can you
> please double check if it has sorted the problem for you ?

I can confirm that for me digikam uses alot of memory, but no longer leaks
memory.  I have just tagged 200+ of my 31,000+ outstanding face tags (I have
ALOT of portrait photography dating back to 2003) and the memory usage is
consistantly between 780-850MB, with an average of 800MB.

Btw devs, should I post a link the the solution (for me) on the digikam
website, as a comment to the beta4 release?  Or should I let this flow
downstream first?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ritesh Raj Sarraf-3]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #63 from Ritesh Raj Sarraf <[hidden email]> ---
17:00:58 rrs@zan:~$ dstat -am
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
------memory-usage-----
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw | used
buff  cach  free
  2   1  97   0   0   0|2179k  103k|   0     0 |   0     0 | 559  2702 |2513M
131M 1038M 3895M
 11   2  87   0   0   0|  54M    0 |   0     0 |   0     0 |1111  1800 |2895M
131M 1091M 3459M
 11   1  87   0   0   0|  53M   88k|   0     0 |   0     0 |1057  1501 |3272M
131M 1145M 3029M
 11   2  87   0   0   0|  54M 1240k|   0     0 |   0     0 |1171  1436 |3659M
131M 1198M 2589M
 11   2  87   0   0   0|  46M    0 | 428B    0 |   0     0 | 906  1316 |3988M
131M 1244M 2215M
 12   1  87   0   0   0|  46M    0 |   0     0 |   0     0 | 948  1731 |4321M
131M 1290M 1835M
 11   1  87   0   0   0|  38M    0 |   0    65B|   0     0 | 885  1872 |4593M
131M 1328M 1525M
 12   1  87   0   0   0|  44M    0 |   0     0 |   0     0 | 929  1745 |4908M
131M 1372M 1166M
 11   2  87   0   0   0|  46M    0 |   0     0 |   0     0 | 932  1386 |5236M
131M 1418M  792M
 12   1  87   0   0   0|  47M   36k|   0     0 |   0     0 | 965  1349 |5575M
131M 1465M  406M
 11   3  86   1   0   0|  49M    0 |   0     0 |   0     0 |1682  1666 |5927M
89.3M 1397M  163M
 11   3  86   0   0   0|  48M   88k|  42B   60B|   0    88k|  12k 2628 |6266M
38.9M 1128M  144M
  5   9  76   9   0   0|  11M   35M|   0     0 |   0    80M|  25k 3218 |7319M
328k  157M  100M
  1   4  69  26   0   0|3872k   79M|   0     0 |   0    98M|  16k  938 |7332M
328k  142M  102M
  1   3  73  22   0   0|3196k   86M| 428B    0 |  48k   83M|  12k  967 |7330M
328k  145M  102M
  1   3  72  25   0   0| 976k   77M|   0    65B| 160k   84M|  13k  885 |7328M
328k  144M  104M
  1   2  64  33   0   0|9024k   36M|   0     0 | 816k   25M|8423  1742 |7328M
328k  147M  101M
  1   1  60  39   0   0|4856k 9916k|   0     0 |1080k   16M| 622  1476 |7327M
328k  148M  101M
  0   3  45  52   0   0|1024k   63M|   0     0 | 188k   63M|  12k  920 |7322M
328k  146M  108M
  1   2  74  23   0   0|1120k   56M|   0     0 | 152k   52M|7944   809 |7329M
328k  148M  100M
  0   3  63  33   0   0|1256k   74M|   0     0 |  88k   79M|  12k  881 |7330M
328k  146M  100M
  0   2  56  42   0   0| 404k   46M|   0     0 | 272k   57M|5198   995 |7324M
352k  148M  104M
  0   1  55  44   0   0| 912k   34M|   0     0 | 908k   21M|4964  1115 |7326M
352k  148M  103M
  0   1  50  48   0   0|2136k   13M|   0     0 | 680k   15M|5817   980 |7328M
352k  148M  101M
  0   0  62  37   0   0|5072k 2048k|   0     0 |1212k    0 | 605  1207 |7327M
352k  149M  101M
  0   1  64  35   0   0| 820k   30M|   0    65B| 808k   32M|4568  1209 |7320M
348k  148M  109M
  1   1  78  20   0   0|1792k   19M|   0     0 | 688k   20M| 941  1638 |7326M
352k  148M  102M
  0   2  54  44   0   0|1096k   32M|   0     0 | 496k   38M|4716  1035 |7324M
348k  149M  104M
  0   1  64  34   0   0|5708k   14M|   0     0 |1940k   10M|2101  1601 |7331M
492k  145M  100M
  0   1  47  51   0   0| 760k   37M|   0     0 | 608k   32M|1192   868 |7330M
492k  146M  101M
  1   1  61  37   0   0|1416k   13M|   0     0 |1408k   20M|1587  1108 |7324M
492k  145M  107M
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
------memory-usage-----
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw | used
buff  cach  free
  0   0  53  46   0   0| 900k 2560k|   0     0 | 896k    0 | 865   999 |7325M
492k  145M  106M
  1   1  58  41   0   0|1620k 8212k|   0     0 |1152k 6148k|3741  1193 |7330M
492k  146M  101M
  0   2  67  32   0   0| 468k   58M|   0     0 | 296k   63M|1338   718 |7323M
492k  146M  108M
  0   1  63  36   0   0| 520k   24M|   0    65B| 524k   23M|1973   780 |7324M
500k  146M  107M
  0   1  63  36   0   0|1756k   25M|   0     0 |1372k   24M|7324   987 |7324M
516k  146M  106M
  1   0  66  34   0   0|3056k    0 |   0     0 |1200k    0 |1026  1494 |7327M
516k  148M  102M
  0   0  64  35   0   0|2104k    0 |   0     0 |1356k    0 | 643  1255 |7327M
516k  149M  101M
  0   1  71  28   0   0|2464k   21M|   0     0 | 872k   15M|4744  1087 |7326M
516k  150M  101M
  0   2  59  39   0   0|2260k   39M|   0     0 | 548k   36M|  10k  960 |7325M
504k  151M  101M
  0   1  64  35   0   0|2796k 3584k|   0     0 |1500k   12M| 499  1133 |7320M
504k  151M  105M
  1   0  63  36   0   0|3804k    0 |   0     0 |1340k    0 | 843  1453 |7322M
644k  153M  101M
  0   7  53  40   0   0|3520k    0 |4363B    0 |1392k    0 |1212  1148 |7322M
644k  154M  101M
  0   0  77  23   0   0|2148k 4132k|   0     0 |1528k    0 | 835   928 |7322M
548k  153M  101M
  0   0  75  25   0   0|2940k    0 |   0    65B|1520k    0 | 528   964 |7323M
496k  153M  100M
  0   0  69  31   0   0|3972k 4096B|   0     0 |1252k    0 | 565  1198 |7323M
496k  153M  101M
  0   0  60  39   0   0|3296k 2048k|   0     0 |1060k    0 | 602  1403 |7324M
488k  147M  105M
  0   0  74  25   0   0|3960k    0 |   0     0 |1200k    0 |1084  1312 |7325M
488k  150M  101M
  0   0  75  25   0   0|1644k 2060k|   0     0 | 952k    0 | 942  1055 |7325M
496k  150M  101M
  0   3  75  23   0   0|3216k 2048k|  42B   60B|1096k  104k|1210  1284 |5440M
376k  142M 1995M
  0   3  48  49   0   0|5372k    0 |   0     0 |1380k    0 |1311  1561 | 514M
644k  140M 6922M
  0   1  78  21   0   0|4812k    0 |   0     0 |1908k    0 |1319  2009 | 516M
652k  143M 6917M
  0   0  81  19   0   0|4372k    0 |   0     0 |1500k    0 |1163  1518 | 518M
652k  146M 6913M
  0   0  75  25   0   0|6104k  516k|   0    65B|1748k    0 |1324  1766 | 520M
928k  150M 6906M
  0   0  74  26   0   0|5268k    0 |   0   212B|1796k    0 |1161  1576 | 522M
928k  153M 6901M
  0   0  82  17   0   0|4004k    0 |   0     0 |1416k    0 |1069  1417 | 523M
928k  156M 6897M
  0   1  87  12   0   0|5808k   32k|   0     0 |1956k    0 |1211  1875 | 525M
936k  159M 6892M
  0   0  88  12   0   0|2220k   50M| 428B    0 | 384k    0 | 821   837 | 520M
936k  161M 6895M
  1   0  88  11   0   0|4304k 6252k|   0     0 |1056k    0 | 517  1615 | 519M
936k  164M 6892M
  1   0  88  10   0   0|3052k    0 |   0     0 |1108k    0 | 599  1660 | 520M
936k  166M 6890M^C



@Giles,

Do you have any idea why Digikam is reading disk so bad ? Right when the leak
is triggered, you can see, it reads 500+ MiB of data and then similary writes
it down again.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mick Sulley]

https://bugs.kde.org/show_bug.cgi?id=323888

Mick Sulley <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #64 from Mick Sulley <[hidden email]> ---
I tried the fix in comments 57 & 58 and the problem still exists for me.
I am running DK Version 3.5.0 Using KDE Development Platform 4.13.0
on Linux Mint 17 64 bit
The problem occurs when manually adding a single face tag.
Please let me know if I can help by providing logs or other info

Thanks
Mick

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #65 from Gilles Caulier <[hidden email]> ---
Hi,

We need a fresh feedback using 4.0.0 just released.

Also, to have a log using valgrind about memory leak can help. Use this command
in CLI, and report console trace to investiguate :

"valgrind --tool=memcheck --leak-check=full --error-limit=no digikam"

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ritesh Raj Sarraf-3]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #66 from Ritesh Raj Sarraf <[hidden email]> ---
Created attachment 86795
  --> https://bugs.kde.org/attachment.cgi?id=86795&action=edit
digikam 4.0 still leaking memory

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ritesh Raj Sarraf-3]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #67 from Ritesh Raj Sarraf <[hidden email]> ---
(In reply to comment #65)

> Hi,
>
> We need a fresh feedback using 4.0.0 just released.
>
> Also, to have a log using valgrind about memory leak can help. Use this
> command in CLI, and report console trace to investiguate :
>
> "valgrind --tool=memcheck --leak-check=full --error-limit=no digikam"
>
> Gilles Caulier

Comment #66 has what you asked. This now is the pristine copy of Digikam as
uploaded in Debian, i.e. the libPGF in use is external.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323888

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.5.0                       |4.0.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Scott]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #68 from Scott <[hidden email]> ---
Sorry for the long delay,

I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10 to
14.04 and from 4.0.0beta to stable.  But for me, after deleting recognition.db
it fixed the issue again, so I am starting to think there is more than meets
the eye.

The problem with libPGF is gone, but clearly one with libkface remains...

Scott.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[kde-bugs]

https://bugs.kde.org/show_bug.cgi?id=323888

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #69 from [hidden email] ---
I can confirm that nothing has changed for me with 4.0.0 - running face
detection even on one single picture will make digikam fill 70GB memory and
more (killed it after that). I'm using MariaDB as backend.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #70 from Gilles Caulier <[hidden email]> ---
What's about this memory leak, if SQlite DB is used instead ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[kde-bugs]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #71 from [hidden email] ---
(In reply to comment #70)
> What's about this memory leak, if SQlite DB is used instead ?

If I read it correctly, the valgrind log from comment #66 uses SQLite, so that
doesn't seem to make a difference.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[alexander.s.m]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #72 from Alexander Meshcheryakov <[hidden email]> ---
It seems that everyone ignored my comments where I have investigated the issue
a bit

https://bugs.kde.org/show_bug.cgi?id=323888#c5
https://bugs.kde.org/show_bug.cgi?id=323888#c6

In a nutshell my findings show that in some cases recognition.db becomes two
times larger, so if this is repeated several times recognition.db becomes
unbearable for digikam.

I have described steps to reproduce, please try to reproduce according to my
guide. It takes less than 3 minutes.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Ritesh Raj Sarraf-3]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #73 from Ritesh Raj Sarraf <[hidden email]> ---
(In reply to comment #68)
> Sorry for the long delay,
>
> I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10
> to 14.04 and from 4.0.0beta to stable.  But for me, after deleting
> recognition.db it fixed the issue again, so I am starting to think there is
> more than meets the eye.
>
> The problem with libPGF is gone, but clearly one with libkface remains...


Thank you for sharing this informaion Scott. Based on your feedback I went and
did the same. After removing the recognition.db file, it is working a charm.
But I still don't know if we have covered all the cases where the bug was
reproducible.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[kde-bugs]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #74 from [hidden email] ---
I'll check my MariaDB instance for the face recognition DB size. However, will
deleting the DB mean losing the face tags?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[alexander.s.m]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #75 from Alexander Meshcheryakov <[hidden email]> ---
(In reply to comment #74)
> I'll check my MariaDB instance for the face recognition DB size. However,
> will deleting the DB mean losing the face tags?

recognition.db contains just "fingerprints" for auto face detection. It can be
re-created according to your face tags after deletion.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Scott]

https://bugs.kde.org/show_bug.cgi?id=323888

--- Comment #76 from Scott <[hidden email]> ---
(In reply to comment #73)

> (In reply to comment #68)
> > Sorry for the long delay,
> >
> > I can confirm that his bug had regressed when I upgraded Ubuntu from 13.10
> > to 14.04 and from 4.0.0beta to stable.  But for me, after deleting
> > recognition.db it fixed the issue again, so I am starting to think there is
> > more than meets the eye.
> >
> > The problem with libPGF is gone, but clearly one with libkface remains...
>
>
> Thank you for sharing this informaion Scott. Based on your feedback I went
> and did the same. After removing the recognition.db file, it is working a
> charm. But I still don't know if we have covered all the cases where the bug
> was reproducible.

I totally agree.  While in my specific case recompiling libPGF, putting it in
the right places and removing recognition.db when the bug regressed works, it's
clear that it did not for everyone.

Either steps were not followed, or there is something more.  But either way, it
would be nice to know just what is corrupting recognition.db in the first
place...

On trying to repeat the corruption (I think it may be related to the detection
and recognition option), i am now getting a crash when detecting, but tagging
is still working.

As for deleting face tags, as said before, that recognition.db is for
"fingerprints" only.  The tags and the face boxes themselves seem to be kept in
digikam4.db, if not in the metadata itself (that is of course if the write
metadata to images option is on!).  For my piece of mind, and portability, I
personally leave the metadata writing on...

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323888

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|major                       |crash

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel