[digikam] [Bug 375035] New: Out of memory during face tagging

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

            Bug ID: 375035
           Summary: Out of memory during face tagging
           Product: digikam
           Version: 5.4.0
          Platform: Archlinux Packages
                OS: Linux
            Status: UNCONFIRMED
          Severity: crash
          Priority: NOR
         Component: Faces-Engine
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

digikam crashes (killed by kernel) during tagging of detected faces due to
out-of-memory.

Last entries (lots of them) of digikam in the system journal:
   digikam.facesengine: Clone Array size [ 1 ,  16384 ] of type  5

recognition.db is about 214MB
around 29700 tagged faces

It used to be better in 5.3 (admittedly the number of tagged faces were
smaller, say ~27000)

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from [hidden email] ---
Please try the Linux Universal AppImage bundle instead system based package.
There is nothing to install. Run it as well.

http://download.kde.org/stable/digikam/

Face tagging use OpenCV in background. I suspect that system based OpenCV is
badly packaged (a puzzle). The AppImage include OpenCV with the minimum
component enabled.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #2 from [hidden email] ---
Tried the AppImage, unfortunately the same effect.

It seems OK if only a few (3, 4) persons are tagged. As soon as there were too
many (different) persons tagged in the current session, the system runs out of
memory.

I checked the system log, it contains a lot of following entries (with slightly
different numbers):
digikam.facedb: Checkout compressed histogram  10039  for identity  4  with
size  9377

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #3 from [hidden email] ---
Downgraded to 5.3, to no avail.

I had a closer look into the system log: it seems that for every new face,
Digikam runs this "digikam.facedb: Checkout compressed histogram".

Say, I'm currently tagging 'Alice'. I see the 'Checkout compressed histogram'
only for the first time I tag 'Alice'. Tagging following 'Alice' faces don't
trigger a checkout. Now, when I want to tag 'Bob', the 'Checkout compressed
histogram' runs again, eats up all my memory, and Digikam gets killed.

So, work-around for me: tag all faces of one person, restart Digikam, tag
another person, and so on. Not a very nice workflow :-/

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

Maik Qualmann <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #4 from Maik Qualmann <[hidden email]> ---
Is your recognition.db still created with digiKam-4.xx? Please rename the
recognition.db and try again. Actually this problem was fixed in the last
versions of digiKam-4.xx. But it was necessary to create a new DB.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #5 from [hidden email] ---
Yes, the recognition.db was created newly by Digikam 5.x (next to digikam4.db
and the thumbnails)

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #6 from [hidden email] ---
After activating swap digikam doesn't crash anymore -- it eats up most of the
RAM, though.
I guess it doesn't allocate any more memory because OpenCV is miffed and throws
exceptions for every face-tagged picture:
    digikam.facesengine: Default exception from OpenCV

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

Mario Frank <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #7 from Mario Frank <[hidden email]> ---
(In reply to lesf from comment #6)
> After activating swap digikam doesn't crash anymore -- it eats up most of
> the RAM, though.
> I guess it doesn't allocate any more memory because OpenCV is miffed and
> throws exceptions for every face-tagged picture:
>     digikam.facesengine: Default exception from OpenCV

Can you test with the current AppImage again? I have done some changes in 5.5.0
which reduce the count of created threads. This may also have lead to out of
memory problems. It is possible that your problem does not occur any more.

The most current AppImages are always stored here by Gilles:
https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #8 from [hidden email] ---
Unfortunately, this appimage doesn't work either.
Still the same effect as I wrote in comment #2: tagging a few persons is OK,
but after that, the system runs out of memory.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #9 from [hidden email] ---
Do you use Sqlite or Mysql as database ?

Can you reproduce the problem using a fresh account and a fresh database (data
can be corrupted with previous crashes) ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #10 from [hidden email] ---
I use Sqlite database.

I can reproduce the problem.
Steps:
- Created a new user account
- Prepared a data set of ~58000 pictures. Stripped all metadata [exiftool -r
-overwrite_original -P -all= DIR], as they contained face tags and regions
- Started digikam (digikam-5.5.0-01-x86-64.appimage from 27.2.2017, Revision
7e3182fb6562c8bc01d689dcce9b99c7215b2ae9)
- Run face detection with vanilla settings -- around 28000 faces were detected
- In the 'unknown' tag, I selected groups of images (ignoring whether they were
actual faces or faces from different persons) and assigned them to different
'persons'. Current status:
  'Person A' ~ 10.000 tags
  'Person B' ~ 10.000 tags
  'Person C' ~ 4.500 tags
  7 other persons with 20 to 400 tags
- Now, selecting more images and assigning them to a person resulted in a
forced kill by the kernel due to memory exhaustion.

recognition.db is ~219MB

The system has 8GB RAM, and no swap. My normal working set of usually running
programs is ~1 GB RAM.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #11 from [hidden email] ---
new 5.6.0 pre-release as bundle is available here :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Please check if this problem still reproducible with these versions.

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

--- Comment #12 from [hidden email] ---
Issue still exists for 5.6.0 pre-release

Setup:
- 62800 pictures, JPG only
- stripped metadata, re-compressed with ImageMagick to save disc space and
exclude JPG issues (mogrify -strip -quality 50)

Start digikam:
  cd /var/tmp/dkdebug
  ./digikam-5.6.0-01-x86-64.appimage --config
/var/tmp/dkdebug/digikam-5.6.0.config

Vanilla digikam setup

Started face detection with default settings
2 x crashes (if the "Unknown" tag is selected while the detection is running)
~ 28900 faces (+ other) detected
Created 3 persons
Assigned ~10'000 faces to Person A, ~10'000 faces to Person B, ~5'000 faces to
Person C by selecting the thumbnails and assigning a name

Now, if a picture is opened and there are unassigned face tags in it, digikam
crashes with out-of-memory if I set a name to an unassigned face

digikam4.db: 28M
recognition.db: 213M
thumbnails-digikam.db: 1,7G

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=375035

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|5.4.0                       |5.6.0

--
You are receiving this mail because:
You are the assignee for the bug.