[digikam] [Bug 331224] New: Sync metadata eventually slows machine to unusable state

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

            Bug ID: 331224
           Summary: Sync metadata eventually slows machine to unusable
                    state
    Classification: Unclassified
           Product: digikam
           Version: 4.0.0-beta2
          Platform: Compiled Sources
                OS: Linux
            Status: UNCONFIRMED
          Severity: crash
          Priority: NOR
         Component: Maintenance
          Assignee: [hidden email]
          Reporter: [hidden email]

I have a collection of RAW files that were tagged using Digikam on another
machine, the metadata written to the XMP files.  Accessing these files via a
second installation of Digikam on a laptop the second install does not read in
the XMP data.
Trying to use the Maintenance feature to sync the metadata to the database over
the network creates a moving process indicator (with no percentage/activity
text) but not activity seems to be taking place.  The hard drive on the laptop
shows activity, but eventually the system slows so much that it doesn't respond
to mouse-clicks.
I tried plugging a USB removable drive of the same images into the laptop with
the same result.
As this was a rather large collection, I pointed the maintenance tools at
different directories (including one containing only 22 images) and the same
issue occurred.

Reproducible: Always

Steps to Reproduce:
1. Select Tools -> Maintenance...
2. Select a single album.
3. Select "Sync Metadata and Database" with sync direction "From image metadata
to database"
Actual Results:  
Digikam runs but does does not complete.  The system eventually slows down to
the point where the computer has to be shut off to stop the process.

Expected Results:  
Metadata would be read in from XMP files and populate the database.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Gilles Caulier <[hidden email]> ---
Do you use multicore option from maintenance tool ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #2 from Jeff Robinson <[hidden email]> ---
(In reply to comment #1)
> Do you use multicore option from maintenance tool ?
>
> Gilles Caulier

I did not have multicore enabled (I had missed that option), but I have the
same results with that option turned on.  Performance gradually degrades until
I have trouble even stopping the running programs.

To see if it was a network issue, I also copied a directory of 22 RAW Pentax
files (.PEF) to my laptop and tried the maintenance feature again locally.  I
appear to be getting the same results.

Is there a way I can check to make certain my Digikam database hasn't been
corrupted and that's causing this problem?

I have a sneaking suspicion that I'm running out of memory (8G physical memory
and swap).

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #3 from Gilles Caulier <[hidden email]> ---
 a memory leak can be very easily checkable using an external tool as htop for
ex. Just check if memory allocated by digiKam raise progressively...

The second stage is to understand where memory is lost. this can be into Exiv2
shared library which process whole metadata management with file (outside DB of
course).

If you have a suspected problem with DB, you can try to run a fresh digiKam
from a test account on your computer. Import new image a run maintenance tool
in same conditions.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #4 from Jeff Robinson <[hidden email]> ---
(In reply to comment #3)

>  a memory leak can be very easily checkable using an external tool as htop
> for ex. Just check if memory allocated by digiKam raise progressively...
>
> The second stage is to understand where memory is lost. this can be into
> Exiv2 shared library which process whole metadata management with file
> (outside DB of course).
>
> If you have a suspected problem with DB, you can try to run a fresh digiKam
> from a test account on your computer. Import new image a run maintenance
> tool in same conditions.
>
> Gilles Caulier

I created a new account and was able to have the same 22 files recognised
properly, bringing the information from the XMP files in as expected.

Going back to the original account, I took the step of renaming the existing
Digikam database and thumbnail data and restarting.

I added a local directory which was catalogued properly, including information
from the XMP sidecar for the RAW files.

I then added my networked portfolio drive (which took about 3 hours to
catalogue), and this also seems to be working properly.

I did try running the maintenance tool on this new install and the same issue
seems to occur.  Using htop I can watch the Digikam process add about 2MB of
memory usage every 3 seconds or so, with one of the cores always around 90%.

I may have been seeing two different issues with one caused by a damaged
database.  The maintenance tool problem still seems to persist for me, though.

The Exiv2 library I'm using is 0.23 from Slackware-Current, but I wouldn't know
how to go about finding out if it is the cause of a memory leak.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #5 from Gilles Caulier <[hidden email]> ---
To identify where memory is leak, run digiKam in a console through valgrind,
and reproduce all operation in GUI. Application will slow down with valgrind.
Report console trace here for future investigations.

Look details here :

http://www.digikam.org/contrib

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #6 from Jeff Robinson <[hidden email]> ---
bash-4.2$ valgrind --tool=memcheck --leak-check=full --error-limit=no digikam
==19952== Memcheck, a memory error detector
==19952== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==19952== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==19952== Command: digikam
==19952==
QSqlDatabasePrivate::removeDatabase: connection 'ConnectionTest' is still in
use, all queries will cease to work.
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECD5: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description()
(iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties()
(iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp()
(digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952==
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECEB: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description()
(iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties()
(iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp()
(digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952==
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECB0: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description()
(iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties()
(iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp()
(digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952==
==19952== Conditional jump or move depends on uninitialised value(s)
==19952==    at 0x1171ECC1: wcsnlen (in /lib64/libc-2.17.so)
==19952==    by 0x1171E44F: wcsrtombs (in /lib64/libc-2.17.so)
==19952==    by 0x116B4DA0: wcstombs (in /lib64/libc-2.17.so)
==19952==    by 0x7B978BB: ??? (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B99465: cmsReadICCTextEx (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7B9A1E5: cmsTakeProductDesc (in /usr/lib64/liblcms.so.1.0.19)
==19952==    by 0x7FCA82F: Digikam::IccProfile::description()
(iccprofile.cpp:377)
==19952==    by 0x7FE1239: Digikam::IccSettings::loadAllProfilesProperties()
(iccsettings.cpp:585)
==19952==    by 0x5AB43E: Digikam::DigikamApp::DigikamApp()
(digikamapp.cpp:227)
==19952==    by 0x49A9A9: main (main.cpp:189)
==19952==
==19952== Syscall param ioctl(generic) points to uninitialised byte(s)
==19952==    at 0x1176E249: syscall (in /lib64/libc-2.17.so)
==19952==    by 0x2431E3EF: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x24333CFD: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2431EB9D: v4lconvert_create_with_dev_ops (in
/usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2257033D: v4l2_fd_open (in /usr/lib64/libv4l2.so.0.0.0)
==19952==    by 0x30755507: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x3074FC64: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x307536B0: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x1741C064: g_object_get_valist (in
/usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x1741C4F6: g_object_get (in
/usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x2FFF615B: ??? (in
/usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==    by 0x2FFF71BA: ??? (in
/usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==  Address 0xffeffda64 is on thread 1's stack
==19952==
==19952== Syscall param ioctl(generic) points to uninitialised byte(s)
==19952==    at 0x1176E249: syscall (in /lib64/libc-2.17.so)
==19952==    by 0x2431E3EF: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x24333532: ??? (in /usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2431EB9D: v4lconvert_create_with_dev_ops (in
/usr/lib64/libv4lconvert.so.0.0.0)
==19952==    by 0x2257033D: v4l2_fd_open (in /usr/lib64/libv4l2.so.0.0.0)
==19952==    by 0x30755507: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x3074FC64: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x307536B0: ??? (in
/usr/lib64/gstreamer-0.10/libgstvideo4linux2.so)
==19952==    by 0x1741C064: g_object_get_valist (in
/usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x1741C4F6: g_object_get (in
/usr/lib64/libgobject-2.0.so.0.3600.4)
==19952==    by 0x2FFF615B: ??? (in
/usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==    by 0x2FFF71BA: ??? (in
/usr/lib64/kde4/plugins/phonon_backend/phonon_gstreamer.so)
==19952==  Address 0xffeffdecb is on thread 1's stack
==19952==
digikam(19952) Phonon::KdePlatformPlugin::createBackend: using backend:
"GStreamer"

There is no additional output after the "GStreamer" line, even when running the
maintenance tool.  I'll try the next instructions on the webpage for hangs and
other dysfunctions next.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #7 from Gilles Caulier <[hidden email]> ---
digiKam 4.0.0 is out :

http://www.digikam.org/node/713

Please check if this entry still valid with this new version.

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #8 from Jeff Robinson <[hidden email]> ---
(In reply to comment #7)
> digiKam 4.0.0 is out :
>
> http://www.digikam.org/node/713
>
> Please check if this entry still valid with this new version.
>
> Thanks in advance
>
> Gilles Caulier

I can confirm that this issue continues in my install of digikam 4.0.0 (x64).

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|4.0.0-beta2                 |4.0.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #9 from Gilles Caulier <[hidden email]> ---
What's about this file using last digiKam 4.2.0 ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Sync metadata eventually    |Sync metadata eventually
                   |slows machine to unusable   |slow-down computer to
                   |state                       |unusable state

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #10 from Gilles Caulier <[hidden email]> ---
digiKam 4.5.0 have been released.

Crash still reproducible with this release ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Jeff Robinson]

https://bugs.kde.org/show_bug.cgi?id=331224

--- Comment #11 from Jeff Robinson <[hidden email]> ---
(In reply to Gilles Caulier from comment #10)
> digiKam 4.5.0 have been released.
>
> Crash still reproducible with this release ?
>
> Gilles Caulier

I can confirm that with Digikam 4.5.0 I have successfully synchronized first a
directory of 40 images and then two directories with more than 400 images, and
everything seems to have worked correctly.

The system took approximately 3 minutes to do 400+ images.  I believe this bug
should be closed.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331224

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
   Version Fixed In|                            |4.6.0
             Status|UNCONFIRMED                 |RESOLVED

--
You are receiving this mail because:
You are the assignee for the bug.


Make the world a better place. Donate to our year end fundraiser https://www.kde.org/fundraisers/yearend2014/
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel