[digikam] [Bug 331742] New: BQM takes a lot of time processing >200 images

https://bugs.kde.org/show_bug.cgi?id=331742

            Bug ID: 331742
           Summary: BQM takes a lot of time processing >200 images
    Classification: Unclassified
           Product: digikam
           Version: 3.5.0
          Platform: openSUSE RPMs
                OS: Linux
            Status: UNCONFIRMED
          Severity: crash
          Priority: NOR
         Component: Batch Queue Manager
          Assignee: [hidden email]
          Reporter: [hidden email]

openSUSE x86_64 / KDE4.12.2

In an album with around 1000 images I selected 410 images (jpeg) through the
search, then place them in the BQM.
Selecting the rescaling tool and start the batch process.
Output goes to a new album.
Original size was around 5000px width and I selected to reduce size to 1920px
width.
The process of rescaling itself was done in approx. 2 Minutes, but after than I
recognized a lot of SQL activity which took around 30 minutes.
So all in all the whole process of rescaling took > 30 minutes, while
throughout this time digiKam wasn't responding in any way.

Attached is an image that shows the CPU-Usage during this process.
Visibly it divides in 4 blocks:
First high CPU-Usage - images are rescale, after that all new images are
physically there.
Second block of low CPU-Usage - some SQL-Activity
Third block of intermittent CPU-Usage - lots of (re)reads to the newly created
files and lots of SQL-Activity.
Forth block of constant CPU-Usage - moderate (re)reads to the newly created
files and moderate SQL-Activity to the thumbnail-DB.



Reproducible: Always

Steps to Reproduce:
1. Select >200 (in my case 410) images for batch processing
2. Output goes to a new album
3. Selected tool rescale with 1920px
Actual Results:  
Process takes much too long, digiKam is not responding during batch process

Expected Results:  
Process should be much faster (and in the past 2.x version it was)

Due to not responding for over 30 minutes the user must think the process is
crashed or hangs.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #1 from [hidden email] ---
Created attachment 85413
  --> https://bugs.kde.org/attachment.cgi?id=85413&action=edit
CPU-Usage while batch processing

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #2 from Gilles Caulier <[hidden email]> ---
digiKam 4.0.0 is out :

http://www.digikam.org/node/713

Please check if this entry still valid with this new version.

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #3 from [hidden email] ---
Still valid - I just checked with the new package (digiKam 4.0.0-8.3) from
openSUSE 13.1 / KDE 4.13.
But the behavior changed a bit:
digiKam is not really hanging and still shows activity, but it is very slowly
responding.
So you still see the status bar moving and the queue is getting refreshed.

And I noticed another issue with this behavior:
When you output the resulting images in a new album and wait til the task is
finished, the new album shows less images then it has on disk.

BTW: Nevertheless digiKam is awesome and 4.0 is really excellent - great job
:-)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #4 from [hidden email] ---
I run another test with MySQL DB-Backend and found some error messages in the
logfile may this would be helpful:

140516 18:33:58 [Warning] Aborted connection 36 to db: 'digikamdb' user:
'digikam' host: '' (Unknown error)
140516 18:36:59 [Warning] Aborted connection 37 to db: 'digikamdb' user:
'digikam' host: '' (Unknown error)
140516 18:37:03 [Warning] Aborted connection 87 to db: 'digikamdb' user:
'digikam' host: '' (Unknown error)
140516 18:41:53 [Warning] Aborted connection 91 to db: 'digikamdb' user:
'digikam' host: '' (Unknown error)
140516 19:29:21 [Warning] Aborted connection 32 to db: 'digikamdb' user:
'digikam' host: '' (Unknown error)

The last warning repeated 50 times with the same timestamp (19:29:21) at this
time I shutdown digiKam

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #5 from Gilles Caulier <[hidden email]> ---
Regi,

This file still valid using last digiKam 4.2.0 ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|crash                       |normal

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|BQM takes a lot of time     |This takes a lot of time
                   |processing >200 images      |processing > 200 images in
                   |                            |BQM

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #6 from [hidden email] ---
(In reply to Gilles Caulier from comment #5)
> Regi,
>
> This file still valid using last digiKam 4.2.0 ?
>
> Gilles Caulier

Hi Gilles,
sorry to say, but yes.

While I'm trying to find out what is a "save" number of images for processing,
I recognized that if I start with less images (let's say 150) - in the first
run everything's fine.
Repeating the process at least 3 times, the initial described issue happens.
I don't know if this is pointing to a memory leak?

Regi

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.5.0                       |4.2.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #7 from Gilles Caulier <[hidden email]> ---
*** Bug 346012 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Konstantin-2]

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #8 from Konstantin <[hidden email]> ---
In Bug 346012 there some useful addictional info that explress that this bug
affect all of BQM operations(it more visible if "use all cores" turn on and if
output would be in other folder, but it affect BQM even without these
conditions).

I think it's all because of DB locks like
digikam(9707)/digikam (core)
Digikam::DatabaseCoreBackendPrivate::checkRetrySQLiteLockError: Detected locked
database file. There is an active transaction. Waited but giving up now.
The more these locks showed - the more application is not responding.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Maik Qualmann]

https://bugs.kde.org/show_bug.cgi?id=331742

Maik Qualmann <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #9 from Maik Qualmann <[hidden email]> ---
Created attachment 92305
  --> https://bugs.kde.org/attachment.cgi?id=92305&action=edit
BatchQueueManager.patch

The slow down of BQM and the locked database message I could reproduce. This
patch fixes it for me. Before we close this bug, he should be tested with more
than 2 CPU cores.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|This takes a lot of time    |This takes a lot of time
                   |processing > 200 images in  |processing > 200 images in
                   |BQM                         |BQM [patch]

--- Comment #10 from Gilles Caulier <[hidden email]> ---
Confirmed. Patch solve the problem here too (8 cores)

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Maik Qualmann]

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #11 from Maik Qualmann <[hidden email]> ---
Thanks, I move the patch into branch 'master'.

Maik

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Maik Qualmann]

https://bugs.kde.org/show_bug.cgi?id=331742

Maik Qualmann <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|UNCONFIRMED                 |RESOLVED
   Version Fixed In|                            |4.10.0
      Latest Commit|                            |http://commits.kde.org/digi
                   |                            |kam/901788011ff6489a63ccdbb
                   |                            |5ade0b6438f74f0e8

--- Comment #12 from Maik Qualmann <[hidden email]> ---
Git commit 901788011ff6489a63ccdbb5ade0b6438f74f0e8 by Maik Qualmann.
Committed on 28/04/2015 at 19:05.
Pushed by mqualmann into branch 'master'.

apply patch #92305 to suspend collection scan when the BQM is running
FIXED-IN: 4.10.0

M  +2    -1    NEWS
M  +2    -0    utilities/queuemanager/main/queuemgrwindow.cpp

http://commits.kde.org/digikam/901788011ff6489a63ccdbb5ade0b6438f74f0e8

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=331742

--- Comment #13 from Gilles Caulier <[hidden email]> ---
Git commit 93bb098c76861316a536fbf9aef582881a03b2a4 by Gilles Caulier.
Committed on 29/04/2015 at 13:15.
Pushed by cgilles into branch 'frameworks'.

backport commit #901788011ff6489a63ccdbb5ade0b6438f74f0e8 from git/master to
frameworks branch

M  +2    -0    utilities/queuemanager/main/queuemgrwindow.cpp

http://commits.kde.org/digikam/93bb098c76861316a536fbf9aef582881a03b2a4

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=331742

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|BatchQueueManager           |BatchQueueManager-Workflow

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel