[digikam] [Bug 318577] New: Batch queue fails to process multiple RAW files simultaneously (in parallel)

[Eduard Huguet]

https://bugs.kde.org/show_bug.cgi?id=318577

            Bug ID: 318577
           Summary: Batch queue fails to process multiple RAW files
                    simultaneously (in parallel)
    Classification: Unclassified
           Product: digikam
           Version: 3.1.0
          Platform: Fedora RPMs
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: NOR
         Component: Batch Queue Manager
          Assignee: [hidden email]
          Reporter: [hidden email]

I'm trying to process a batch of RAW files in Digikam in order to convert them
to uncompressed PNG format. To do so, I add the RAW files to the queue, then
set up RAW open options and the "Convert to PNG".

When files are processed, only a minor part of them are opened succesfully,
while most of them fail. Logging traces in .xsession-errors show error messages
regarding corruption found while trying to open the RAW files.

However, the RAW files (from a Sony Alpha-1, which is a long-time supported
model) are not corrupted, and can be imported with no issues if I open them
individually.

The interesting part, however, is that if I boot the system with "maxcpus=1"
kernel setting, then the files are batch-processed sucessfully, which suggest
some kind of problem related to multiple processing of RAW files simultaneosly
(my computer has 4 cores, so the BQM attempts to process 4 files at a time
without that option).

I suppose the problem is related to libdcraw or similar, not Digikam itself,
but maybe Digikam should allow for some workaround (like an option to limit MT
in BQM, or either disabling it completely if RAW files are processed until the
error gets fixed).





Reproducible: Always

Steps to Reproduce:
1. select multiple RAW files from main Digikam screen, add them to a new queue
2. set up RAW opening options, then a conversion tool (in my case "Convert to
PNG")
3. start processing files
Actual Results:  
Most files are processed with errors, thus leading to empty result PNG files.
The error appears to happen when RAW import tool tries to load the file.

Expected Results:  
All RAW files successfully converted to PNG.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Gilles Caulier <[hidden email]> ---
With 3.2.0-beta2, i can process Alpha A67 RAW in PNG without any problem here
(i7 computer, 6 cores)...

http://www.flickr.com/photos/digikam/8661749113/sizes/o/in/photostream/

What's your RAW workflow in BQM ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Eduard Huguet]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #2 from Eduard Huguet <[hidden email]> ---
(In reply to comment #1)
> With 3.2.0-beta2, i can process Alpha A67 RAW in PNG without any problem
> here (i7 computer, 6 cores)...
>
> http://www.flickr.com/photos/digikam/8661749113/sizes/o/in/photostream/
>
> What's your RAW workflow in BQM ?
>
> Gilles Caulier

Hi, and thank you for the quick reaction :-)

In this  case, the one described above: I simply select the RAW files I want to
convert (which I downloaded from the camera), and I add them to a new queue.
Once in BQM, I adjust RAW opening options (nothing special: AMAZe filtering and
leave most other options as default - have checked using Bilinear with the same
results...), then add a single process step consisting on the "Convert to PNG"
tool.

I do this because I prefer to batch-convert all RAW files to PNG first, then
edit every PNG one by one. I used to do that using old "Batch RAW converter"
tool, but now this one has gone, as it has been replaced by BQM (which totally
makes sense to me, anyway). However, I'm finding this unexpected problem. IIRC,
the old tool was not processing files in parallel, so that might be the reason
this problem arises now.

It might be something related to specific versions of Digikam, libdcraw, etc...
packed in Fedora 18.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #3 from Gilles Caulier <[hidden email]> ---
Since 3.0.0, kipi RAW converter processed files in parallel. You can try it
from Gwenview  for ex. Tool is just disabled in digiKam. It use the same
background process than BQM to paralelize operations, but implementation is
more complex of course due of capability to chain tools in a workflow.

You don't need to turn your computer to single cpu through linux kernel option.
In BQM, it's possible to turn off paralelization by patch source code like this
:

//-----------------------------------------------------------------------------------------------------------------
diff --git a/utilities/queuemanager/manager/actionthread.cpp
b/utilities/queuemanager/manager/actionthread.cpp
index 7a0a004..5f33727 100644
--- a/utilities/queuemanager/manager/actionthread.cpp
+++ b/utilities/queuemanager/manager/actionthread.cpp
@@ -55,6 +55,8 @@ public:
 ActionThread::ActionThread(QObject* const parent)
     : RActionThreadBase(parent), d(new Private)
 {
+    setMaximumNumberOfThreads(1);
+
     qRegisterMetaType<ActionData>();

     connect(this, SIGNAL(finished()),
//-----------------------------------------------------------------------------------------------------------------

Please try it just to confirm the problem with paralelization.

Q : why i cannot reproduce this problem on my computer. I use Sony Alpha camera
in RAW and as you, i play with processed images in PNG...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Eduard Huguet]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #4 from Eduard Huguet <[hidden email]> ---
(In reply to comment #3)

> Since 3.0.0, kipi RAW converter processed files in parallel. You can try it
> from Gwenview  for ex. Tool is just disabled in digiKam. It use the same
> background process than BQM to paralelize operations, but implementation is
> more complex of course due of capability to chain tools in a workflow.
>
> You don't need to turn your computer to single cpu through linux kernel
> option. In BQM, it's possible to turn off paralelization by patch source
> code like this :
>
> //---------------------------------------------------------------------------
> --------------------------------------
> diff --git a/utilities/queuemanager/manager/actionthread.cpp
> b/utilities/queuemanager/manager/actionthread.cpp
> index 7a0a004..5f33727 100644
> --- a/utilities/queuemanager/manager/actionthread.cpp
> +++ b/utilities/queuemanager/manager/actionthread.cpp
> @@ -55,6 +55,8 @@ public:
>  ActionThread::ActionThread(QObject* const parent)
>      : RActionThreadBase(parent), d(new Private)
>  {
> +    setMaximumNumberOfThreads(1);
> +
>      qRegisterMetaType<ActionData>();
>
>      connect(this, SIGNAL(finished()),
> //---------------------------------------------------------------------------
> --------------------------------------
>
> Please try it just to confirm the problem with paralelization.
>
> Q : why i cannot reproduce this problem on my computer. I use Sony Alpha
> camera in RAW and as you, i play with processed images in PNG...
>
> Gilles Caulier

Interesting. I'll try later at home with the old batch convert tool in
Gwenview, hadn't thought about it.

Regarding about disabling parallelization by patching source code, I might give
it a try later. I have a long experience regarding C++ & Qt, I just need to
find the time to set up the compilation environment for Digikam, which I assume
I will be complex (lots of dependencies needed, I guess...)

And regarding final question: I don't know, frankly :-( . I'll retry later at
home just to be sure that the problem is related to parallelization. Maybe it
has to be with the version of the RAW decoding library (libdcraw,
libopenraw...?) used in F18 (as I suspect that the bug is really not in Digikam
itself, but there instead...).

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #5 from Gilles Caulier <[hidden email]> ---
Git commit dd3dc1cdd2292f2c12b2927250619ae4781f3a61 by Gilles Caulier.
Committed on 27/07/2013 at 09:10.
Pushed by cgilles into branch 'master'.

Batch Queue Manager : add new option to turn on/off multi-core support. This
option is now turn off by default due to some dysfunctions under Windows, until
it will be hack and fixed.
Note that problem are not reproducible under OSX and Linux here. Options is
implemnted for digiKam 3.3.0 release.
Related: bug 315025, bug 318198, bug 320358

M  +13   -1    utilities/queuemanager/manager/actionthread.cpp
M  +1    -1    utilities/queuemanager/manager/actionthread.h
M  +4    -1    utilities/queuemanager/manager/queuesettings.h
M  +9    -1    utilities/queuemanager/manager/workflowmanager.cpp
M  +14   -2    utilities/queuemanager/views/queuesettingsview.cpp

http://commits.kde.org/digikam/dd3dc1cdd2292f2c12b2927250619ae4781f3a61

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #6 from Gilles Caulier <[hidden email]> ---
*** Bug 320358 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Batch queue fails to        |Batch queue fails to
                   |process multiple RAW files  |process multiple files
                   |simultaneously (in          |simultaneously (in
                   |parallel)                   |parallel)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #7 from Gilles Caulier <[hidden email]> ---
*** Bug 315025 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #8 from Gilles Caulier <[hidden email]> ---
Eduard,

This file still valid using last digiKam 4.2.0 ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Eduard Huguet]

https://bugs.kde.org/show_bug.cgi?id=318577

--- Comment #9 from Eduard Huguet <[hidden email]> ---
Hi, Gilles
Just testing it now using 4.2.0 from Fedora 20, and it seems to work fine.
Thank you very much for your hard work!

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=318577

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
   Version Fixed In|                            |4.3.0
             Status|UNCONFIRMED                 |RESOLVED

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=318577

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|BatchQueueManager           |BatchQueueManager-Workflow

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel