Batch Queue Manager and database lock.

[Gilles Caulier-4]

Hi all,

I remember some fixes done in more and less recent version about BQM
about database lock, especially from Maik.

While this summer i take a lots of panorama in French Alps. I use
typically BQM in background to adjust the colors of target files (JPG)
using a dedicated BQM workflow :

1/ Local Contrast to render image as pseudo HDR (the photo is more luminous).
2/ WB tool to increase the saturation.

For ex : https://goo.gl/photos/FVisJdYHd4i8mZzF9


Processing Pano files of 200Mpx in BQM take age. I don't use
multi-core because my laptop has just 4 cores (i5). I want to use
another one to perform triage, or use editor, or run Panorama tool.

And the problem appear. While BQM tool is running, navigating in album
contents give empty views. No item are visible. When BQM is done, all
re-appear. It's always reproducible.

I don't suspect a database lock, but something wrong with the
ActionThreadBase instance from BQM which reserve all cores and block
database interface which use also ActionThreadBase interface.

Typically, i think that set max cpu core to use with QThreadPool
(inside ActionThreadBase) is something static. One QThreadPool is
certainly a singleton in Qt and changing the number of threads in one
instance of ActionThreadBase change the behavior for all
ActionThreadBase instance.

What do you think about ? Can somebody can confirm the problem ?

[Maik Qualmann]

I can reproduce the problem. I look for the cause.

Maik

Am Samstag, 12. August 2017, 17:53:52 CEST schrieb Gilles Caulier:

> Hi all,
>
> I remember some fixes done in more and less recent version about BQM
> about database lock, especially from Maik.
>
> While this summer i take a lots of panorama in French Alps. I use
> typically BQM in background to adjust the colors of target files (JPG)
> using a dedicated BQM workflow :
>
> 1/ Local Contrast to render image as pseudo HDR (the photo is more
> luminous). 2/ WB tool to increase the saturation.
>
> For ex : https://goo.gl/photos/FVisJdYHd4i8mZzF9
>
>
> Processing Pano files of 200Mpx in BQM take age. I don't use
> multi-core because my laptop has just 4 cores (i5). I want to use
> another one to perform triage, or use editor, or run Panorama tool.
>
> And the problem appear. While BQM tool is running, navigating in album
> contents give empty views. No item are visible. When BQM is done, all
> re-appear. It's always reproducible.
>
> I don't suspect a database lock, but something wrong with the
> ActionThreadBase instance from BQM which reserve all cores and block
> database interface which use also ActionThreadBase interface.
>
> Typically, i think that set max cpu core to use with QThreadPool
> (inside ActionThreadBase) is something static. One QThreadPool is
> certainly a singleton in Qt and changing the number of threads in one
> instance of ActionThreadBase change the behavior for all
> ActionThreadBase instance.
>
> What do you think about ? Can somebody can confirm the problem ?

[Maik Qualmann]

Gilles,

please test the behavior of digiKam after my last commits.

Maik

Am Samstag, 12. August 2017, 19:33:10 CEST schrieb Maik Qualmann:

> I can reproduce the problem. I look for the cause.
>
> Maik
>
> Am Samstag, 12. August 2017, 17:53:52 CEST schrieb Gilles Caulier:
> > Hi all,
> >
> > I remember some fixes done in more and less recent version about BQM
> > about database lock, especially from Maik.
> >
> > While this summer i take a lots of panorama in French Alps. I use
> > typically BQM in background to adjust the colors of target files (JPG)
> > using a dedicated BQM workflow :
> >
> > 1/ Local Contrast to render image as pseudo HDR (the photo is more
> > luminous). 2/ WB tool to increase the saturation.
> >
> > For ex : https://goo.gl/photos/FVisJdYHd4i8mZzF9
> >
> >
> > Processing Pano files of 200Mpx in BQM take age. I don't use
> > multi-core because my laptop has just 4 cores (i5). I want to use
> > another one to perform triage, or use editor, or run Panorama tool.
> >
> > And the problem appear. While BQM tool is running, navigating in album
> > contents give empty views. No item are visible. When BQM is done, all
> > re-appear. It's always reproducible.
> >
> > I don't suspect a database lock, but something wrong with the
> > ActionThreadBase instance from BQM which reserve all cores and block
> > database interface which use also ActionThreadBase interface.
> >
> > Typically, i think that set max cpu core to use with QThreadPool
> > (inside ActionThreadBase) is something static. One QThreadPool is
> > certainly a singleton in Qt and changing the number of threads in one
> > instance of ActionThreadBase change the behavior for all
> > ActionThreadBase instance.
> >
> > What do you think about ? Can somebody can confirm the problem ?

[Maik Qualmann]

Yes, the number of threads we use applies to all instances. We should always
use all CPU cores. In BQM it makes no sense to limit it to a CPU core. Where
was the problem in the maintenance tool?

Maik

Am Samstag, 12. August 2017, 17:53:52 CEST schrieb Gilles Caulier:
> Typically, i think that set max cpu core to use with QThreadPool
> (inside ActionThreadBase) is something static. One QThreadPool is
> certainly a singleton in Qt and changing the number of threads in one
> instance of ActionThreadBase change the behavior for all
> ActionThreadBase instance.

[Gilles Caulier-4]

2017-08-13 8:40 GMT+02:00 Maik Qualmann <[hidden email]>:
> Yes, the number of threads we use applies to all instances. We should always
> use all CPU cores. In BQM it makes no sense to limit it to a CPU core. Where
> was the problem in the maintenance tool?

Your patches do not compile with Qt 5.6 LTS:

[ 39%] Building CXX object
core/libs/threads/CMakeFiles/dthread_src.dir/actionthreadbase.cpp.o
/mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp: In member
function ‘void Digikam::ActionThreadBase::cancel()’:
/mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp:177:18:
error: ‘class QThreadPool’ has no member named ‘tryTake’
        d->pool->tryTake(job);
                 ^
core/libs/threads/CMakeFiles/dthread_src.dir/build.make:62: recipe for
target 'core/libs/threads/CMakeFiles/dthread_src.dir/action
threadbase.cpp.o' failed

Using one CPU core make a sence everywhere. in BQM, if you use
multicore (8 for ex on my host computer) you will run 8 BQM task at
the same time. The memory allocation can be huge, depending of image
sizes to process. If all memory is allocated, DK will crash.

There is an entry in bugzilla if i remember, to be more intelligent
with cpu cores to handle, depending of system resources available.

The problem is exactly the same with Maintenance tools.

and also, the problem is more critical with 32 bits OS where memory
avaialble is limited to 4Gb, where 1Gb is typically already reserved
for the OS.

Gilles

[Maik Qualmann]

Gilles,

Yes, you are right with the memory usage in the BQM at multicore. Compiling is
now fixed. QThreadPool is global, we should not limit the number of possible
threads. However, with single core option, always start only new job when the
other is finished.

Maik

Am Sonntag, 13. August 2017, 12:09:57 CEST schrieb Gilles Caulier:

> 2017-08-13 8:40 GMT+02:00 Maik Qualmann <[hidden email]>:
> > Yes, the number of threads we use applies to all instances. We should
> > always use all CPU cores. In BQM it makes no sense to limit it to a CPU
> > core. Where was the problem in the maintenance tool?
>
> Your patches do not compile with Qt 5.6 LTS:
>
> [ 39%] Building CXX object
> core/libs/threads/CMakeFiles/dthread_src.dir/actionthreadbase.cpp.o
> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp: In member
> function ‘void Digikam::ActionThreadBase::cancel()’:
> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp:177:18:
> error: ‘class QThreadPool’ has no member named ‘tryTake’
>         d->pool->tryTake(job);
>                  ^
> core/libs/threads/CMakeFiles/dthread_src.dir/build.make:62: recipe for
> target 'core/libs/threads/CMakeFiles/dthread_src.dir/action
> threadbase.cpp.o' failed
>
> Using one CPU core make a sence everywhere. in BQM, if you use
> multicore (8 for ex on my host computer) you will run 8 BQM task at
> the same time. The memory allocation can be huge, depending of image
> sizes to process. If all memory is allocated, DK will crash.
>
> There is an entry in bugzilla if i remember, to be more intelligent
> with cpu cores to handle, depending of system resources available.
>
> The problem is exactly the same with Maintenance tools.
>
> and also, the problem is more critical with 32 bits OS where memory
> avaialble is limited to 4Gb, where 1Gb is typically already reserved
> for the OS.
>
> Gilles

[Gilles Caulier-4]

I just tested your patch, and this do not solve the problem.

I add 100 files to rotate and unsharp to BQM (JPEG) and run with one
coreto use. I go back to album view and try to naviguate between
album. All still empty until BQM has completed one task in progress.
In this case, the selected album appear. BQM continue in background.
If i switch to another album, the icon view is empty... until next BQM
task is complete.

I see in you patch that you tune job collection item priority to
process. Take a care to not confuse "priority of thread and priority
to process a thread. If i remember the priority here is the delay to
process a thread in the QThreadPool (a delay in ms before to process
the job). I'm not 100%% sure, but look to QThreadPool::start() API
details.

Gilles

2017-08-13 12:26 GMT+02:00 Maik Qualmann <[hidden email]>:

> Gilles,
>
> Yes, you are right with the memory usage in the BQM at multicore. Compiling is
> now fixed. QThreadPool is global, we should not limit the number of possible
> threads. However, with single core option, always start only new job when the
> other is finished.
>
> Maik
>
> Am Sonntag, 13. August 2017, 12:09:57 CEST schrieb Gilles Caulier:
>> 2017-08-13 8:40 GMT+02:00 Maik Qualmann <[hidden email]>:
>> > Yes, the number of threads we use applies to all instances. We should
>> > always use all CPU cores. In BQM it makes no sense to limit it to a CPU
>> > core. Where was the problem in the maintenance tool?
>>
>> Your patches do not compile with Qt 5.6 LTS:
>>
>> [ 39%] Building CXX object
>> core/libs/threads/CMakeFiles/dthread_src.dir/actionthreadbase.cpp.o
>> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp: In member
>> function ‘void Digikam::ActionThreadBase::cancel()’:
>> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp:177:18:
>> error: ‘class QThreadPool’ has no member named ‘tryTake’
>>         d->pool->tryTake(job);
>>                  ^
>> core/libs/threads/CMakeFiles/dthread_src.dir/build.make:62: recipe for
>> target 'core/libs/threads/CMakeFiles/dthread_src.dir/action
>> threadbase.cpp.o' failed
>>
>> Using one CPU core make a sence everywhere. in BQM, if you use
>> multicore (8 for ex on my host computer) you will run 8 BQM task at
>> the same time. The memory allocation can be huge, depending of image
>> sizes to process. If all memory is allocated, DK will crash.
>>
>> There is an entry in bugzilla if i remember, to be more intelligent
>> with cpu cores to handle, depending of system resources available.
>>
>> The problem is exactly the same with Maintenance tools.
>>
>> and also, the problem is more critical with 32 bits OS where memory
>> avaialble is limited to 4Gb, where 1Gb is typically already reserved
>> for the OS.
>>
>> Gilles

[Maik Qualmann]

Gilles,

please test the last commit. DigiKam hast almost better reaction also in the
maintenance tool. ((:-))

Maik

Am Sonntag, 13. August 2017, 12:51:35 CEST schrieb Gilles Caulier:

> I just tested your patch, and this do not solve the problem.
>
> I add 100 files to rotate and unsharp to BQM (JPEG) and run with one
> coreto use. I go back to album view and try to naviguate between
> album. All still empty until BQM has completed one task in progress.
> In this case, the selected album appear. BQM continue in background.
> If i switch to another album, the icon view is empty... until next BQM
> task is complete.
>
> I see in you patch that you tune job collection item priority to
> process. Take a care to not confuse "priority of thread and priority
> to process a thread. If i remember the priority here is the delay to
> process a thread in the QThreadPool (a delay in ms before to process
> the job). I'm not 100%% sure, but look to QThreadPool::start() API
> details.
>
> Gilles
>
> 2017-08-13 12:26 GMT+02:00 Maik Qualmann <[hidden email]>:
> > Gilles,
> >
> > Yes, you are right with the memory usage in the BQM at multicore.
> > Compiling is now fixed. QThreadPool is global, we should not limit the
> > number of possible threads. However, with single core option, always
> > start only new job when the other is finished.
> >
> > Maik
> >
> > Am Sonntag, 13. August 2017, 12:09:57 CEST schrieb Gilles Caulier:
> >> 2017-08-13 8:40 GMT+02:00 Maik Qualmann <[hidden email]>:
> >> > Yes, the number of threads we use applies to all instances. We should
> >> > always use all CPU cores. In BQM it makes no sense to limit it to a CPU
> >> > core. Where was the problem in the maintenance tool?
> >>
> >> Your patches do not compile with Qt 5.6 LTS:
> >>
> >> [ 39%] Building CXX object
> >> core/libs/threads/CMakeFiles/dthread_src.dir/actionthreadbase.cpp.o
> >> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp: In member
> >> function ‘void Digikam::ActionThreadBase::cancel()’:
> >> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp:177:18:
> >> error: ‘class QThreadPool’ has no member named ‘tryTake’
> >>
> >>         d->pool->tryTake(job);
> >>        
> >>                  ^
> >>
> >> core/libs/threads/CMakeFiles/dthread_src.dir/build.make:62: recipe for
> >> target 'core/libs/threads/CMakeFiles/dthread_src.dir/action
> >> threadbase.cpp.o' failed
> >>
> >> Using one CPU core make a sence everywhere. in BQM, if you use
> >> multicore (8 for ex on my host computer) you will run 8 BQM task at
> >> the same time. The memory allocation can be huge, depending of image
> >> sizes to process. If all memory is allocated, DK will crash.
> >>
> >> There is an entry in bugzilla if i remember, to be more intelligent
> >> with cpu cores to handle, depending of system resources available.
> >>
> >> The problem is exactly the same with Maintenance tools.
> >>
> >> and also, the problem is more critical with 32 bits OS where memory
> >> avaialble is limited to 4Gb, where 1Gb is typically already reserved
> >> for the OS.
> >>
> >> Gilles

[Gilles Caulier-4]

Maik,

Yes all work perfectly now. Elemental : creating one thread poll by
ActionThreadBase instance is the right solution. Great job.

Note : i think some file in bugzilla can be closed now with this fix,
but this wil be difficult to identify the side effect of this
dysfunction everywhere in application.

Gilles

2017-08-16 19:15 GMT+02:00 Maik Qualmann <[hidden email]>:

> Gilles,
>
> please test the last commit. DigiKam hast almost better reaction also in the
> maintenance tool. ((:-))
>
> Maik
>
> Am Sonntag, 13. August 2017, 12:51:35 CEST schrieb Gilles Caulier:
>> I just tested your patch, and this do not solve the problem.
>>
>> I add 100 files to rotate and unsharp to BQM (JPEG) and run with one
>> coreto use. I go back to album view and try to naviguate between
>> album. All still empty until BQM has completed one task in progress.
>> In this case, the selected album appear. BQM continue in background.
>> If i switch to another album, the icon view is empty... until next BQM
>> task is complete.
>>
>> I see in you patch that you tune job collection item priority to
>> process. Take a care to not confuse "priority of thread and priority
>> to process a thread. If i remember the priority here is the delay to
>> process a thread in the QThreadPool (a delay in ms before to process
>> the job). I'm not 100%% sure, but look to QThreadPool::start() API
>> details.
>>
>> Gilles
>>
>> 2017-08-13 12:26 GMT+02:00 Maik Qualmann <[hidden email]>:
>> > Gilles,
>> >
>> > Yes, you are right with the memory usage in the BQM at multicore.
>> > Compiling is now fixed. QThreadPool is global, we should not limit the
>> > number of possible threads. However, with single core option, always
>> > start only new job when the other is finished.
>> >
>> > Maik
>> >
>> > Am Sonntag, 13. August 2017, 12:09:57 CEST schrieb Gilles Caulier:
>> >> 2017-08-13 8:40 GMT+02:00 Maik Qualmann <[hidden email]>:
>> >> > Yes, the number of threads we use applies to all instances. We should
>> >> > always use all CPU cores. In BQM it makes no sense to limit it to a CPU
>> >> > core. Where was the problem in the maintenance tool?
>> >>
>> >> Your patches do not compile with Qt 5.6 LTS:
>> >>
>> >> [ 39%] Building CXX object
>> >> core/libs/threads/CMakeFiles/dthread_src.dir/actionthreadbase.cpp.o
>> >> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp: In member
>> >> function ‘void Digikam::ActionThreadBase::cancel()’:
>> >> /mnt/data/GIT/5.x/core/libs/threads/actionthreadbase.cpp:177:18:
>> >> error: ‘class QThreadPool’ has no member named ‘tryTake’
>> >>
>> >>         d->pool->tryTake(job);
>> >>
>> >>                  ^
>> >>
>> >> core/libs/threads/CMakeFiles/dthread_src.dir/build.make:62: recipe for
>> >> target 'core/libs/threads/CMakeFiles/dthread_src.dir/action
>> >> threadbase.cpp.o' failed
>> >>
>> >> Using one CPU core make a sence everywhere. in BQM, if you use
>> >> multicore (8 for ex on my host computer) you will run 8 BQM task at
>> >> the same time. The memory allocation can be huge, depending of image
>> >> sizes to process. If all memory is allocated, DK will crash.
>> >>
>> >> There is an entry in bugzilla if i remember, to be more intelligent
>> >> with cpu cores to handle, depending of system resources available.
>> >>
>> >> The problem is exactly the same with Maintenance tools.
>> >>
>> >> and also, the problem is more critical with 32 bits OS where memory
>> >> avaialble is limited to 4Gb, where 1Gb is typically already reserved
>> >> for the OS.
>> >>
>> >> Gilles
>
>