[digiKam-users] How to have DK utilize multiple cores for maintenance tasks

[Thomas D]

Hi,

As part of another thread, I just learned how to increase the max size of thumbnails.

When enabling large thumbnails in DK, I also have to rebuild all the thumbnails.

This, however, is a huge computational task that takes a long time.

I noticed that DK seems to only utilize about 7 % CPU time. My machine has 6 cores and I would like DK to utilize multiple cores. Maybe it already does and I/O is the bottleneck? Or do I need to enable multi-threaded thumbnail rebuilding somewhere?

BR

Thomas

[Chris Green]

Thomas D <[hidden email]> wrote:

> [-- text/plain, encoding 7bit, charset: UTF-8, 15 lines --]
>
> Hi,
>
> As part of another thread, I just learned how to increase the max size of
> thumbnails.
> When enabling large thumbnails in DK, I also have to rebuild all the
> thumbnails.
> This, however, is a huge computational task that takes a long time.
> I noticed that DK seems to only utilize about 7 % CPU time. My machine has
> 6 cores and I would like DK to utilize multiple cores. Maybe it already
> does and I/O is the bottleneck? Or do I need to enable multi-threaded
> thumbnail rebuilding somewhere?
>

Very, very few things use multiple cores.  The core i/o in the Linux
kernel isn't (as far as I know) multithreaded, nor are any of the
basic libraries.  Thus, even if you run lots of different 'things' at
the same time - e.g. firefox, digikam, a mail program, etc. the CPU
still only loads one processor.

I have seen a couple of programs which do actually use more than one
processor, one was a password cracking program and the other (if I
remember right) was an image processing program of some sort.

I think the reason these programs were able to use multiple CPUs is
that, once loaded they have all the data they need and don't interact
with anything else while 'thinking'.  It's thus relatively easy to
make them multi-threaded.

--
Chris Green
·

[Remco Viëtor]

On mercredi 14 octobre 2020 09:53:42 CEST Chris Green wrote:

> Thomas D <[hidden email]> wrote:
> > [-- text/plain, encoding 7bit, charset: UTF-8, 15 lines --]
> >
> > Hi,
> >
> > As part of another thread, I just learned how to increase the max size of
> > thumbnails.
> > When enabling large thumbnails in DK, I also have to rebuild all the
> > thumbnails.
> > This, however, is a huge computational task that takes a long time.
> > I noticed that DK seems to only utilize about 7 % CPU time. My machine has
> > 6 cores and I would like DK to utilize multiple cores. Maybe it already
> > does and I/O is the bottleneck? Or do I need to enable multi-threaded
> > thumbnail rebuilding somewhere?
>
> Very, very few things use multiple cores.  The core i/o in the Linux
> kernel isn't (as far as I know) multithreaded, nor are any of the
> basic libraries.  Thus, even if you run lots of different 'things' at
> the same time - e.g. firefox, digikam, a mail program, etc. the CPU
> still only loads one processor.
> (...)

And reading from/writing to disk is *very* slow compared to processing. So no
reason to look for trouble making thumbnail generation multi-threaded: no
benefit, and MT-code is hard to get correct.

Note that 7% CPU time corresponds to about half a core on a 6-core CPU...
And every thumbnail requires at least one disk read (original image) and one
disk write (thumbnail).

Remco

[Maik Qualmann]

DigiKam already uses multicore in many places. Almost all image editing tools use all CPU cores. Copying files is scaled to all cores. Face recognition can use all CPU cores, etc. We have a problem if loading images. A thumbnail can usually be quickly extracted from an existing preview, if not, the complete image has to be loaded into memory and then scaled down, depending on the image type. I have Giga Pixel test images from users that take up 3.5 gigabytes in memory. If we scale this to all cores, we need more than 20 gigabytes of main memory for such images. This means that the image loader would have to tell all tasks beforehand how much memory is required and the other loader tasks would have to pause. But the file IO on network drives also has limits. And last but not least, 6 tasks cannot write to the SQLite DB at the same time, only one at a time...

Maik

Am Mi., 14. Okt. 2020 um 10:16 Uhr schrieb Remco Viëtor <[hidden email]>:

On mercredi 14 octobre 2020 09:53:42 CEST Chris Green wrote:
> Thomas D <[hidden email]> wrote:
> > [-- text/plain, encoding 7bit, charset: UTF-8, 15 lines --]
> >
> > Hi,
> >
> > As part of another thread, I just learned how to increase the max size of
> > thumbnails.
> > When enabling large thumbnails in DK, I also have to rebuild all the
> > thumbnails.
> > This, however, is a huge computational task that takes a long time.
> > I noticed that DK seems to only utilize about 7 % CPU time. My machine has
> > 6 cores and I would like DK to utilize multiple cores. Maybe it already
> > does and I/O is the bottleneck? Or do I need to enable multi-threaded
> > thumbnail rebuilding somewhere?
>
> Very, very few things use multiple cores.  The core i/o in the Linux
> kernel isn't (as far as I know) multithreaded, nor are any of the
> basic libraries.  Thus, even if you run lots of different 'things' at
> the same time - e.g. firefox, digikam, a mail program, etc. the CPU
> still only loads one processor.
> (...)

And reading from/writing to disk is *very* slow compared to processing. So no
reason to look for trouble making thumbnail generation multi-threaded: no
benefit, and MT-code is hard to get correct.

Note that 7% CPU time corresponds to about half a core on a 6-core CPU...
And every thumbnail requires at least one disk read (original image) and one
disk write (thumbnail).

Remco