[digiKam-users] Anyone with a +300K photos internal mysql database?

[Rafael Linux]

I have two issues in Digikam, but  no one confirmed them. I imagine these
issues are only happening when having more than 300.000 photos and using
internal mysql database, but I need desesperately confirm with some other
user, cause I'm sure is related to the way Digikam manages an high quantity
of photos. These issues are:

Launching Digikam takes about 7 minutes to get it working and the other
issue is that when changing elements order takes near 5 minutes in a folder
with only 50 images (or less)  https://bugs.kde.org/show_bug.cgi?id=396086
<https://bugs.kde.org/show_bug.cgi?id=396086>  

Can anyone confirm this problems? I consider they "grave" cause can't I try
to work with Digikam professionaly, and this time is precious when I'm
working and desperate cause have no time!!!

Thank you








--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Gilles Caulier-4]

Hi,

Few question : 

- Did you use a network file sharing to hosts your collections ?

- Did you try to reproduce the problem with sqlite as database ?

- Can you reproduce the dysfunction with all collection hosted to a SSD ?

- Which digiKm version did you use ?

Note : here i use :

- SSD Samsung 1Tb to host all my collection (video and photo)

- sqlite with database file hosted on a SSD

- Linux system hosted on SSD.

- 32Gb of RAM

- i7 8 cores.

- More than 250.000 files cataloged.

- digiKam 6.0.0-beta1 (current implement as i code in fact)

There is no dysfunction visible in this configuration. Sure the first scan is long, but after, startup is faster.

I register few search virtual folders in my workflow.

Similarity searches are processed too.

Gilles Caulier

2018-08-21 11:36 GMT+02:00 Rafael Linux <[hidden email]>:

I have two issues in Digikam, but  no one confirmed them. I imagine these
issues are only happening when having more than 300.000 photos and using
internal mysql database, but I need desesperately confirm with some other
user, cause I'm sure is related to the way Digikam manages an high quantity
of photos. These issues are:

Launching Digikam takes about 7 minutes to get it working and the other
issue is that when changing elements order takes near 5 minutes in a folder
with only 50 images (or less)  https://bugs.kde.org/show_bug.cgi?id=396086
<https://bugs.kde.org/show_bug.cgi?id=396086> 

Can anyone confirm this problems? I consider they "grave" cause can't I try
to work with Digikam professionaly, and this time is precious when I'm
working and desperate cause have no time!!!

Thank you








--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[woenx]

I don't have as many pictures, only around 100K, and I can confirm it takes a
while to startup. In adition to that, the program is not responsible while
searching for new items, so that adds a few more minutes to the whole
starting process. However, my pictures are stored in a NAS and accessed
through wifi, so I blame the low network speed for those loading times.

But yes, it would be great if digikam could have faster starting times.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

*- Did you use a network file sharing to hosts your collections ?* No
*- Did you try to reproduce the problem with sqlite as database ?* No
I choose internal MySQL thinking precisely in that will be 370.000 photos to
manage. Anyway, if I look who is taking CPU when launching Digikam, I can
observe that Digikam is taken one core at 100%, while MySQL is reaching
barely 17%. I don't know what is doing Digikam at launch but have no sense
that is taking 100% on one core of the CPU.
*- Can you reproduce the dysfunction with all collection hosted to a SSD ?*
Yes
*- Which digiKam version did you use ?* 5.9 and tried with 6 beta, and same
problems

This is the configuration of the hardware and software:
- Intel i7, 16GiB of RAM
- SSD Toshiba 500MiB BTRFS filesystem, hosting OpenSUSE and user folder
("/home" with mysql database included).
- 370.000 JPG+RAW photos in a 4TiB SATA3 HD EXT4 file system (JPG and RAW,
no video)

I agree the first time Digikam (logically) took several hours to create read
all files and store all data, but then, each startup takes a lot of minutes.
One or two minutes ... will be acceptable, but we are talking of more than 5
minutes.

The same related to reordering (as I explain in the bug).

I use htop and *atop* system tools, to see if these issues are memory or I/O
related, but the only think that meets the eye in those tools is Digikam,
taking 100% CPU  :(



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Mikael Kujanpää]

Gilles Caulier kirjoitti 21.8.2018 12:47:
> - SSD Samsung 1Tb to host all my collection (video and photo)
> - sqlite with database file hosted on a SSD
> - Linux system hosted on SSD.
> - 32Gb of RAM
> - i7 8 cores.
> - More than 250.000 files cataloged.
> - digiKam 6.0.0-beta1 (current implement as i code in fact)

Just for the reference:
- Digikam 5.9
- 109561 (719 Gb) files and folders hosted on network share (nfs). On
the server, images are physically on HDD.
- /home and system files are on SSD
- I use sqlite
- i7 4770k processor, 16 Gb memory

Digikam startup takes 54 seconds from clicking the icon to the point
when database has been cleaned and new items have been scanned (0 new
items in this test). During that period Digikam is unusable, but after
that everything works as well as I could hope. I find the performance
pretty good. Of course I wish that startup wouldn't take so long, but
then again, it's over 100k pictures (0,49 ms/picture) so I can't really
complain.

Rafael, I suggest to test the sqlite option. I had my collection also on
MySql server in the past, and while it worked okay, I have to say that
sqlite has been better. It's faster and new Digikam versions work
without issues (I was forced to make the switch when schema upgrade
failed on my db between digikam major versions and no-one knew on this
mailing list what caused the weird sql error).

--
Mikki

[Gilles Caulier-4]

Just a question. Look in Setup/Miscs dialog page to see if Scan for new items at startup option is enabled ?

Gilles Caulier

2018-08-21 12:42 GMT+02:00 Rafael Linux <[hidden email]>:

*- Did you use a network file sharing to hosts your collections ?* No
*- Did you try to reproduce the problem with sqlite as database ?* No
I choose internal MySQL thinking precisely in that will be 370.000 photos to
manage. Anyway, if I look how is taking CPU when launching Digikam, I can
observe that Digikam is taken one core at 100%, while MySQL is reaching
barely 17%. I don't know what is doing Digikam at launch but have no sense
that is taking 100% on one core of the CPU.
*- Can you reproduce the dysfunction with all collection hosted to a SSD ?*
Yes
*- Which digiKam version did you use ?* 5.9 and tried with 6 beta, and same
problems

This is the configuration of the hardware and software:
- Intel i7, 16GiB of RAM
- SSD Toshiba 500MiB BTRFS filesystem, hosting OpenSUSE and user folder
("/home" with mysql database included).
- 370.000 JPG+RAW photos in a 4TiB SATA3 HD EXT4 file system (JPG and RAW,
no video)

I agree the first time Digikam (logically) took several hours to create read
all files and store all data, but then, each startup takes a lot of minutes.
One or two minutes ... will be acceptable, but we are talking of more than 5
minutes.

The same related to reordering (as I explain in the bug).

I use htop and *atop* system tools, to see if these issues are memory or I/O
related, but the only think that meets the eye in those tools is Digikam,
taking 100% CPU  :(

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

Option about you ask for is enabled. But while Digikam is starting, the
"startup window" remains visible those seven minutes (if I don't click that
startup window, cause in that case simply that startup window closes and I
don't know then if Digikam is loading or not).

Anyway, just after that startup window is automatically closed (after seven
or more minutes) the status bar shows nothing about exploring anything. It's
not doing any scan after that - or is so fast that I can't see it - , and,
as I said, "atop" doesn't show any massive reading from any application (nor
Digikam) while Digikam is starting. So, IMHO, is not scanning related and in
the case it were true, it should happen AFTER the startup window is
auto-closed and showed in the status bar as a progress bar.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[AndriusWild]

Are you talking about the splash screen?
I believe you can disable it actually.
Also, you can start digikam in the terminal and watch what is being posted
there while digikam hangs for 7+ minutes.

-----Original Message-----
From: Digikam-users [mailto:[hidden email]] On Behalf Of
Rafael Linux
Sent: August 21, 2018 6:46 AM
To: [hidden email]
Subject: Re: [digiKam-users] Anyone with a +300K photos internal mysql
database?

Option about you ask for is enabled. But while Digikam is starting, the
"startup window" remains visible those seven minutes (if I don't click that
startup window, cause in that case simply that startup window closes and I
don't know then if Digikam is loading or not).

Anyway, just after that startup window is automatically closed (after seven
or more minutes) the status bar shows nothing about exploring anything. It's
not doing any scan after that - or is so fast that I can't see it - , and,
as I said, "atop" doesn't show any massive reading from any application (nor
Digikam) while Digikam is starting. So, IMHO, is not scanning related and in
the case it were true, it should happen AFTER the startup window is
auto-closed and showed in the status bar as a progress bar.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[digikam-2]

On Tue, 21 Aug 2018 04:36:01 -0500 (CDT)
Rafael Linux <[hidden email]> wrote:

> I have two issues in Digikam, but  no one confirmed them. I imagine
> these issues are only happening when having more than 300.000
> photos and using internal mysql database, but I need desesperately
> confirm with some other user, cause I'm sure is related to the way
> Digikam manages an high quantity of photos. These issues are:
>
> Launching Digikam takes about 7 minutes to get it working and the
> other issue is that when changing elements order takes near 5
> minutes in a folder with only 50 images (or less)
> https://bugs.kde.org/show_bug.cgi?id=396086
> <https://bugs.kde.org/show_bug.cgi?id=396086>  

I had exactly the same problem but with about 50,000 images. DK was
slower than molasses.

After a while, somebody suggested that I run smartctl on my driveS
and sure enough, the drive with the images was on the way to where
hard drives go after a useful life...

I replaced the failing hd and everything went back to normal.

Do SSDs support smartctl? Don't know, if not: is there an equivalent
for SSDs? Is your system running too hot?...

--
sknahT

vyS

[leoutation]

On 8/21/18 11:36 AM, Rafael Linux wrote:
> I have two issues in Digikam, but  no one confirmed them. I imagine these
> issues are only happening when having more than 300.000 photos and using
> internal mysql database, but I need desesperately confirm with some other
> user, cause I'm sure is related to the way Digikam manages an high quantity
> of photos. These issues are:
>
> Launching Digikam takes about 7 minutes to get it working and the other
> issue is that when changing elements order takes near 5 minutes in a folder

Using external mariadb db (my case), is better, you can optimize it.
https://mariadb.com/kb/en/library/optimization-and-tuning/

[Rafael Linux]

Digikam-2, your problem was not the same that I have. In your case, any
Digikam I/O process should be a headache, cause was related with the hard
disk. But it's not my case. All hardware is new and I watch out for
hardware/software issues before tell something here. My case only happens
when Digikam needs to make some specific internal processes: In the startup
(while showing splash screen) and each time I reorder elements in a folder.
In fact, I begin to believe that maybe the same issue is in background.
Maybe when Digikam is loading, makes some kind or sorting. The same that we
can do in any album and in my case takes anormal time.

leoutation, as I said, mySQL is near idle on these cases, so is not the
source of issues. However, Digikam fire up to 100% of CPU use during that
lapse of time. Thank you for your suggestion.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Gilles Caulier-4]

Rafael,

I insist. Can you reproduce the problem when the scan at startup option is disabled. It's important because if the problem disapear, there is a chance to isolate the problem with virtual search album loading at startup.

Maik, you don't fix this problem in 6.0.0 ?

Another explainations about slow time loading at startup are given in this bugzilla file :

https://bugs.kde.org/show_bug.cgi?id=396086

Gilles Caulier

2018-08-21 14:45 GMT+02:00 Rafael Linux <[hidden email]>:

Option about you ask for is enabled. But while Digikam is starting, the
"startup window" remains visible those seven minutes (if I don't click that
startup window, cause in that case simply that startup window closes and I
don't know then if Digikam is loading or not).

Anyway, just after that startup window is automatically closed (after seven
or more minutes) the status bar shows nothing about exploring anything. It's
not doing any scan after that - or is so fast that I can't see it - , and,
as I said, "atop" doesn't show any massive reading from any application (nor
Digikam) while Digikam is starting. So, IMHO, is not scanning related and in
the case it were true, it should happen AFTER the startup window is
auto-closed and showed in the status bar as a progress bar.

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Maik Qualmann]

If the last used tab was the album tab, the virtual search albums will no
longer be executed on startup in digiKam-6.0.0. Of course, if the last used
tab was a search album, depending on the search, the start may take longer.

Another question to Rafael, how many albums does your collection have? From
30,000 albums it is here with me slowly interesting, that can take dan already
a few minutes. This is a problem we have not solved yet and where I see the
problem internally in Qt. I'll check it again here with Qt-5.11.

Maik

Am Donnerstag, 23. August 2018, 08:51:56 CEST schrieb Gilles Caulier:

> Rafael,
>
> I insist. Can you reproduce the problem when the scan at startup option is
> disabled. It's important because if the problem disapear, there is a chance
> to isolate the problem with virtual search album loading at startup.
>
> Maik, you don't fix this problem in 6.0.0 ?
>
> Another explainations about slow time loading at startup are given in this
> bugzilla file :
>
> https://bugs.kde.org/show_bug.cgi?id=396086
>
> Gilles Caulier
>
> 2018-08-21 14:45 GMT+02:00 Rafael Linux <[hidden email]>:
> > Option about you ask for is enabled. But while Digikam is starting, the
> > "startup window" remains visible those seven minutes (if I don't click
> > that
> > startup window, cause in that case simply that startup window closes and I
> > don't know then if Digikam is loading or not).
> >
> > Anyway, just after that startup window is automatically closed (after
> > seven
> > or more minutes) the status bar shows nothing about exploring anything.
> > It's
> > not doing any scan after that - or is so fast that I can't see it - , and,
> > as I said, "atop" doesn't show any massive reading from any application
> > (nor
> > Digikam) while Digikam is starting. So, IMHO, is not scanning related and
> > in
> > the case it were true, it should happen AFTER the startup window is
> > auto-closed and showed in the status bar as a progress bar.
> >
> >
> >
> > --
> > Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-> > f1735189.html

[Maik Qualmann]

It actually seems to be the number of albums. I have now created 20000 empty
albums and now I also have a very slow start and changing the sorting
direction of even small albums takes forever. I have something that I can
investigate...

Maik

Am Donnerstag, 23. August 2018, 23:09:56 CEST schrieben Sie:

> If the last used tab was the album tab, the virtual search albums will no
> longer be executed on startup in digiKam-6.0.0. Of course, if the last used
> tab was a search album, depending on the search, the start may take longer.
>
> Another question to Rafael, how many albums does your collection have? From
> 30,000 albums it is here with me slowly interesting, that can take dan
> already a few minutes. This is a problem we have not solved yet and where I
> see the problem internally in Qt. I'll check it again here with Qt-5.11.
>
> Maik
>
> Am Donnerstag, 23. August 2018, 08:51:56 CEST schrieb Gilles Caulier:
> > Rafael,
> >
> > I insist. Can you reproduce the problem when the scan at startup option is
> > disabled. It's important because if the problem disapear, there is a
> > chance
> > to isolate the problem with virtual search album loading at startup.
> >
> > Maik, you don't fix this problem in 6.0.0 ?
> >
> > Another explainations about slow time loading at startup are given in this
> > bugzilla file :
> >
> > https://bugs.kde.org/show_bug.cgi?id=396086
> >
> > Gilles Caulier
> >
> > 2018-08-21 14:45 GMT+02:00 Rafael Linux <[hidden email]>:
> > > Option about you ask for is enabled. But while Digikam is starting, the
> > > "startup window" remains visible those seven minutes (if I don't click
> > > that
> > > startup window, cause in that case simply that startup window closes and
> > > I
> > > don't know then if Digikam is loading or not).
> > >
> > > Anyway, just after that startup window is automatically closed (after
> > > seven
> > > or more minutes) the status bar shows nothing about exploring anything.
> > > It's
> > > not doing any scan after that - or is so fast that I can't see it - ,
> > > and,
> > > as I said, "atop" doesn't show any massive reading from any application
> > > (nor
> > > Digikam) while Digikam is starting. So, IMHO, is not scanning related
> > > and
> > > in
> > > the case it were true, it should happen AFTER the startup window is
> > > auto-closed and showed in the status bar as a progress bar.
> > >
> > >
> > >
> > > --
> > > Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-> >
> > > f1735189.html

[Rafael Linux]

I'm sorry Gilles, I replied yesterday from my mail manager, but the email was
rejected today (I don't know really why).

In the email I tell you that the photographer to who I installed Digikam is
on holidays till next week, so I can't try your suggestion (and really I
don't remember if I tried that before) so we need wait to next week to try
that change. So this I can't neither see how many folders (albums) he have, but I'm sure that there are more than 13k.

Anyway runned my bash script in my hardware, a simple desktop linux PC with
no SSD, and add the folders created with the script to Digikam, but it was a
headache, cause in some occasions that I launched Digikam, harddisk begins
to work forever till KDE seems to be freezed and I needed to try a desktop
restart (Ctrl + Alt +  2 x BackSpace), so I didn't try again cause it's my
day to day PC to work and I haven't time to try in that way.
 



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on
holidays this week, and I have no access to it, but I estimate that could be
+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the
issues weren't in the hardware or the database (mysql process never fired up
any CPU core).

If I can help anyway, let me know, but take into account that I can't do any
test in any moment, cause the real system is not mine, so I have not access
to it at my own  



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Guardian]

I am dropping in here.

I have 305k photos in many folders on a network share.
Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"
And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing
Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures.
But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----
Von: Digikam-users <[hidden email]> Im Auftrag von Rafael Linux
Gesendet: Freitag, 24. August 2018 00:54
An: [hidden email]
Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be
+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

As I wrote you, why do not test with the beta version. Digikam developers
said they improved this issue in the version 6 beta. I'm very interested
too, cause as I said, I can't test it whenever I want, it's not my PC.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Guardian]

Just installed version 6

First start took 9 minutes and then crashed immediately

Loading tools took about the same time as reading database.

Second start the same behavior. Digikam crashed right after start.

Maybe I create a new DB

-----Ursprüngliche Nachricht-----
Von: Digikam-users <[hidden email]> Im Auftrag von Rafael Linux
Gesendet: Freitag, 19. Oktober 2018 11:26
An: [hidden email]
Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

As I wrote you, why do not test with the beta version. Digikam developers said they improved this issue in the version 6 beta. I'm very interested too, cause as I said, I can't test it whenever I want, it's not my PC.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

Bad news then. To recreate database would not be a option in my case, cause
user has been documenting with tags and other info several photos and I'm
afraid that info could be lost.  Thank you for the info. Waiting for more.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html