[digiKam-users] Anyone with a +300K photos internal mysql database?

[Maik Qualmann]

That's really not good news. Since digiKam-6.0.0-beta2 has significantly increased in speed here with MySQL. See here:

http://digikam.1695700.n4.nabble.com/digiKam-users-The-task-quot-usr-bin-digikam-qwindowtitle-digikam-quot-appears-twice-in-htop-td4706945.html#a4706956

How many albums (folders) does your collection have?

The images are local or over network?

Maik

Am Fr., 19. Okt. 2018 um 12:38 Uhr schrieb Rafael Linux <[hidden email]>:

Bad news then. To recreate database would not be a option in my case, cause
user has been documenting with tags and other info several photos and I'm
afraid that info could be lost.  Thank you for the info. Waiting for more.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rob D]

I have found that the majority of the startup time is spent reading the similarity database.

Startup takes about 8 minutes on a image collection of about 400K images. Most of the time is spent on the query "SELECT value FROM ImageSimilarity WHERE ( imageid1=? OR imageid2=? ) AND algorithm=? ..." (I don't have any of the query after the ?. Sorry)

Nothing is logged to the console while reading the database.If I replace the similarity database with a blank one, startup is reduced to about 34 seconds.

If I re-build the database, start up times are again slow.

This is running DK6 Beta2 Oct 15 release.

Rob

On Fri, 2018-10-19 at 08:22 +0000, Dumelle, Timo (E+K) wrote:

I am dropping in here.

I have 305k photos in many folders on a network share.

Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"

And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing

Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures. 

But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----

Von: Digikam-users <

[hidden email]

> Im Auftrag von Rafael Linux

Gesendet: Freitag, 24. August 2018 00:54

An: 

[hidden email]

Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be

+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  

--

Sent from: 

http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[PackElend]

Could you share how to figure out that. 

That would help others doing the same debugging 

Le ven. 19 oct. 2018 à 16:39, Rob Dueckman <[hidden email]> a écrit :

I have found that the majority of the startup time is spent reading the similarity database.

Startup takes about 8 minutes on a image collection of about 400K images. Most of the time is spent on the query "SELECT value FROM ImageSimilarity WHERE ( imageid1=? OR imageid2=? ) AND algorithm=? ..." (I don't have any of the query after the ?. Sorry)

Nothing is logged to the console while reading the database.If I replace the similarity database with a blank one, startup is reduced to about 34 seconds.

If I re-build the database, start up times are again slow.

This is running DK6 Beta2 Oct 15 release.

Rob

On Fri, 2018-10-19 at 08:22 +0000, Dumelle, Timo (E+K) wrote:

I am dropping in here.

I have 305k photos in many folders on a network share.

Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"

And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing

Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures. 

But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----

Von: Digikam-users <

[hidden email]

> Im Auftrag von Rafael Linux

Gesendet: Freitag, 24. August 2018 00:54

An: 

[hidden email]

Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be

+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  

--

Sent from: 

http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rob D]

Sorry I forgot to mention that I'm using MySQL/MariaDB standalone as my DigiKam database

To see what processes are running on your MySQL/MariaDB server, issue the following command from the mysql shell when connected as a user with access to the DigiKam database:

show full processlist;

You should see a list of current database processes including any running queries.

Rob

On Fri, 2018-10-19 at 16:42 +0200, Stefan Mueller wrote:

Could you share how to figure out that. 

That would help others doing the same debugging 

Le ven. 19 oct. 2018 à 16:39, Rob Dueckman <[hidden email]> a écrit :

I have found that the majority of the startup time is spent reading the similarity database.

Startup takes about 8 minutes on a image collection of about 400K images. Most of the time is spent on the query "SELECT value FROM ImageSimilarity WHERE ( imageid1=? OR imageid2=? ) AND algorithm=? ..." (I don't have any of the query after the ?. Sorry)

Nothing is logged to the console while reading the database.If I replace the similarity database with a blank one, startup is reduced to about 34 seconds.

If I re-build the database, start up times are again slow.

This is running DK6 Beta2 Oct 15 release.

Rob

On Fri, 2018-10-19 at 08:22 +0000, Dumelle, Timo (E+K) wrote:

I am dropping in here.

I have 305k photos in many folders on a network share.

Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"

And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing

Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures. 

But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----

Von: Digikam-users <

[hidden email]

> Im Auftrag von Rafael Linux

Gesendet: Freitag, 24. August 2018 00:54

An: 

[hidden email]

Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be

+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  

--

Sent from: 

http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Rafael Linux]

Very useful way to analyze the issue but, how can I do that monitorization if
I'm using internal MySQL (but the same slooooow start)? Is there any way?
Thank you



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Guardian]

Hi Rob,

 

i can’t find a table called imageSimilarity in my MySQL db.

 

Can you point out where it should be?

 

Von: Digikam-users <[hidden email]> Im Auftrag von Rob Dueckman
Gesendet: Freitag, 19. Oktober 2018 16:40
An: digiKam - Home Manage your photographs as a professional with the power of open source <[hidden email]>
Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

 

I have found that the majority of the startup time is spent reading the similarity database.

 

Startup takes about 8 minutes on a image collection of about 400K images. Most of the time is spent on the query "SELECT value FROM ImageSimilarity WHERE ( imageid1=? OR imageid2=? ) AND algorithm=? ..." (I don't have any of the query after the ?. Sorry)

 

Nothing is logged to the console while reading the database.If I replace the similarity database with a blank one, startup is reduced to about 34 seconds.

 

If I re-build the database, start up times are again slow.

 

This is running DK6 Beta2 Oct 15 release.

 

 

Rob

 

 

 

On Fri, 2018-10-19 at 08:22 +0000, Dumelle, Timo (E+K) wrote:

I am dropping in here.

I have 305k photos in many folders on a network share.

Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"

And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing

Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures. 

But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----

Von: Digikam-users <

[hidden email]

[hidden email]

 

> Im Auftrag von Rafael Linux

Gesendet: Freitag, 24. August 2018 00:54

An: 

[hidden email]

[hidden email]

 

Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be

+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  

--

Sent from: 

http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

 

[Guardian]

And to add:

I got it running with an other Windows Client.

Now it’s the 6 beta 2

Startup still slow – and it seems I don’t have a similarity DB (there is no entry for the DB Name)

 

Von: Digikam-users <[hidden email]> Im Auftrag von Dumelle, Timo (E+K)
Gesendet: Montag, 22. Oktober 2018 10:35
An: digiKam - Home Manage your photographs as a professional with the power of open source <[hidden email]>
Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

 

Hi Rob,

 

i can’t find a table called imageSimilarity in my MySQL db.

 

Can you point out where it should be?

 

Von: Digikam-users <[hidden email]> Im Auftrag von Rob Dueckman
Gesendet: Freitag, 19. Oktober 2018 16:40
An: digiKam - Home Manage your photographs as a professional with the power of open source <[hidden email]>
Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

 

I have found that the majority of the startup time is spent reading the similarity database.

 

Startup takes about 8 minutes on a image collection of about 400K images. Most of the time is spent on the query "SELECT value FROM ImageSimilarity WHERE ( imageid1=? OR imageid2=? ) AND algorithm=? ..." (I don't have any of the query after the ?. Sorry)

 

Nothing is logged to the console while reading the database.If I replace the similarity database with a blank one, startup is reduced to about 34 seconds.

 

If I re-build the database, start up times are again slow.

 

This is running DK6 Beta2 Oct 15 release.

 

 

Rob

 

 

 

On Fri, 2018-10-19 at 08:22 +0000, Dumelle, Timo (E+K) wrote:

I am dropping in here.

I have 305k photos in many folders on a network share.

Many of the folders on that share don't contain pictures, but since the paths are kinda "mixed up" with other files I can't just scan the photos.

Dk needs about 5 - 10 minutes to start up while hanging on "reading database"

And nearly every operation that goes beyond just viewing pics in albums cause it to either not responding for a long time or crashing

Currently it is finding duplicates which seems to proceed pretty fast.

Picasa for example has absolute no problems with this pictures. 

But I need server functions for about 20 clients to connect to the DB

Can I help with any of these tests?

-----Ursprüngliche Nachricht-----

Von: Digikam-users <

[hidden email]

 

> Im Auftrag von Rafael Linux

Gesendet: Freitag, 24. August 2018 00:54

An: 

[hidden email]

 

Betreff: Re: [digiKam-users] Anyone with a +300K photos internal mysql database?

Maik, as I wrote to Gilles, I can't see how many folders cause the user is on holidays this week, and I have no access to it, but I estimate that could be

+13k albums, mostly with JPG photos.

I want to thank you your test, cause I was sure that the reason of the issues weren't in the hardware or the database (mysql process never fired up any CPU core).

If I can help anyway, let me know, but take into account that I can't do any test in any moment, cause the real system is not mine, so I have not access to it at my own  

--

Sent from: 

http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html