Too many open DB connections

[Maik Qualmann]

Hi,

we have a problem with some database operations (MySQL) with too many open
connections established. For example, recognize faces. Because QThreadStorage
deletes the thread data when the thread is exit and thus also closes the DB,
the DB is opened again at the next DB action. To test the patch extensively, I
think only to commit for digiKam 5.9.0.

Maik

[Maik Qualmann]

Makes problems with SQLite...

Maik

Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> Hi,
>
> we have a problem with some database operations (MySQL) with too many open
> connections established. For example, recognize faces. Because
> QThreadStorage deletes the thread data when the thread is exit and thus
> also closes the DB, the DB is opened again at the next DB action. To test
> the patch extensively, I think only to commit for digiKam 5.9.0.
>
> Maik

[Gilles Caulier-4]

Hi Maik,

You want mean same problem with sqlite ?

Yes, we have plenty of bugs about this topic, for ex for Similarity, face, and thumbnails databases operation.

This mangle the memory and finally, for computer with small memory, crash the application violently.

If we can fix this problem, this will be a big step.

As my computer has plenty of memory, the problem is difficult to reproduce. I need to make a dedicated VM to test your patch.

So the problem is not to group transactions to prevent a bloating state with DB ? Fo rex, i know that thumbs registration is very verboze with the db and slow down the interface step by step, especially with remote DB. Right ?

Anyway, it's fine for me to introduce the patch for 5.9.0.

Gilles

2017-12-17 9:46 GMT+01:00 Maik Qualmann <[hidden email]>:

Makes problems with SQLite...

Maik

Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> Hi,
>
> we have a problem with some database operations (MySQL) with too many open
> connections established. For example, recognize faces. Because
> QThreadStorage deletes the thread data when the thread is exit and thus
> also closes the DB, the DB is opened again at the next DB action. To test
> the patch extensively, I think only to commit for digiKam 5.9.0.
>
> Maik

[Maik Qualmann]

The patch was not the problem with the SQLite. The error message on the face
recognize (not search) was after a long run that too many threads were
created. We must clean up the creating threads, this make the new patch. Also
the problem with too many DB connections is fixed. Gilles, what do you think
about commiting the patch?

Maik

Am Sonntag, 17. Dezember 2017, 12:01:17 CET schrieb Gilles Caulier:

> Hi Maik,
>
> You want mean same problem with sqlite ?
>
> Yes, we have plenty of bugs about this topic, for ex for Similarity, face,
> and thumbnails databases operation.
> This mangle the memory and finally, for computer with small memory, crash
> the application violently.
> If we can fix this problem, this will be a big step.
>
> As my computer has plenty of memory, the problem is difficult to reproduce.
> I need to make a dedicated VM to test your patch.
>
> So the problem is not to group transactions to prevent a bloating state
> with DB ? Fo rex, i know that thumbs registration is very verboze with the
> db and slow down the interface step by step, especially with remote DB.
> Right ?
>
> Anyway, it's fine for me to introduce the patch for 5.9.0.
>
> Gilles
>
> 2017-12-17 9:46 GMT+01:00 Maik Qualmann <[hidden email]>:
> > Makes problems with SQLite...
> >
> > Maik
> >
> > Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> > > Hi,
> > >
> > > we have a problem with some database operations (MySQL) with too many
> >
> > open
> >
> > > connections established. For example, recognize faces. Because
> > > QThreadStorage deletes the thread data when the thread is exit and thus
> > > also closes the DB, the DB is opened again at the next DB action. To
> > > test
> > > the patch extensively, I think only to commit for digiKam 5.9.0.
> > >
> > > Maik

[Gilles Caulier-4]

Maik,

My first test, done in my lead computer with 32Gb of RAM and 8 cores do not report a special dysfunction. The memory allocation is normal, and no time latency while processing with face recognition.  But this have been already the case before. I processed more that 150.000 files in 45 mn.

I will now play with a VM and less memory.

Gilles

2017-12-17 12:01 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi Maik,

You want mean same problem with sqlite ?

Yes, we have plenty of bugs about this topic, for ex for Similarity, face, and thumbnails databases operation.

This mangle the memory and finally, for computer with small memory, crash the application violently.

If we can fix this problem, this will be a big step.

As my computer has plenty of memory, the problem is difficult to reproduce. I need to make a dedicated VM to test your patch.

So the problem is not to group transactions to prevent a bloating state with DB ? Fo rex, i know that thumbs registration is very verboze with the db and slow down the interface step by step, especially with remote DB. Right ?

Anyway, it's fine for me to introduce the patch for 5.9.0.

Gilles

2017-12-17 9:46 GMT+01:00 Maik Qualmann <[hidden email]>:

Makes problems with SQLite...

Maik

Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> Hi,
>
> we have a problem with some database operations (MySQL) with too many open
> connections established. For example, recognize faces. Because
> QThreadStorage deletes the thread data when the thread is exit and thus
> also closes the DB, the DB is opened again at the next DB action. To test
> the patch extensively, I think only to commit for digiKam 5.9.0.
>
> Maik

[Gilles Caulier-4]

Hum, before to commit, let's me test in my host computer. Applying and recompiling... Wait and see.

Gilles

2017-12-17 16:59 GMT+01:00 Maik Qualmann <[hidden email]>:

The patch was not the problem with the SQLite. The error message on the face
recognize (not search) was after a long run that too many threads were
created. We must clean up the creating threads, this make the new patch. Also
the problem with too many DB connections is fixed. Gilles, what do you think
about commiting the patch?

Maik

Am Sonntag, 17. Dezember 2017, 12:01:17 CET schrieb Gilles Caulier:
> Hi Maik,
>
> You want mean same problem with sqlite ?
>
> Yes, we have plenty of bugs about this topic, for ex for Similarity, face,
> and thumbnails databases operation.
> This mangle the memory and finally, for computer with small memory, crash
> the application violently.
> If we can fix this problem, this will be a big step.
>
> As my computer has plenty of memory, the problem is difficult to reproduce.
> I need to make a dedicated VM to test your patch.
>
> So the problem is not to group transactions to prevent a bloating state
> with DB ? Fo rex, i know that thumbs registration is very verboze with the
> db and slow down the interface step by step, especially with remote DB.
> Right ?
>
> Anyway, it's fine for me to introduce the patch for 5.9.0.
>
> Gilles
>
> 2017-12-17 9:46 GMT+01:00 Maik Qualmann <[hidden email]>:
> > Makes problems with SQLite...
> >
> > Maik
> >
> > Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> > > Hi,
> > >
> > > we have a problem with some database operations (MySQL) with too many
> >
> > open
> >
> > > connections established. For example, recognize faces. Because
> > > QThreadStorage deletes the thread data when the thread is exit and thus
> > > also closes the DB, the DB is opened again at the next DB action. To
> > > test
> > > the patch extensively, I think only to commit for digiKam 5.9.0.
> > >
> > > Maik

[Gilles Caulier-4]

Q : your new patch is smaller than first one. It's normal ? I need to apply both or first is merged in second and improved ?

Gilles

2017-12-17 17:02 GMT+01:00 Gilles Caulier <[hidden email]>:

Hum, before to commit, let's me test in my host computer. Applying and recompiling... Wait and see.

Gilles

2017-12-17 16:59 GMT+01:00 Maik Qualmann <[hidden email]>:

The patch was not the problem with the SQLite. The error message on the face
recognize (not search) was after a long run that too many threads were
created. We must clean up the creating threads, this make the new patch. Also
the problem with too many DB connections is fixed. Gilles, what do you think
about commiting the patch?

Maik

Am Sonntag, 17. Dezember 2017, 12:01:17 CET schrieb Gilles Caulier:
> Hi Maik,
>
> You want mean same problem with sqlite ?
>
> Yes, we have plenty of bugs about this topic, for ex for Similarity, face,
> and thumbnails databases operation.
> This mangle the memory and finally, for computer with small memory, crash
> the application violently.
> If we can fix this problem, this will be a big step.
>
> As my computer has plenty of memory, the problem is difficult to reproduce.
> I need to make a dedicated VM to test your patch.
>
> So the problem is not to group transactions to prevent a bloating state
> with DB ? Fo rex, i know that thumbs registration is very verboze with the
> db and slow down the interface step by step, especially with remote DB.
> Right ?
>
> Anyway, it's fine for me to introduce the patch for 5.9.0.
>
> Gilles
>
> 2017-12-17 9:46 GMT+01:00 Maik Qualmann <[hidden email]>:
> > Makes problems with SQLite...
> >
> > Maik
> >
> > Am Sonntag, 17. Dezember 2017, 09:39:06 CET schrieb Maik Qualmann:
> > > Hi,
> > >
> > > we have a problem with some database operations (MySQL) with too many
> >
> > open
> >
> > > connections established. For example, recognize faces. Because
> > > QThreadStorage deletes the thread data when the thread is exit and thus
> > > also closes the DB, the DB is opened again at the next DB action. To
> > > test
> > > the patch extensively, I think only to commit for digiKam 5.9.0.
> > >
> > > Maik