Reserved memory usage for facial recognition

[nghiaduong]

Hi all,

I am testing a spatial database implementation on SQLite for facial recognition of faces engine. For now, we can perform K-Nearest search on a table but I am concerned about its performance. For a table of 16000 entries, the search time is about 700ms and it's a little bit slow. Therefore, I am considering storing a KD-Tree structure directly on memory. I estimate that the memory needed for a data node is from 512 bytes to 1500 bytes, both on memory and on the database.

I don't know whether it is too much. Could you let me know the amount of memory that can be reserved for this part of faces engine, please? Thank you in advance.

Best regards,

Nghia

[Maik Qualmann]

We reserve memory for the DImg cache, depending on the existing one, up to
200MB.

https://invent.kde.org/graphics/digikam/-/blob/master/core/libs/threadimageio/
fileio/loadingcache.cpp#L169

So around 200MB for the face engine, if they are still securely in the system,
I would find okay for current computers with around 8GB of memory.

But SQLite is quite fast, did you set the parameter to activate the SQLite
cache?

Maik

Am Sonntag, 12. Juli 2020, 08:27:22 CEST schrieb Nghia Duong:

> Hi all,
>
> I am testing a spatial database implementation on SQLite for facial
> recognition of faces engine. For now, we can perform K-Nearest search on a
> table but I am concerned about its performance. For a table of 16000
> entries, the search time is about 700ms and it's a little bit slow.
> Therefore, I am considering storing a KD-Tree structure directly on memory.
> I estimate that the memory needed for a data node is from 512 bytes to 1500
> bytes, both on memory and on the database.
>
> I don't know whether it is too much. Could you let me know the amount of
> memory that can be reserved for this part of faces engine, please? Thank
> you in advance.

[Gilles Caulier-4]

Hi Nghia,

The amount memory available on host system is detected at startup by
DImg using code from core/libs/kmemoryinfo/

You must remember this code, as we have used it in my office last
winter -:=)))...

You said that 700ms is slow. How did you evaluate this state ? Did you
compare with other algorithm ?

Did you compile the code without debug symbols and with G++
optimizations (there is just one CMake configuration flag to switch
for that)

Best

Gilles

Le dim. 12 juil. 2020 à 10:51, Maik Qualmann <[hidden email]> a écrit :

>
> We reserve memory for the DImg cache, depending on the existing one, up to
> 200MB.
>
> https://invent.kde.org/graphics/digikam/-/blob/master/core/libs/threadimageio/
> fileio/loadingcache.cpp#L169
>
> So around 200MB for the face engine, if they are still securely in the system,
> I would find okay for current computers with around 8GB of memory.
>
> But SQLite is quite fast, did you set the parameter to activate the SQLite
> cache?
>
> Maik
>
> Am Sonntag, 12. Juli 2020, 08:27:22 CEST schrieb Nghia Duong:
> > Hi all,
> >
> > I am testing a spatial database implementation on SQLite for facial
> > recognition of faces engine. For now, we can perform K-Nearest search on a
> > table but I am concerned about its performance. For a table of 16000
> > entries, the search time is about 700ms and it's a little bit slow.
> > Therefore, I am considering storing a KD-Tree structure directly on memory.
> > I estimate that the memory needed for a data node is from 512 bytes to 1500
> > bytes, both on memory and on the database.
> >
> > I don't know whether it is too much. Could you let me know the amount of
> > memory that can be reserved for this part of faces engine, please? Thank
> > you in advance.
>
>
>
>

[nghiaduong]

I tried to perform K-Nearest algorithm with a binary search directly on memory and a binary search on the database, with a dataset of about 16000 faces. The one with the KD-Tree on the memory is pretty quick, which takes about 120ms to recognize a face. The one performed directly on the database is slower (700ms per search). The current version of SQL driver supported by Qt doesn't support spatial indexing so I have to implement the binary search by multiple consecutive queries, so it takes a bit of time.

I estimate that it would take about 10Mb to stock 16000 faces information. Therefore I think 200Mb cache will be enough to store the entire KD-Tree structure on RAM for the facial recognition process.

Nghia