Face recognition AI / DL and adding features

[cerp]

Hi everybody!

I have been testing 7.0.0 and I love the AI features that recognizes  
the faces.

I would like to extend it to recognize other types of features,  
because we use digikam to manage an archive of botanical pictures, and  
it would be very useful for us. The first test would be to identify  
plants pictures within the database  (not the individual species, just  
whether the picture contains a plant).

I looked at the existing documentation, but it not really sufficiently  
in depth, it only gives a generic overview.

Is there documentation that explains what approach was used to  
developing the new component for face recognition?

Of course, I would be happy to share with the community the outputs of  
any development.

Regards

Corrado Topi

[Gilles Caulier-4]

Hi,

The non face detection / recognition is a topic already discussed in
this room and with the developers in the past.

Typically, before to work of monuments, plants, animals, forms, etc,
we need to finalize the face management. It's under progress while
this summer as we have 2 students working on this topic.

I resumed the direction to plan for future development in this bugzilla entry :

https://bugs.kde.org/show_bug.cgi?id=416988

Porting Photils to digiKam plugin interface as a new generic plugin is
the first step. This will permit the evaluation of the performance of
the algorithm.

Best

Gilles Caulier

Le lun. 17 août 2020 à 11:53, <[hidden email]> a écrit :

>
> Hi everybody!
>
> I have been testing 7.0.0 and I love the AI features that recognizes
> the faces.
>
> I would like to extend it to recognize other types of features,
> because we use digikam to manage an archive of botanical pictures, and
> it would be very useful for us. The first test would be to identify
> plants pictures within the database  (not the individual species, just
> whether the picture contains a plant).
>
> I looked at the existing documentation, but it not really sufficiently
> in depth, it only gives a generic overview.
>
> Is there documentation that explains what approach was used to
> developing the new component for face recognition?
>
> Of course, I would be happy to share with the community the outputs of
> any development.
>
> Regards
>
> Corrado Topi
>
>

[cerp]

Dear Gilles,

Thanks for the link. I looked at the bugzilla thread, and it is very useful.

In my case, I think we may need to start developing a DL classifier  
earlier, I cannot wait for Digikam to have stabilised the face search,  
so I may have to go a different way if I am not allowed to create a  
Digikam plugin for plants classification. We have several 10s of  
thousands of pictures, with many TB of pictures, and manual screening  
is now impossible. I am happy to share my results of course.

Thanks for the link to Photils, which seems to be useful for us. I am  
not completely clear: is photils the engine that is currently used in  
Digikam for face recognition with the new AI / DL approach, or is this  
in the future?

Best Regards

Quoting Gilles Caulier <[hidden email]>:

> Hi,
>
> The non face detection / recognition is a topic already discussed in
> this room and with the developers in the past.
>
> Typically, before to work of monuments, plants, animals, forms, etc,
> we need to finalize the face management. It's under progress while
> this summer as we have 2 students working on this topic.
>
> I resumed the direction to plan for future development in this  
> bugzilla entry :
>
> https://bugs.kde.org/show_bug.cgi?id=416988
>
> Porting Photils to digiKam plugin interface as a new generic plugin is
> the first step. This will permit the evaluation of the performance of
> the algorithm.
>
> Best
>
> Gilles Caulier
>
> Le lun. 17 août 2020 à 11:53, <[hidden email]> a écrit :
>>
>> Hi everybody!
>>
>> I have been testing 7.0.0 and I love the AI features that recognizes
>> the faces.
>>
>> I would like to extend it to recognize other types of features,
>> because we use digikam to manage an archive of botanical pictures, and
>> it would be very useful for us. The first test would be to identify
>> plants pictures within the database  (not the individual species, just
>> whether the picture contains a plant).
>>
>> I looked at the existing documentation, but it not really sufficiently
>> in depth, it only gives a generic overview.
>>
>> Is there documentation that explains what approach was used to
>> developing the new component for face recognition?
>>
>> Of course, I would be happy to share with the community the outputs of
>> any development.
>>
>> Regards
>>
>> Corrado Topi
>>
>>

[Gilles Caulier-4]

Le lun. 17 août 2020 à 13:34, <[hidden email]> a écrit :

>
> Dear Gilles,
>
> Thanks for the link. I looked at the bugzilla thread, and it is very useful.
>
> In my case, I think we may need to start developing a DL classifier
> earlier, I cannot wait for Digikam to have stabilised the face search,
> so I may have to go a different way if I am not allowed to create a
> Digikam plugin for plants classification. We have several 10s of
> thousands of pictures, with many TB of pictures, and manual screening
> is now impossible. I am happy to share my results of course.

About plants we have this entry to create a new plugin :
https://bugs.kde.org/show_bug.cgi?id=394544


>
> Thanks for the link to Photils, which seems to be useful for us. I am
> not completely clear: is photils the engine that is currently used in
> Digikam for face recognition with the new AI / DL approach, or is this
> in the future?

No. digiKam as a dedicated internal engine to process detection and
recognition based on AI.

Photils is an alternative which compute a fingerprint of images, send
it to a AI engine which try to identify objects, monuments, animals,
etc... It return a list of keywords.

Gilles Caulier

[cerp]

Dear Gilles,

Thanks a lot. Some comments in the text.

Quoting Gilles Caulier <[hidden email]>:

> Le lun. 17 août 2020 à 13:34, <[hidden email]> a écrit :
>>
>> Dear Gilles,
>>
>> Thanks for the link. I looked at the bugzilla thread, and it is very useful.
>>
>> In my case, I think we may need to start developing a DL classifier
>> earlier, I cannot wait for Digikam to have stabilised the face search,
>> so I may have to go a different way if I am not allowed to create a
>> Digikam plugin for plants classification. We have several 10s of
>> thousands of pictures, with many TB of pictures, and manual screening
>> is now impossible. I am happy to share my results of course.
>
> About plants we have this entry to create a new plugin :
> https://bugs.kde.org/show_bug.cgi?id=394544

Thank you! This is exceptionally useful.

>
>
>>
>> Thanks for the link to Photils, which seems to be useful for us. I am
>> not completely clear: is photils the engine that is currently used in
>> Digikam for face recognition with the new AI / DL approach, or is this
>> in the future?

I looked at Photils and tried it. I think it may be useful but not  
really optimal for our use. We need something that we can train at  
different levels of detail (initially plant /  not plan, then family,  
species, subspecies etc. etc.) , possibly based on Deep Learning.

>
> No. digiKam as a dedicated internal engine to process detection and
> recognition based on AI.

What technology does Digicam use? I assume Digikam is using DL, but  
what starting from which libraries? I assume you did not develop  
everything from the ground up, because there are so many interesting  
libraries available.

>
> Photils is an alternative which compute a fingerprint of images, send
> it to a AI engine which try to identify objects, monuments, animals,
> etc... It return a list of keywords.

I understood the mechanism, and trialled it, and it turns out to be  
too generic for our use. I would prefer something we can train, and  
that can be used at different level of details (initially plant /  not  
plant, then family, species, subspecies etc. etc.).


Thanks a lot.

>
> Gilles Caulier

[Gilles Caulier-4]

Le lun. 17 août 2020 à 20:26, <[hidden email]> a écrit :

>
> Dear Gilles,
>
> Thanks a lot. Some comments in the text.
>
> Quoting Gilles Caulier <[hidden email]>:
>
> > Le lun. 17 août 2020 à 13:34, <[hidden email]> a écrit :
> >>
> >> Dear Gilles,
> >>
> >> Thanks for the link. I looked at the bugzilla thread, and it is very useful.
> >>
> >> In my case, I think we may need to start developing a DL classifier
> >> earlier, I cannot wait for Digikam to have stabilised the face search,
> >> so I may have to go a different way if I am not allowed to create a
> >> Digikam plugin for plants classification. We have several 10s of
> >> thousands of pictures, with many TB of pictures, and manual screening
> >> is now impossible. I am happy to share my results of course.
> >
> > About plants we have this entry to create a new plugin :
> > https://bugs.kde.org/show_bug.cgi?id=394544
>
> Thank you! This is exceptionally useful.
>
> >
> >
> >>
> >> Thanks for the link to Photils, which seems to be useful for us. I am
> >> not completely clear: is photils the engine that is currently used in
> >> Digikam for face recognition with the new AI / DL approach, or is this
> >> in the future?
>
> I looked at Photils and tried it. I think it may be useful but not
> really optimal for our use. We need something that we can train at
> different levels of detail (initially plant /  not plan, then family,
> species, subspecies etc. etc.) , possibly based on Deep Learning.

I don't yet tried Photils as well. the plan is to make a Generic
plugin for digiKam which will use it and propose keywords to tag
images.

>
> >
> > No. digiKam as a dedicated internal engine to process detection and
> > recognition based on AI.
>
> What technology does Digicam use? I assume Digikam is using DL, but
> what starting from which libraries? I assume you did not develop
> everything from the ground up, because there are so many interesting
> libraries available.

Of course, we don't re-invent the wheel. Take a look to the 7.0.0,
there are all technical details, history, and plan explained :

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

>
> >
> > Photils is an alternative which compute a fingerprint of images, send
> > it to a AI engine which try to identify objects, monuments, animals,
> > etc... It return a list of keywords.
>
> I understood the mechanism, and trialled it, and it turns out to be
> too generic for our use. I would prefer something we can train, and
> that can be used at different level of details (initially plant /  not
> plant, then family, species, subspecies etc. etc.).

Yes. I talk with the student working on face engine AI based, and this
plan are possible but complex. The problem is the model of data for
the deep learning engine. 2 solutions : use a pre computed model (the
case currently for the face) or compute the model step by step
depending of the king of form to detect and recognize. This last case
is really more complex and long to implement, test, and validate.

Regards

Gilles Caulier