Database separation for similarity searches

[Gilles Caulier-4]

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

[Gilles Caulier-4]

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

[Mario Frank]

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier <[hidden email]>:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Mario Frank]

Hey Gilles,

I looked at the code yesterday and it is not yet finished. I will apply some changes to improve performance and report on it before committing.

For which release is the db separation planned? 5.9 or 6.0?

Best

Mario

Am 06.01.2018 um 17:43 schrieb Mario Frank:

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier [hidden email]:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Maik Qualmann]

Hi Mario,

the 5.9.0 master branch is only for minor bug fixes.
Commit these changes to the 6.0.0 branch.

Maik

Am Montag, 22. Januar 2018, 16:55:25 CET schrieb Mario Frank:

> Hey Gilles,
>
> I looked at the code yesterday and it is not yet finished. I will apply
> some changes to improve performance and report on it before committing.
>
> For which release is the db separation planned? 5.9 or 6.0?
>
> Best
>
> Mario
>
> Am 06.01.2018 um 17:43 schrieb Mario Frank:
> > Hey Gilles,
> >
> > I think she was not completely finished. I will have time for digiKam
> > again after this weekend. I will take a look. I've got unfinished work
> > anyway.
> >
> > Cheers
> > Mario
> >
> > Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier
> >
> > <[hidden email]>:
> >     Note : with this commit, i sync this branch with master without
> >     conflict.
> >    
> >     https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d15
> >     51
> >    
> >     Gilles
> >    
> >     2018-01-06 16:13 GMT+01:00 Gilles Caulier
> >    
> >     <[hidden email] <mailto:[hidden email]>>:
> >         Hi all,
> >        
> >         This summer, Swaty has work on database separation for
> >         similarity searches. The goal was to separate the fuzzy
> >         fingerprints storage from core DB to a dedicated DB. This will
> >         simplify the backup and prevent to overload the core DB with
> >         large data by image (around 500 bytes by fingerprints)
> >        
> >         The work is not completed as i know. All have be done
> >         in origin/gsoc17-db-sep branch from git repository.
> >        
> >         What's the status of this code exactly ? Can we continue with
> >         this implementation, or it will be better to re-start this
> >         work from scratch ?
> >        
> >         The DB separation is important for the future. Thank sin
> >         advance for your feedback.
> >        
> >         Best
> >        
> >         Gilles

[Gilles Caulier-4]

Hi Mario,

for digiKam 6.0.0 definitively. I sync the branch with master step by step. It's fine to merge db-sep branch to 6.0.0 branch that i created 2 week ago.

Gilles

2018-01-22 16:55 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I looked at the code yesterday and it is not yet finished. I will apply some changes to improve performance and report on it before committing.

For which release is the db separation planned? 5.9 or 6.0?

Best

Mario

Am 06.01.2018 um 17:43 schrieb Mario Frank:

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier [hidden email]:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Mario Frank]

Hey Gilles,

I got the code working finally, yesterday evening. It now compiles but the similarity db is not yet initialised correctly on startup.

There were far more changes and extensions necessary than I expected. So far about the current status.

Mario

Am 22.01.2018 um 19:51 schrieb Gilles Caulier:

Hi Mario,

for digiKam 6.0.0 definitively. I sync the branch with master step by step. It's fine to merge db-sep branch to 6.0.0 branch that i created 2 week ago.

Gilles

2018-01-22 16:55 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I looked at the code yesterday and it is not yet finished. I will apply some changes to improve performance and report on it before committing.

For which release is the db separation planned? 5.9 or 6.0?

Best

Mario

Am 06.01.2018 um 17:43 schrieb Mario Frank:

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier [hidden email]:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Gilles Caulier-4]

Ok,

I just sync the branch with master. At least, the branch is up to date, as 6.0.0. this will be more easy to merge in 6.0.0 later.

Gilles

2018-01-25 17:08 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I got the code working finally, yesterday evening. It now compiles but the similarity db is not yet initialised correctly on startup.

There were far more changes and extensions necessary than I expected. So far about the current status.

Mario

Am 22.01.2018 um 19:51 schrieb Gilles Caulier:

Hi Mario,

for digiKam 6.0.0 definitively. I sync the branch with master step by step. It's fine to merge db-sep branch to 6.0.0 branch that i created 2 week ago.

Gilles

2018-01-22 16:55 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I looked at the code yesterday and it is not yet finished. I will apply some changes to improve performance and report on it before committing.

For which release is the db separation planned? 5.9 or 6.0?

Best

Mario

Am 06.01.2018 um 17:43 schrieb Mario Frank:

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier [hidden email]:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Mario Frank]

Hey Gilles,

I completed the adoptions yesterday and just pushed the changes to the db-sep branch.

I tried to describe still necessary or potentially necessary changes in the commit message.

As I am no specialist in MySQL, especially the dbconfig file should be revised by someone who

is more experienced with MySQL. Also, the Core DB schema will have to be adopted in a new revision

as the ImageHaarMatrix is not used in CoreDB anymore.

I tested the changes with SQLite which works fine.

Here is the commit, sadly, quite large commit:

https://cgit.kde.org/digikam.git/commit/?h=gsoc17-db-sep&id=e4dc5b9260ad771ed1dbae33c65c41eb50906b3e

Now that we have a separate similarity database, the fuzzy search can be boosted enormously.

For this, I have to store the calculated similarities in the database so I can reuse them.

So, there will be some adoptions on haariface for 6.0.0.

Cheers,

Mario

Am 25.01.2018 um 17:46 schrieb Gilles Caulier:

Ok,

I just sync the branch with master. At least, the branch is up to date, as 6.0.0. this will be more easy to merge in 6.0.0 later.

Gilles

2018-01-25 17:08 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I got the code working finally, yesterday evening. It now compiles but the similarity db is not yet initialised correctly on startup.

There were far more changes and extensions necessary than I expected. So far about the current status.

Mario

Am 22.01.2018 um 19:51 schrieb Gilles Caulier:

Hi Mario,

for digiKam 6.0.0 definitively. I sync the branch with master step by step. It's fine to merge db-sep branch to 6.0.0 branch that i created 2 week ago.

Gilles

2018-01-22 16:55 GMT+01:00 Mario Frank <[hidden email]>:

Hey Gilles,

I looked at the code yesterday and it is not yet finished. I will apply some changes to improve performance and report on it before committing.

For which release is the db separation planned? 5.9 or 6.0?

Best

Mario

Am 06.01.2018 um 17:43 schrieb Mario Frank:

Hey Gilles,

I think she was not completely finished. I will have time for digiKam again after this weekend. I will take a look. I've got unfinished work anyway.

Cheers
Mario

Am 6. Januar 2018 16:16:37 MEZ schrieb Gilles Caulier [hidden email]:

Note : with this commit, i sync this branch with master without conflict.

https://commits.kde.org/digikam/d8511b14fd3e9c3406357858f557bdefa03d1551

Gilles

2018-01-06 16:13 GMT+01:00 Gilles Caulier <[hidden email]>:

Hi all,

This summer, Swaty has work on database separation for similarity searches. The goal was to separate the fuzzy fingerprints storage from core DB to a dedicated DB. This will simplify the backup and prevent to overload the core DB with large data by image (around 500 bytes by fingerprints)

The work is not completed as i know. All have be done in origin/gsoc17-db-sep branch from git repository.

What's the status of this code exactly ? Can we continue with this implementation, or it will be better to re-start this work from scratch ?

The DB separation is important for the future. Thank sin advance for your feedback.

Best

Gilles

--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.