[digikam] [Bug 320666] New: Fuzzy search for any image

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #18 from Mario Frank <[hidden email]> ---
Created attachment 103203
  --> https://bugs.kde.org/attachment.cgi?id=103203&action=edit
Patch for triggering duplicates search for multiple tags with context menu and
introducing the similarity value as column in table view..

Already made some tests. But more testers are better.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #19 from Wolfgang Scheffner <[hidden email]> ---
(In reply to Mario Frank from comment #18)
> Created attachment 103203 [details]
> Patch for triggering duplicates search for multiple tags with context menu
> and introducing the similarity value as column in table view..
>
> Already made some tests. But more testers are better.

I'd volunteer to test but I'm working with the AppImage version right now. I
guess there is no easy way to apply the patch to that?

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #20 from Mario Frank <[hidden email]> ---
(In reply to Wolfgang Scheffner from comment #19)
> (In reply to Mario Frank from comment #18)
> > Created attachment 103203 [details]
> > Patch for triggering duplicates search for multiple tags with context menu
> > and introducing the similarity value as column in table view..
> >
> > Already made some tests. But more testers are better.
>
> I'd volunteer to test but I'm working with the AppImage version right now. I
> guess there is no easy way to apply the patch to that?

Hey Wolfgang,

No, there is no easy way to do that.
I looked at the AppImage build information. I would have to create an AppImage
and for this, I would have to create a CentOS VM. This would take some time.

Can you compile digikam on your system or do you just use binary versions?

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #21 from [hidden email] ---
Mario,

All VM are ready at home. I can do it this week end including patch, not
before.

Note : remember that 5.4.0 will be release this Sunday evening.

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #22 from Mario Frank <[hidden email]> ---
(In reply to caulier.gilles from comment #21)
> Mario,
>
> All VM are ready at home. I can do it this week end including patch, not
> before.
>
> Note : remember that 5.4.0 will be release this Sunday evening.
>
> Gilles

Hey Gilles,
I tested the patch again yesterday evening. The database handling works as
expected. No errors were there. The patch reduces the amount of DB usage since
the similarity is only stored for duplicates. Fuzzy search, drop search and
sketch search do not use similarities from DB any more. Since these HAAR
searches are done on thumbnail load anyway which also includes searching for
the similar pictures, I just forward the similarities. The current similarity
is stored in the ImageInfo but not persisted in the database. Thus, there are
no DB changes with this approach. I eliminated the storing of the current fuzzy
reference image from application settings since I do not need it any more.
Also,
I use less SIGNAL/SLOT communication which reduces the probability of bugs a
lot. I do not have to react on context switches anymore.

The only things that cannot be solved easily:
1) The column Similarity is shown for the table view everywhere if it was
selected. But only when there is a similarity, the column entry is non-zero.
So, in every table view that is not in the context of the fuzzy search sidebar,
the images have a similarity of 0.0. I do not see a solution to hide that
column dynamically. But since many properties (e.g. geo location) are not set
for every picture, I would not see it as a glitch.
2) In sketch search, the similarity values are in fact the similarity scores
normalised to a positive value between 0 and 100. It is not a percentage.
Nevertheless, this is no dysfunction but intended for now.

Since the patch also fixes bugs like the context menu problems
https://bugs.kde.org/show_bug.cgi?id=374191 , reduces DB usage overhead and
polishes my former SIGNAL/SLOT communication, I would like to merge it as soon
as possible.

What do you think?

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #23 from Wolfgang Scheffner <[hidden email]> ---
(In reply to Mario Frank from comment #20)

> (In reply to Wolfgang Scheffner from comment #19)
> > (In reply to Mario Frank from comment #18)
> >
> > I'd volunteer to test but I'm working with the AppImage version right now. I
> > guess there is no easy way to apply the patch to that?
>
> Hey Wolfgang,
>
> No, there is no easy way to do that.
> I looked at the AppImage build information. I would have to create an
> AppImage and for this, I would have to create a CentOS VM. This would take
> some time.
>
> Can you compile digikam on your system or do you just use binary versions?

I can compile digiKam on my system but I don't know how to include your patch.
If you can give me instructions for that I can try this afternoon/evening.

--
You are receiving this mail because:
You are the assignee for the bug.

[Simon Frei]

@Wolfgang:
You need to download the attached patch to the core directory. There you run
    git apply *patchname*
where *patchname* is the actual name of the patch. Then you continue
building as usual.

I am currently disentangling the changes regarding
https://bugs.kde.org/show_bug.cgi?id=374191 and this bug, because there
is something I would like to suggest on the patch for the former. I will
post the separated pasts as soon as I have them - I hope that is ok.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #24 from Simon <[hidden email]> ---
I am currently disentangling the changes regarding
https://bugs.kde.org/show_bug.cgi?id=374191 and this bug, because there
is something I would like to suggest on the patch for the former. I will
post the separated pasts as soon as I have them - I hope that is ok.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #25 from Mario Frank <[hidden email]> ---
(In reply to Simon from comment #24)
> I am currently disentangling the changes regarding
> https://bugs.kde.org/show_bug.cgi?id=374191 and this bug, because there
> is something I would like to suggest on the patch for the former. I will
> post the separated pasts as soon as I have them - I hope that is ok.

Simon, better take the current patch in this file for disentangling. In this
patch, I have the overloaded functions which are not present in the patch in
https://bugs.kde.org/show_bug.cgi?id=374191 .

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #26 from Mario Frank <[hidden email]> ---
(In reply to Wolfgang Scheffner from comment #23)

> (In reply to Mario Frank from comment #20)
> > (In reply to Wolfgang Scheffner from comment #19)
> > > (In reply to Mario Frank from comment #18)
> > >
> > > I'd volunteer to test but I'm working with the AppImage version right now. I
> > > guess there is no easy way to apply the patch to that?
> >
> > Hey Wolfgang,
> >
> > No, there is no easy way to do that.
> > I looked at the AppImage build information. I would have to create an
> > AppImage and for this, I would have to create a CentOS VM. This would take
> > some time.
> >
> > Can you compile digikam on your system or do you just use binary versions?
>
> I can compile digiKam on my system but I don't know how to include your
> patch. If you can give me instructions for that I can try this
> afternoon/evening.

Hi Wolfgang,

to apply the patch, you have to go into the core directory. There you first
fetch the current state of the master branch with git pull.
Then you test the patch with
git apply --check PatchFilePath
If there are no warnings or errors, you type
git apply --apply PatchFilePath

Then you can go in the build folder and compile the system with make install.

I use an alias to start digikam wich I added to my bashrc:
alias
digikam-dev="KDESYCOCA=/home/eladrion/local/opt/digikam/var/tmp/kde-eladrion/ksycoca5
XDG_DATA_DIRS=/home/eladrion/local/opt/digikam/share:/usr/share:/usr/share:/usr/local/share
QT_PLUGIN_PATH=/home/eladrion/local/opt/digikam/lib64/plugins:/home/eladrion/local/opt/digikam/lib/plugins:
/home/eladrion/local/opt/digikam/bin/digikam"
alias
digikam-dev-valgrind="KDESYCOCA=/home/eladrion/local/opt/digikam/var/tmp/kde-eladrion/ksycoca5
XDG_DATA_DIRS=/home/eladrion/local/opt/digikam/share:/usr/share:/usr/share:/usr/local/share
QT_PLUGIN_PATH=/home/eladrion/local/opt/digikam/lib64/plugins:/home/eladrion/local/opt/digikam/lib/plugins:
valgrind /home/eladrion/local/opt/digikam/bin/digikam"

You will have to adopt the paths for your environment. But these aliases make
it easy to test. The second alias runs digikam under valgrind.

HTH

--
You are receiving this mail because:
You are the assignee for the bug.

[Simon Frei]

On 06/01/17 11:37, Mario Frank wrote:

> https://bugs.kde.org/show_bug.cgi?id=320666
>
> --- Comment #25 from Mario Frank <[hidden email]> ---
> (In reply to Simon from comment #24)
>> I am currently disentangling the changes regarding
>> https://bugs.kde.org/show_bug.cgi?id=374191 and this bug, because there
>> is something I would like to suggest on the patch for the former. I will
>> post the separated pasts as soon as I have them - I hope that is ok.
> Simon, better take the current patch in this file for disentangling. In this
> patch, I have the overloaded functions which are not present in the patch in
> https://bugs.kde.org/show_bug.cgi?id=374191 .

That is what I am doing.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #27 from Simon <[hidden email]> ---
On 06/01/17 11:37, Mario Frank wrote:

> https://bugs.kde.org/show_bug.cgi?id=320666
>
> --- Comment #25 from Mario Frank <[hidden email]> ---
> (In reply to Simon from comment #24)
>> I am currently disentangling the changes regarding
>> https://bugs.kde.org/show_bug.cgi?id=374191 and this bug, because there
>> is something I would like to suggest on the patch for the former. I will
>> post the separated pasts as soon as I have them - I hope that is ok.
> Simon, better take the current patch in this file for disentangling. In this
> patch, I have the overloaded functions which are not present in the patch in
> https://bugs.kde.org/show_bug.cgi?id=374191 .

That is what I am doing.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

Simon <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #103203|0                           |1
        is obsolete|                            |
                 CC|                            |[hidden email]

--- Comment #28 from Simon <[hidden email]> ---
Created attachment 103229
  --> https://bugs.kde.org/attachment.cgi?id=103229&action=edit
Patch for introducing the similarity value as column in table view (part of
103203)

I attached Mario's patch without the sections that relate to
https://bugs.kde.org/show_bug.cgi?id=374191. I checked that it still compiles,
but I have not tested the changes.

--
You are receiving this mail because:
You are the assignee for the bug.

[Simon Frei]

I played around with the patch. However I did not test/use this feature
previously, so I don't know whether the following was introduced with
this patch or already there before:

In "Duplicates" the similarity in table view seem reasonable mostly. But
when there are identical pictures in a group of similar images, there
sometimes is one picture with 100, the other with 101, sometimes just
one picture with 101 and sometimes all values are <100. I don't see any
pattern when this happens. What does the similarity mean anyway in a
group? Average similarity is intuitively understandable, individual
similarity is unclear to me.

In "Images" I get a binary: It is either 100 or 1.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #29 from Simon <[hidden email]> ---
I played around with the patch. However I did not test/use this feature
previously, so I don't know whether the following was introduced with
this patch or already there before:

In "Duplicates" the similarity in table view seem reasonable mostly. But
when there are identical pictures in a group of similar images, there
sometimes is one picture with 100, the other with 101, sometimes just
one picture with 101 and sometimes all values are <100. I don't see any
pattern when this happens. What does the similarity mean anyway in a
group? Average similarity is intuitively understandable, individual
similarity is unclear to me.

In "Images" I get a binary: It is either 100 or 1.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #30 from Mario Frank <[hidden email]> ---
(In reply to Simon from comment #29)

> I played around with the patch. However I did not test/use this feature
> previously, so I don't know whether the following was introduced with
> this patch or already there before:
>
> In "Duplicates" the similarity in table view seem reasonable mostly. But
> when there are identical pictures in a group of similar images, there
> sometimes is one picture with 100, the other with 101, sometimes just
> one picture with 101 and sometimes all values are <100. I don't see any
> pattern when this happens. What does the similarity mean anyway in a
> group? Average similarity is intuitively understandable, individual
> similarity is unclear to me.
>
> In "Images" I get a binary: It is either 100 or 1.

This is odd. First of all:
In "Duplicates", the original picture should be on top when it is sorted by
similarity. The original image is the name of the duplicates album. Thus, it
must have the highest value. That's why the similarity of one picture is always
101. Since a similarity range is given, it is possible, that one picture has
similarity 101 and all others below 100. That's the expected behaviour. What
cannot happen is that some duplicates album has only one entry. The HaarIface
returns an empty list of duplicates if only the original image is present.
Also, it should not be possible that some duplicates album only has images with
similarity less than 100 since the original image is always contained in the
album.

While the average similarity in the duplicates album list represents exactly
this - the average of all similar pictures (without the original), the
similarity in the image view/table view is the similarity of the specific image
to the original one.

Can you attach some screenshots showing the binary "Images" and the case where
all images have a similarity below 100?

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #31 from Simon <[hidden email]> ---
What I described in Duplicates was wrong. The main image was a grouped image,
so it was hidden (http://i.imgur.com/JKOkJx7.png), but it was actually there
(http://i.imgur.com/2d3OFpM.png). If this is an issue at all, it is definitely
a separate one.
I think using a value of 101 is confusing. If one does not know what it
signifies it looks like a bug. I would just assign a similarity of 100. That
way the main image is still at least in the top group, and when the one
potentially above has a similarity of 100, so is anyway identical.

The "binary" problem in the image view looks like this:
http://i.imgur.com/r1dhZLq.png
It really is binary, if I use a range below 50% all images have similarity 0
and a similarity 100 for range above 50%.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #32 from Mario Frank <[hidden email]> ---
(In reply to Simon from comment #31)

> What I described in Duplicates was wrong. The main image was a grouped
> image, so it was hidden (http://i.imgur.com/JKOkJx7.png), but it was
> actually there (http://i.imgur.com/2d3OFpM.png). If this is an issue at all,
> it is definitely a separate one.
> I think using a value of 101 is confusing. If one does not know what it
> signifies it looks like a bug. I would just assign a similarity of 100. That
> way the main image is still at least in the top group, and when the one
> potentially above has a similarity of 100, so is anyway identical.
>
> The "binary" problem in the image view looks like this:
> http://i.imgur.com/r1dhZLq.png
> It really is binary, if I use a range below 50% all images have similarity 0
> and a similarity 100 for range above 50%.

Hey, the grouped image is something that would be some other issue, I think.
Could be something in the imagelister.
I changed the similarity of the original image to 100.00. I agree that this
could be confusing.

The "binary" problem is really strange. I cannot reproduce it.
Can you run digikam from console? When you search for similar images, the
similarities are logged
with the digikam.database: prefix.

e.g.:
digikam.database: 181148 "42.4814%"

Are the values logged also binary?

--
You are receiving this mail because:
You are the assignee for the bug.

[Simon Frei]

The percentages logged to stdout are meaningful (i.e. not binary).

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=320666

--- Comment #33 from Simon <[hidden email]> ---
The percentages logged to stdout are meaningful (i.e. not binary).

--
You are receiving this mail because:
You are the assignee for the bug.