[digikam] [Bug 323718] New: Rebuild all thumbnails does not get rid of all thumbnails first

[Gerard Dirkse]

https://bugs.kde.org/show_bug.cgi?id=323718

            Bug ID: 323718
           Summary: Rebuild all thumbnails does not get rid of all
                    thumbnails first
    Classification: Unclassified
           Product: digikam
           Version: 3.2.0
          Platform: openSUSE RPMs
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: Maintenance
          Assignee: [hidden email]
          Reporter: [hidden email]

Noticed that I had more then 4 times as many thumbnails as I have images.
Looking in Filepaths and CustomeIdentifiers I see reference to old file names,
so decided to do a FULL thumbnail rebuild, in the expectation that it would
delete everything in in the thumbs database and then start rebuilding it.
After rebuild still more then 4 times as many thumbnails as images and still
reference to old filenames.

Reproducible: Always

Steps to Reproduce:
1. Have images on a NFS mounted share,
2. Change the mount point and update AlbumRoots in digikam DB
3. Start up Digikam
4. There will now be thumbnails which still have old mount point in filepath,
and there will be lots more thumbnails then there are images.

Actual Results:  
Rebuild all thumbnails has NO effect.  Old rubbish remains in thumbnail DB

Expected Results:  
Clean thumbnail DB with as many thumbnails as I have images.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Gilles Caulier <[hidden email]> ---
Probably your DB file have been corrupted. I never reproduce this problem here.
I currently work on Maintenance tool to support Multicore CPU. I run a lots of
test on huge image collection...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerard Dirkse]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #2 from [hidden email] <[hidden email]> ---
1) So why do I have 4 times the number of thumbnails then I have images ? (may
have been caused by my renaming actions)
2) Why does rebuild ALL thumbnails not first delete COMPLETE content of tables
in thumbs ? (that would solve it).

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #3 from Gilles Caulier <[hidden email]> ---
1) So why do I have 4 times the number of thumbnails then I have images ? (may
have been caused by my renaming actions)

Or it's versionning feature. If you edit and save as new version, a new file is
created and show as current version of item in icon-view. All others previous
version are cached from icon-view, excepted if you turn off right option from
Setup dialog.

For each version file, one thumbnail is created...

2) Why does rebuild ALL thumbnails not first delete COMPLETE content of tables
in thumbs ? (that would solve it).

It must. item deletion in DB is performed item by item in fact.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerard Dirkse]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #4 from [hidden email] <[hidden email]> ---
Very rarely I use versioning, 9 out of 10 times, I choose overwrite existing
version, so that does not explain the more then 4 times number of thumbnails
then there are images.

Browsing (using phpMyAdmin, DB is in MYSQL) the thumbs DB before and after the
rebuild action in tables Filepaths and Customidentifiers (field path and
identifier respectively) I see references to files that don't exist any more,
either as a result moving the NFS mount point, but also from files that have
been renamed using the digikam image rename option.

I would have expected rebuild ALL thumbnails to start with with a 'DELETE *
from ..' each and every table in the thumbs DB and start repopulating them as a
result of the rebuild. That will leave a clean thumbs DB after a rebuild ALL.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerard Dirkse]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #5 from [hidden email] <[hidden email]> ---
You say 'It must. item deletion in DB is performed item by item in fact.', that
is then the bug, because all the cases I mention, i.e. where image with that
name no longer exists will not get deleted.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #6 from Gilles Caulier <[hidden email]> ---
Marcel,

There is a way to clean up thumbs DB before to rebuild all thumbnails ?

Currently, i use this method :

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/utilities/maintenance/thumbstask.cpp#L80

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[monajani]

its seems a nice forum
great discussion


thanks for sharing

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #7 from Gilles Caulier <[hidden email]> ---
Marcel,

Do you see my previous comment ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerard Dirkse]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #8 from [hidden email] <[hidden email]> ---
In the meantime I developed some php programs to go through the database of
images and thumbnails and eliminated almost 75% of my number and size of
thumbnails.  Greatest gain was achieved by using the Uniquehashes table to
eliminate all entries and associated thumbnails from this table where there was
no uniquehas/filesize combination in the images table.  Havent figured out yet
were all these obsolete entries cam from.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #9 from Marcel Wiesweg <[hidden email]> ---
Gilles, would you simply like to clean out all thumbnails? In SQL, that's
simply "DELETE FROM Tumbnails" to delete all thumbnail data. The trigger should
clean the rest of the tables.
If we want a sort of garbage collector, it would need to be something along
what Gerard has developed, checking that a uniqueHash/filesize identifier from
the albumDB still exists in the main database.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #10 from Gilles Caulier <[hidden email]> ---
Marcel

as you can see in code from thumbnailtask.cpp:line 84 :

d->catcher->thread()->deleteThumbnail(d->path);

We only delete valid previous file registered in DB. It do not clear all other
dummy entries.

A garbage collector can be a powerful tool to prevent to rebuild all items. but
for each album to process, we can clean all items, including all garbage
entries... this can be most simple to implement. There is no method implemented
in this way currently. Right ?

Note : the real question here is why garbage entries are present in DB. When an
item is removed or disappear, thumb is DB is not removed automatically ?

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #11 from Marcel Wiesweg <[hidden email]> ---
Thumbnails are primarily loaded via hash/file size. So in principle, whenever
file contents or file size change, there can be a leftover entry in the
database. These can only be found via "go through thumbnail db -> check if it
exists in main db".
Today, when digikam changes file content, the thumbnail reuse/replacement is
often managed, but probably not from all places. It cannot be managed when an
external tool does the change. So a classical case for a garbage collector.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Maintenance                 |Database

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Rebuild all thumbnails does |THUMBDB : Rebuild all
                   |not get rid of all          |thumbnails does not get rid
                   |thumbnails first            |of all thumbnails first

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|THUMBDB : Rebuild all       |THUMBDB : rebuild all
                   |thumbnails does not get rid |thumbnails does not get rid
                   |of all thumbnails first     |of all thumbnails first

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=323718

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Database                    |Database-Thumbs

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=323718

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #12 from [hidden email] ---
Is the file still valid using digiKam 5.1.0?
Please test and provide necessary updates.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=323718

--- Comment #13 from [hidden email] ---
Can you reproduce the problem using digiKam Linux AppImage bundle ? The last
bundle is available at this url:

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=323718

Mario Frank <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #14 from Mario Frank <[hidden email]> ---
There is a patch that introduces garbage collection as maintenance stage before
thumbnail rebuild here: https://bugs.kde.org/show_bug.cgi?id=374591 .
This could solve your problem. But be advised: The patch is still in testing
phase. So, backup your databases before you test.

--
You are receiving this mail because:
You are the assignee for the bug.