[digiKam-users] database pointed at subset of same files in rearranged folders

[Ty Mayn]

Some of my newbie reaction to digikam is in

https://mail.kde.org/pipermail/digikam-users/2019-February/028282.html

 And I would still welcome elaboration on questions posed there

   It is necessary that I use windows and currently 5.9digikam  It took me awhile to find the lonely digikamrc in  user/appdata/local.  I am testing  without scan for files at startup  and moving the same test files to different devices but in varied folder arrangements.

      if I rename the root filesystem folder of a collection root to hide it from the stored  named configuration collection, I discover that digikam kindly delivers the stored collection of thumbnails complete  with non embedded digikam tags and captions, but of course no larger preview images,  Thats nice  and it tells me that the core datarecords work hand in hand with the thumbnail database up to the point of calling a full image.

     If I repoint the configured collection to the same image files with a few deleted  (as well as rearranged in folders)  the program efficiently shows only those files discovered.  When i cycle back to point to the full collection (rearranged yet again and under a newly named root on a new device)  then I am pleasantly surprised to find that  coredatabase has retained the tags and captions of the replaced missing files discovered in the repointed  collection,  I notice no disc activity to suggest that thumbnails were rebuilt for those recovered images.

    This tells me that the coredatabase saves records of metadata for images parsed in previous collection trees that have been removed. The records are immediately available when the same images (or fingerprint) are encountered in a rearranged collection .The thumbnails database appears to be saving those thumbnails also (unless I missed witnessing the brief lag to rebuild in my test of a few restored images).

    Is my observation correct? ( I like this behavior)

New question:   If I tag a larger collection (1500+) and trouble the processor to create thumbnails, can I do large rearrangements of folders and subfolders at a later time with a regular filemanager ....and... have some reasonable expectation that digikam will find  and matchup its corresponding records for moved rearranged files in a new collection?

   I have only tiny test observations which need confirmation or caution from experienced users.

   Here are my partial observations to old questions in my earlier post

 Can digikam seek out matching files in any folder pathes even with files
distributed across differing folder trees? Yes  existing records and fingerprints are reviewed when loading any new collection. ( Is a duplicate record is placed if the fingerprint matches but the path is new?)
  Is digikam programmed to find and matchup a partial collection of
filenames wherever and whenever encountered? Yes that's loading a new collection.
   If there is such match making to marry data rows to files, then does
that occur only once at the time of parsing out a newly assigned folder
collection, or ongoing at every wakeup of  the program?

    It appears that existing data rows in the core database are  matched to files discovered in an assigned collection only at the time of first configuring a collection..... unless there is configured scan at startup or a manual maintenance scan for new images.  A scan should discover new images that match prior and unloaded database records or entirely new images with no matching database record.

   Thanks to anyone for more detailed answers to the new or old questions posed.

Ty

     

[Maik Qualmann]

I have read your text only in part to understand what it is about. The answer
is no. Whatever you are trying to rename the root folder etc, it will not
work.

1. Digikam is currently not multi OS capable with the same database and
different mount paths. But there is already an untested patch.
2. DigiKam has no rights management. Any user can delete or add tags.
Likewise, it is not possible to hide folders from specific users.

Maik

Am Mittwoch, 20. Februar 2019, 08:30:47 CET schrieb Ty Mayn:

> Some of my newbie reaction to digikam is in
> https://mail.kde.org/pipermail/digikam-users/2019-February/028282.html
>  And I would still welcome elaboration on questions posed there
>    It is necessary that I use windows and currently 5.9digikam  It took me
> awhile to find the lonely digikamrc in  user/appdata/local.  I am testing
> without scan for files at startup  and moving the same test files to
> different devices but in varied folder arrangements.
>       if I rename the root filesystem folder of a collection root to hide
> it from the stored  named configuration collection, I discover that digikam
> kindly delivers the stored collection of thumbnails complete  with non
> embedded digikam tags and captions, but of course no larger preview
> images,  Thats nice  and it tells me that the core datarecords work hand in
> hand with the thumbnail database up to the point of calling a full image.
>      If I repoint the configured collection to the same image files with a
> few deleted  (as well as rearranged in folders)  the program efficiently
> shows only those files discovered.  When i cycle back to point to the full
> collection (rearranged yet again and under a newly named root on a new
> device)  then I am pleasantly surprised to find that  coredatabase has
> retained the tags and captions of the replaced missing files discovered in
> the repointed  collection,  I notice no disc activity to suggest that
> thumbnails were rebuilt for those recovered images.
>     This tells me that the coredatabase saves records of metadata for
> images parsed in previous collection trees that have been removed. The
> records are immediately available when the same images (or fingerprint) are
> encountered in a rearranged collection .The thumbnails database appears to
> be saving those thumbnails also (unless I missed witnessing the brief lag
> to rebuild in my test of a few restored images).
>     Is my observation correct? ( I like this behavior)
> New question:   If I tag a larger collection (1500+) and trouble the
> processor to create thumbnails, can I do large rearrangements of folders
> and subfolders at a later time with a regular filemanager ....and... have
> some reasonable expectation that digikam will find  and matchup its
> corresponding records for moved rearranged files in a new collection?
>    I have only tiny test observations which need confirmation or caution
> from experienced users.
>    Here are my partial observations to old questions in my earlier post
>  Can digikam seek out matching files in any folder pathes even with files
> distributed across differing folder trees? Yes  existing records and
> fingerprints are reviewed when loading any new collection. ( Is a duplicate
> record is placed if the fingerprint matches but the path is new?)
>   Is digikam programmed to find and matchup a partial collection of
> filenames wherever and whenever encountered? Yes that's loading a new
> collection.
>    If there is such match making to marry data rows to files, then does
> that occur only once at the time of parsing out a newly assigned folder
> collection, or ongoing at every wakeup of  the program?
>     It appears that existing data rows in the core database are  matched to
> files discovered in an assigned collection only at the time of first
> configuring a collection..... unless there is configured scan at startup or
> a manual maintenance scan for new images.  A scan should discover new
> images that match prior and unloaded database records or entirely new
> images with no matching database record.
>    Thanks to anyone for more detailed answers to the new or old questions
> posed.
> Ty

[Ty Mayn]

thanks for clarifying that there is no rights management. And regarding the database I am examining its behavior during repointing.

   One focus as a new user is understanding how images may be found again by the database when those images are rearranged into new folder patterns.

   My testing is with about 15 image files in about 5 or 6 folders, some as subfolders. Each time of hiding the folder collection by a rename of path its clear that the database delivers its digikam metadata by display with the thumbnails even if the real file path of the image files has been obscured

    I have taken the small test database (15 files) and upgraded to Digikam6 and loaded a much larger collection of a few hundred files in about 10 folders.....knowing that the originals of those 15 files are waiting there to be found on the first  intake of the larger collection. 

   Sure enough the programming is designed to cycle through a matchmaking of the existing files as they are discovered  in the new larger collection. All prior tags for those 15 test files are preserved and displayed.   I wonder if the program even skips the brief effort to generate a new thumbnail for those known preexisting records that point to a thumbnail.  Presumably this is all based on discoveriing  a suitable identical fingerprint for the file based on name, dates,filesize and a hash value and loading transferring the metadata to the new record identifier (for all I know the same database record number may be retained).

     I am not a programmer and I dont know what to call this preservation instinct that is built into digikam.  Some programmer could have just as easily designed to throw away all records and metadata when unloading a collection or when loading a new collection that is missing some of the formerly mapped image files.

    I have seen preservation behavior even where an old collection is hidden by renaming, then unloaded, then replaced by loading a different collection with absent files, then pointing to a new pattern of folders and files that returns the missing files for inclusion ( rearranged folders).

     My reason for describing this is to learn what experienced users call this and to know how dependable the behavior is.    I dont see this behavior outlined in the digikam user manual which is excellent but finite in scope (an undated work in progress AFAIK) 

    If I were naming this "feature"   I would call it a database with "unused record retention" and look upon it as an asset rather than liability.  It deserves a warning in regards to what command actually purges unused records (presumably by the maintenance tool) It also deserves comment to know if there is any non manual trigger that kills surviving records that are not in use in a currently loaded collection.

       I would cite the collection loading tools as  having  "record repurposing by fingerprint"  for  "new collections with intersection/overlap  of former record identities" . . It is clear to me that a prior database pointed at a new image collection can have some inheritance of metadata based on a matchmaking design.    This could benefit from explanation of the behavior by programmers who know the design.  A big question in this is whether the main database is purged automatically (non manually) when certain events occur .... such as reloading the thumbnail database.

    Users dont make any parts of the plane, each   just tries to understand the ways it can be piloted.

Ty

    

On Thu, Feb 21, 2019 at 6:27 AM Maik Qualmann <[hidden email]> wrote:

I have read your text only in part to understand what it is about. The answer
is no. Whatever you are trying to rename the root folder etc, it will not
work.

1. Digikam is currently not multi OS capable with the same database and
different mount paths. But there is already an untested patch.
2. DigiKam has no rights management. Any user can delete or add tags.
Likewise, it is not possible to hide folders from specific users.

Maik

Am Mittwoch, 20. Februar 2019, 08:30:47 CET schrieb Ty Mayn:
> Some of my newbie reaction to digikam is in
> https://mail.kde.org/pipermail/digikam-users/2019-February/028282.html
>  And I would still welcome elaboration on questions posed there
>    It is necessary that I use windows and currently 5.9digikam  It took me
> awhile to find the lonely digikamrc in  user/appdata/local.  I am testing
> without scan for files at startup  and moving the same test files to
> different devices but in varied folder arrangements.
>       if I rename the root filesystem folder of a collection root to hide
> it from the stored  named configuration collection, I discover that digikam
> kindly delivers the stored collection of thumbnails complete  with non
> embedded digikam tags and captions, but of course no larger preview
> images,  Thats nice  and it tells me that the core datarecords work hand in
> hand with the thumbnail database up to the point of calling a full image.
>      If I repoint the configured collection to the same image files with a
> few deleted  (as well as rearranged in folders)  the program efficiently
> shows only those files discovered.  When i cycle back to point to the full
> collection (rearranged yet again and under a newly named root on a new
> device)  then I am pleasantly surprised to find that  coredatabase has
> retained the tags and captions of the replaced missing files discovered in
> the repointed  collection,  I notice no disc activity to suggest that
> thumbnails were rebuilt for those recovered images.
>     This tells me that the coredatabase saves records of metadata for
> images parsed in previous collection trees that have been removed. The
> records are immediately available when the same images (or fingerprint) are
> encountered in a rearranged collection .The thumbnails database appears to
> be saving those thumbnails also (unless I missed witnessing the brief lag
> to rebuild in my test of a few restored images).
>     Is my observation correct? ( I like this behavior)
> New question:   If I tag a larger collection (1500+) and trouble the
> processor to create thumbnails, can I do large rearrangements of folders
> and subfolders at a later time with a regular filemanager ....and... have
> some reasonable expectation that digikam will find  and matchup its
> corresponding records for moved rearranged files in a new collection?
>    I have only tiny test observations which need confirmation or caution
> from experienced users.
>    Here are my partial observations to old questions in my earlier post
>  Can digikam seek out matching files in any folder pathes even with files
> distributed across differing folder trees? Yes  existing records and
> fingerprints are reviewed when loading any new collection. ( Is a duplicate
> record is placed if the fingerprint matches but the path is new?)
>   Is digikam programmed to find and matchup a partial collection of
> filenames wherever and whenever encountered? Yes that's loading a new
> collection.
>    If there is such match making to marry data rows to files, then does
> that occur only once at the time of parsing out a newly assigned folder
> collection, or ongoing at every wakeup of  the program?
>     It appears that existing data rows in the core database are  matched to
> files discovered in an assigned collection only at the time of first
> configuring a collection..... unless there is configured scan at startup or
> a manual maintenance scan for new images.  A scan should discover new
> images that match prior and unloaded database records or entirely new
> images with no matching database record.
>    Thanks to anyone for more detailed answers to the new or old questions
> posed.
> Ty

[Remco Viëtor]

On vendredi 22 février 2019 23:49:25 CET Ty Mayn wrote:
(...)
>     I have seen preservation behavior even where an old collection is
> hidden by renaming, then unloaded, then replaced by loading a different
> collection with absent files, then pointing to a new pattern of folders and
> files that returns the missing files for inclusion ( rearranged folders).
(...)
Collections on a network or removable drive can be temporarily unavailable
(through accident or design). Throwing away the database records for such
collections would be, let's say, unfortunate, especially when it concerns raw
files (which do not contain user-added keywords, captions etc.)

Remco

[Ty Mayn]

 Yes I find that digikam is designed for this bias toward retention of data.

A new user has to discover behavior the behavior which is understood second nature to experience.  So I see that disconnected network locations (and obscured,moved,or even deleted files and folders in my testing) do not trigger any destruction of database records.

This tiny explicit info is not obvious in support materials.

     I am now still on guard to the ways that records do become lost unintentially during file system rearrangements.

     Am i correct to think that the only actual deletion of a database record (ever) is under Maintenance>PerformDatabaseCleansing?   In this case I am also wary to not have a casual check sitting in the box "WholeAlbumsCollection"  when needing to cleanse just one collection.

    Let me clarify the meaning of "scan" also and be sure that a scan does not lose database records ..... is it fair to say it can only add records??  Scan is one of those broad terms like synchronize when one must ask "which direction" "two way or one way" or  what is the input template and what is the output event.

      Is it fair to say that under Maintenance >"scan for new items" depends on the matchmaking features of digikam and it is a one way additive synchronization in which the database is only added to with NO records ever being removed.  The Maintenance graphic/text menu has lots of white space to make this explicit for users. Generally menu screen space is often an  underused teaching and documentation tool.

    Adding a collection is an action with  special features in which a new collection can incorporate one or more unused records transferred from former regions of the database (via a matchmaking protocol). The new region defining the new collection however is not the carrier for unused records which do not qualify for inclusion.I will call my programming an infancy unchanged from a lifetime ago, so  I dont know how to name and answer question by reviewing code, 

Empirically the database seems to be managed so that once a record is in it stays in until manually cleansed.  A rescan for new images in a collection at least does a matchmaking against the records which define that collection.  But the process setting up a new collection is intriguing.  It seems  a new collection searches out a new template of file and folder patterns but matches file fingerprints against existing records that can reside in other collections (or at least records that are unused and not assigned to other collections). A future question for study is whether by fingerprint 2 collections can carry the same image item and associated metadata (nonembedded)

Experienced users might  point out to me how I am off on an imaginary limb or some may provide the words that shows me how the limb is rightly described and a logical part of the whole tree and I will continue climbing with confidence.

Ty

    

On Sat, Feb 23, 2019 at 2:40 AM Remco Viëtor <[hidden email]> wrote:

On vendredi 22 février 2019 23:49:25 CET Ty Mayn wrote:
(...)
>     I have seen preservation behavior even where an old collection is
> hidden by renaming, then unloaded, then replaced by loading a different
> collection with absent files, then pointing to a new pattern of folders and
> files that returns the missing files for inclusion ( rearranged folders).
(...)
Collections on a network or removable drive can be temporarily unavailable
(through accident or design). Throwing away the database records for such
collections would be, let's say, unfortunate, especially when it concerns raw
files (which do not contain user-added keywords, captions etc.)

Remco

[Andrew Goodbody]

On 23/02/2019 18:36, Ty Mayn wrote:
>       Am i correct to think that the only actual deletion of a database
> record (ever) is under Maintenance>PerformDatabaseCleansing?

No. Records will eventually be deleted after a period of, I think, two
weeks, if the target is no longer available. It is a long enough period
that if you have a short term network failure, it will not immediately
remove the data, but also it will not keep records around for ever if
the files were deleted outside of digikam.

I would guess that the scan also records those files that are not
available. I am not sure what would eventually trigger the removal of
the records once the time above has expired, but that could be the scan
or it could be another house keeping process running periodically.

Andrew

[Ty Mayn]

thank you for this caution.  I view the retention of records that dont map to a current loaded collection to be an asset and I would expect the maintenance purge to be sufficient.

I would hope an automatic purge at some arbitrary time (such as 2 weeks that you estimate) should have a dialog message to the user letting it be known that the fateful hour has arrived  for removal. Somehow carrying that out silently after some wait seems the worst of both worlds. It is better to force an unmatched record to  the surface for the user to attend to it or just keep it on standby 

 I would welcome just a statistics page that would state  sequentially each of the named collections with load and unload (or disconnect) dates and a count of the retained records  that are unused in relation to those old collection pathways and furthermore unmatched into the current loaded collections. Then the user is aware and can  cleanse at will.

    It is interesting that cleanse has a toggle to be either album or tag based. Albums are folder trees essentially and in that sense they can be made to align with past named collections. But it would be nice to have a Maintenance menu which can be collection centric...meaning that it can offer a list of past loaded collections and allow the folder tree for each collection to be retrieved and cleansed independently. Overlaps of successive named collections could be problematic.  Maybe I am speaking too soon (or too late) if such tool already exists somewhere.

    On the matter of what a scan does .... I would welcome a programmers observation. As I have stated in this thread it appears to be an additive only process that looks for new file items in the album tree and adds records, but not  removing any record that has become unrepresented in the file image collection.   The  menu says "scan for new items" insinuating that it is on the lookout for new incoming images and  the cleanse tool is right below.

     As I have stated in this thread I believe these "behaviors" need to be named and explicit and they should be displayed  in the text menus where there is ample  whitespace available.

      Although I would like to know the firm answer to these questions via maillist, I would like to see a sort of enhanced Menu which divulges larger hints about the consequences of powerful actions sitting behind the checkboxes.  Some new users are in a candy store and others are just surrounded by fear or at least curiosity.

Ty

  

  

On Sun, Feb 24, 2019 at 1:48 PM Andrew Goodbody <[hidden email]> wrote:

On 23/02/2019 18:36, Ty Mayn wrote:
>       Am i correct to think that the only actual deletion of a database
> record (ever) is under Maintenance>PerformDatabaseCleansing?

No. Records will eventually be deleted after a period of, I think, two
weeks, if the target is no longer available. It is a long enough period
that if you have a short term network failure, it will not immediately
remove the data, but also it will not keep records around for ever if
the files were deleted outside of digikam.

I would guess that the scan also records those files that are not
available. I am not sure what would eventually trigger the removal of
the records once the time above has expired, but that could be the scan
or it could be another house keeping process running periodically.

Andrew