[Bug 267131] New: Digikam config and data cannot be moved, absolute paths embedded in database and digikamrc

https://bugs.kde.org/show_bug.cgi?id=267131

           Summary: Digikam config and data cannot be moved, absolute
                    paths embedded in database and digikamrc
           Product: digikam
           Version: 1.8.0
          Platform: Unlisted Binaries
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: NOR
         Component: Database
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]


Version:           1.8.0 (using Devel)
OS:                Linux

I moved my KDE configs and my photos from /home/foo to /home/bar.  Most KDE
programs allow this without hacking

Digikam records the path to the database absolutely in digikamrc and the paths
to the albums in the AlbumRoots table of its database.

I had to hand-edit digikamrc and the sqlite database to allow digikam to start
up without a database not found error, and then to find the moved albums.

Reproducible: Didn't try




Recommend encoding paths involving current $HOME as $HOME instead of the
aboslute path, then expand these, allowing config and data to be migrated
between homes.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #1 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-04-18 22:05:48 ---
For the config, this can be done by using KConfigGroup's read/writePathEntry. I
dont see a problem here.

Inside the database there may be some complications due to the way we refer to
a harddisk by uuid + path. I see the value of using $HOME though.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=267131


Ilia K. <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #2 from Ilia K. <mail4ilia gmail com>  2011-05-02 04:57:28 ---
Hi,
sorry for the lengthy post, but I tried to be constructive and not only state
a problem, but propose a viable solution. What do you, developers, think?

This issue may also break the following use case (I actually need this!):
Alice and Bob use shared MySQL server for digikam database. The files
themselves are shared over NFS, but Alice mounts them on her computer as
/home/alice/Pictures and Bob on his computer as /mnt/photoes.

Other affected use cases may be thought of: DB & files on mobile HDD, backup
restoration, mount point move/rename, etc.

IMHO, DB should contain all file paths relative to each collection root,
e.g. use UUID as a first element of the name, e.g.
/mnt/photoes/album/file.jpg
becomes
a1a88989-5fb5-4621-b539-522f5aa02ff1/album/file.jpg

The question is where to store the paths for collection roots. For above
usecase, it seems logical to store them in local config.

There is a broader issue of guessing that different absolute file names refer
to the same object:

Alice configures her digikam and adds photos (DB records are created). Now Bob
wants to access the same files and DB entries. Although he knows path to the
collection, how can he know UUID? A possible solution: digikam writes uuid.txt
in a collection root with collection UUID, but I don't like it. What if
collection is read-only? What if Bob now uses his own DB?

Maybe, a better solution: allow to query collection list from DB and present
it to the user along with a descriptive name and probably a path hints (last
two are given by the user who already configured an access to the
collection). So collection settings tab can look like the following mockup.
ASCII tables and their headers are used just to signify the layout. Legend and
behavior are described in the next comment (presumably, comment #3 ).

---------------------------------------------------------------------------
Below are the locations of root albums used to store image files.
Write access is necessary to be able to edit images in these albums.
(Removable media and remote file systems are supported)

You have configured the following collections:
| Type  | Name         | Path                  | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Bob Photos   | /home/bob/Pictures    | [Del]       |

----

Other users of your database have configured the following collections.
To access these collections you will need to provide a path to image files
from your computer.

| Type  | Name         | Possible Path(s)      | Action      |
|-------+--------------+-----------------------+-------------|
| [LOC] | Alice Photos | /home/alice/Pictures  | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [MED] | NIKON D90    | /media/NIKON D90      | [Add] [Del] |
|-------+--------------+-----------------------+-------------|
| [NET] | Archive      | /mnt/server/photos    | [Add] [Del] |
|       |              | /media/archive/photos |             |
|-------+--------------+-----------------------+-------------|

----

[Add new collection...]
---------------------------------------------------------------------------

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #3 from Ilia K. <mail4ilia gmail com>  2011-05-02 04:58:21 ---
cont. of comment #2

Legend for the ASCII mockup:
- [LOC] Icon for local collection + a tooltip:
       "Local Collection, directly stored on your computer hard disk"
- [MED] Icon for removable media collection + a tooltip:
       "Collection on Removable Media, such as CD, DVD and USB drives"
- [NET] Icon for network shares collection + a tooltip:
        "Collection on Network Shares, such as NFS and Samba/CIFS"
- [Add] add button (e.g. a standard green plus icon or the word "Add"),
        a tooltip: "Add this collection to your Digikam"
- [Del] delete button (e.g. a standard red X icon, or the word "Delete"),
        a tooltip: "Remove this collection from the database.
        This will not remove the files, but this collection will disappear
        from your Digikam and Digikam of *all other users* which use this
        database."
- [Add new collection...] a button with the obvious purpose

When there are no collections in the second table the whole table along with
the preceding description can be replaced by a simple (probably, grayed out)
message: "No other collections are configured in your database".

The interaction with the dialog.

[Add new collection...]
starts a dialog/wizard of the same name which allows user to specify:
1. Path
2. Type
3. Short name
After choosing a path, short name and type can be guessed by digikam, but
allowed to be changed.

[Add]
starts the same dialog/wizard as [Add new collection...], but short name is
not editable and type is preselected to [NET] or [MED] according to DB data.
A hint of paths at other user's machines is displayed. Possible addition: "try
this path" button near each item to choose the same path locally.
Anyway, after user chooses a local directory some sanity check may be done in
order to ensure consistency between DB and the chosen path.

[Del]
opens a confirmation dialog.
If collection to be deleted is locally bound and there are other DB users,
allow 3 options:
  1. Delete from this Digikam only
  2. Delete from this Digikam and Digikam of all other users (delete from DB)
  3. Cancel
If collection is not used locally, display options 2 & 3 only.
If DB is not shared, display options 1 & 3 only.

The above dialog mockup illustrates the following situation.
- Alice, Bob and Carol share the same DB.
- Alice has added her local ~/Pictures
- Alice or Carol has added removable card mounted at /media/NIKON D90
- Alice has added network share mounted at /mnt/server/photos
- Card has added the same network share mounted at his computer at
  /media/archive/photos
- Bob has added his local ~/Pictures
- Bob opened the said dialog

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #4 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-05-03 22:12:25 ---
We must take care not to mix local hardware and network storage. Network
storage is not well supported, and we dont have good support from the libraries
to make this easy. It's more or less a hack.
When you say UUID you mean a hardware partition. You can share it by physically
moving a USB hard disk, but it does not apply to network storage. So it's
always more or less local. We need some different identifier for network
storage, which we dont have atm, see above.
I'm not sure you are aware that all normal collections are stored by UUID +
relative path in the db? That's the problem with the original report here, it's
not easily applicable depending on partition layout.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #5 from Ilia K. <mail4ilia gmail com>  2011-05-04 03:14:10 ---
Be UUID I actually meant any unique string used to replace path to collection
in a stored file path. For example instead of
"/home/user/mycollection/album1/file.jpg"
one could store
"XXXX/album1/file.jpg"

Later (while fiddling with bug #272301 ) I discovered you made it even better
by storing relative path only ("/album1/file.jpg") along with a collection key
pointing to the relevant collection in AlbumRoots. Great!

However, this hasn't solved the problem for digikam thumbnails database. It
still stores the absolute paths. It's probably better to store relative path
there as well and make the client (digikam) prefix it by a collection
root. Since this information is stored in another database, it's probably
better to refer to it by some long identifier (e.g. random UUID) instead of
just a number (AlbumRoots primary key id). Being unique, AlbumRoots.identifier
may suit this. So a new AlbumRootsRef table may be added to thumbnails DB with
two columns only: int id, longtext reference. The later will hold
AlbumRoots.identifier or alike and the former may be referenced in FilePaths
table like it's done now in Albums table in the main DB.

But thumbnails DB is not the most important issue here. Currently AlbumRoots
allows only one path to collection root (specificPath). But two different
users may have the same collection under different paths. In DB layer this can
be handled this way for example:
- remove AlbumRoots.specificPath column
- add new table AlbumRootPaths with 3 columns
  - int albumRoot
    refers to AlbumRoots primary key
  - longtext user
    refers to specific digikam instance. This value is a single constant per
    client and is stored in local digikam config. May use UUID here too.
  - longtext specificPath
    path to collection root as accessed by specific digikam client

Now to access an album one should JOIN between Albums, AlbumRoots and
AlbumRootPaths WHERE AlbumRootPaths.user is equal to the string provided by
the current client.

BTW, this way we need not rely on "good" UUID supplied by KIO/OS/whatever as
long as a path does not change (as seen by the same client).

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Roger Larsson]

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #6 from Roger Larsson <roger larsson e-gatan se>  2011-05-04 09:46:59 ---
What is essential is that when digikam feels lost it should ask for a corrected
root - not automatically remove all tags.
[Can it be "Scan for new items at startup (makes startup slower)" that causes
this automatic tag deletion?]

Digikam should work on album restored from a backup to a different disk,
currently it does not.

I mount my disks on label => the tree is the truth.
USB disks are nowadays also automatically mounted by label as /media/LABEL


Local/user mounted (NFS) disks are a difficult problem...
Those would require different album roots for different users...
(As would using private symbolic links in path to database)

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Andreas K. Huettel]

https://bugs.kde.org/show_bug.cgi?id=267131


Andreas K. Huettel <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #7 from Andreas K. Huettel <dilfridge gentoo org>  2011-05-04 12:32:55 ---
I'm following this discussion with an uneasy feeling... Please try to avoid the
mistake that the Amarok developers made, see bug 265567. There, the database
with ratings and statistics for each song identifies each device by UUID. With
the switch from HAL (KDE 4.4) to UDisks (KDE 4.6), each hard disk partition got
a new UUID, which means that everyone loses all playlist statistics, ratings,
...

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #8 from Ilia K. <mail4ilia gmail com>  2011-05-04 15:27:51 ---
Yes Andreas, that's what I've written in comment #5!
Currently, digikam relies on KDE (Solid, if I'm not mistaken) to
supply volume id which must identify the volume. I propose to use such
volumeid as a hint at most and ask a user to supply correct local
path, probably relying on this hint. UUID I've mentioned is just a
random string *generated* by digikam itself.

You give an important example of volumeid change as a result of
OS/library update. This is one more reason to avoid relying on it.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=267131





--- Comment #9 from Marcel Wiesweg <marcel wiesweg gmx de>  2011-05-07 21:51:36 ---
@Ilia: The full path is only a fallback in the thumbs database. Primarily used
is identifaction via file hash, with no path involved at all (and with 2.0, the
file hash has been improved so as to be much more unique)

@Andreas: I heard of no problem with the KDE 4.6 release except for an
uppercase/lowercase problem, so I assume there is no problem. We'd have tons of
bugreports otherwise.

@Roger: If digikam has a collection that was known as "hard-wired" and does not
find it, it will look for a different partition where the same path can be
found and offer a dialog for confirmation. The default behavior is to hide
collections which are not found (like on removable media). Problems arise if a
collection is stored by mount path, because Solid/HAL/udev did not provide the
necessary information. I've seen that a few times when HAL was unable to manage
a complex logical volume layout.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=267131

--- Comment #10 from Marcel Wiesweg <[hidden email]> ---
Regarding the original request:
The config file is user data, so using KConfigGroup::read/writePathEntry with
automatic $HOME expansion seems sensible.
Regarding the database, there's the case of a multi-user where photos are in a
common dir. Now one adds photos in his home dir, if they were encoded as $HOME,
digikam would think them gone with the next user. Yet such a kind of setup
would be inherently broken. I could live with $HOME expansion as well.

But both changes would not be backwards compatible, so we'd do them only with
the next non-backwards-compatible db change. (no such change is planned atm)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=267131

--- Comment #11 from Marcel Wiesweg <[hidden email]> ---
Git commit 9ed1a530b01ac4f94b98cef641eae56cc7e2067c by Marcel Wiesweg.
Committed on 21/07/2012 at 15:27.
Pushed by mwiesweg into branch 'master'.

Accept $HOME as part of the database file path in the config file

In contrast to substituting the home dir with $HOME by using writePathEntry,
this change is fully backwards compatible.

M  +10   -2    libs/database/core/databaseparameters.cpp

http://commits.kde.org/digikam/9ed1a530b01ac4f94b98cef641eae56cc7e2067c

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Teemu Rytilahti]

https://bugs.kde.org/show_bug.cgi?id=267131

Teemu Rytilahti <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #12 from Teemu Rytilahti <[hidden email]> ---
*** Bug 317616 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=267131

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Digikam config and data     |SETUP : digiKam config and
                   |cannot be moved, absolute   |data cannot be moved,
                   |paths embedded in database  |absolute paths embedded in
                   |and digikamrc               |database and digikamrc
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=267131

--- Comment #13 from Gilles Caulier <[hidden email]> ---
Will,

digiKam 4.11.0 is out.

This file still valid ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Net Codger]

https://bugs.kde.org/show_bug.cgi?id=267131

Net Codger <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #14 from Net Codger <[hidden email]> ---
Confirming this bug exists in digiKam 4.6 and the issue has remained since 2009
when it was reported in https://bugs.kde.org/show_bug.cgi?id=215486

When a collection is copied to a new system/disk/partition which naturally has
a different UUID, the database is unable to locate the images or thumbnails.

To rsolve the problem, the UUID and path must be adjusted by manually editing
the digikam4.db AlbumRoots Table.

The bug is that, if digiKam will persist with this method of specifying the
path with UUID in digiKam4.db, there should be a way to edit the path,
including the UUID, from within digiKam settings. If such an option is added,
it should be done so that it is easy for the novice user to browse to the
album/collection and have digiKam handle the UUID lookup and digikam4.db edit
behind the scenes.

Copying a collection to a new disk is a standard user operation for moving,
upgrading, restoring from backups. These operations should not break digiKam to
the point where it cannot be fixed from within the application.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=267131

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|1.8.0                       |4.6.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=267131

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Database                    |Database-Setup

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=267131

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #15 from [hidden email] ---
Is the file still valid using digiKam 5.1.0?
Users are requested to test and provide necessary updates.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=267131

Simon <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #16 from Simon <[hidden email]> ---
Yes this is still the same. There is no way to move a collection. The location
of a collection is stored in the database, so it has to be manually adjusted in
the database. Currently digikam sees that all files vanished, so deletes them
in database and the only thing you can do is creating a new collection at the
new location, obviously losing all information stored in the database. A
solution within the current framework would be a function within digikam to
move a collection.
In my opinion much better would be to take a different approach at storing
collection locations: Keep them separate from the database either in the
existing or a new configuration file. That means all the paths in the database
should be relative to root and the path to root is stored in this configuration
file. This allows for easily changing the location of a collection and even for
two digikam instances to share a database (not simultaneously, of course, but
if that is a worry, that should be solved by locking). A use case would be if
data and database were shared over a network and both could be accessed on
different computers.

--
You are receiving this mail because:
You are the assignee for the bug.