BUG? move to album / overwrite destination problem

[John Stumbles]

I have an album containing pictures, and a sub-directory containing a
sub-set of those pictures, plus some modified versions. I've rated the
images in the sub-directory and now I want to put all pictures from the
sub-directory into the main album. I know this will involve clashes with
pictures that already exist in the main album but I just want to
overwrite those.

So I select all pictures in the subdirectory, right-click to 'move to
album' and select the main album directory. DK throws up a warning that
a 'File Already Exists' and I press the 'Overwrite' button but DK just
pops up a warning window about the same file again ... and again and
again. It seems to be broken :-(

Incidentally the duplicate files in the main album and the sub-directory
are hard links, not copies.

I guess I can overwrite the files using other tools, gui or cli, but I
want to do it with DK so it remembers the ratings I've applied to files
in the subdir: I suspect that if I move them outside of DK it will just
seem them as having vanished and forget the ratings I'd applied to them.
(As a workaround maybe I can use some cli hacking to delete the
duplicate copies in the main dir and then move sub-dir files.)

--
John Stumbles                                    http://stumbles.org.uk
:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

Hello John,

digiKam stores hash-keys for each picture in its database.
So it should be able to detect picture movement outside of
digiKam.

Another (safer) way would be to store the ratings _in_ the
jpg-file.

 "Settings" -> "Configure digiKam" -> "Metadata" ->
"Behaviour" -> [x] "Save image ratings in metadata embedded
in files"

In this case you can always start "read metadata from
image", to read them back in the digiKam database. But
digiKam should recognize those metadata in the files
automatically.

Regards,
     Peter

On 15.05.2012 01:25, John Stumbles wrote:

> I have an album containing pictures, and a sub-directory
> containing a sub-set of those pictures, plus some modified
> versions. I've rated the images in the sub-directory and now
> I want to put all pictures from the sub-directory into the
> main album. I know this will involve clashes with pictures
> that already exist in the main album but I just want to
> overwrite those.
>
> So I select all pictures in the subdirectory, right-click to
> 'move to album' and select the main album directory. DK
> throws up a warning that a 'File Already Exists' and I press
> the 'Overwrite' button but DK just pops up a warning window
> about the same file again ... and again and again. It seems
> to be broken :-(
>
> Incidentally the duplicate files in the main album and the
> sub-directory are hard links, not copies.
>
> I guess I can overwrite the files using other tools, gui or
> cli, but I want to do it with DK so it remembers the ratings
> I've applied to files in the subdir: I suspect that if I
> move them outside of DK it will just seem them as having
> vanished and forget the ratings I'd applied to them. (As a
> workaround maybe I can use some cli hacking to delete the
> duplicate copies in the main dir and then move sub-dir files.)
>

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[John Stumbles]

On 15/05/12 18:23, Peter Albrecht wrote:
> Hello John,
>
> digiKam stores hash-keys for each picture in its database.
> So it should be able to detect picture movement outside of
> digiKam.

Yes, I'm aware of that, and it generally detects movements of picture
files quite reliably. However in the particular situation I have here
there are identical picture files (they're actually hard-linked) so if I
move the instance it has ratings for into the directory containing the
one that's not rated I'm pretty sure it would simply think that the
rated copy has been deleted.


> Another (safer) way would be to store the ratings _in_ the
> jpg-file.

That would use up unnecessary extra space in my snapshot-backup system
storing new copies of picture files when their embedded metadata changed.

As I say I can workaround this (apparent) bug by deleting the un-rated
copies of the files, getting dk to notice that these have disappeared,
and then moving the rated copies. (Midnight Commander is quite handy for
this as it can select all files in a directory which are different from
those in another directory, and work on that selected set of files.)

--
John Stumbles                                    http://stumbles.org.uk
:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Remco Viëtor]

On Tuesday 15 May 2012 21:03:32 John Stumbles wrote:

> On 15/05/12 18:23, Peter Albrecht wrote:
> > Hello John,
> >
> > digiKam stores hash-keys for each picture in its database.
> > So it should be able to detect picture movement outside of
> > digiKam.
>
> Yes, I'm aware of that, and it generally detects movements of picture
> files quite reliably. However in the particular situation I have here
> there are identical picture files (they're actually hard-linked) so if I
> move the instance it has ratings for into the directory containing the
> one that's not rated I'm pretty sure it would simply think that the
> rated copy has been deleted.
>

I think it's actually the hard linking that causes the problem:
Digikam sees that the file you want to copy to exists, asks for permission to
overwrite, and then tries to open that file for writing, truncating the
contents. But that file is also opened for reading through the other file
name...
And, if you simply move the modified files into the parent directory, you are
just deleting one of the hard links. Then on next start-up, digikam would
indeed see that the tagged files have disappeared, and removes the associated
information from its database.
>
> > Another (safer) way would be to store the ratings _in_ the
> > jpg-file.
>
> That would use up unnecessary extra space in my snapshot-backup system
> storing new copies of picture files when their embedded metadata changed.
>
Also, storing metadata in the image files isn't recommended for RAW files (and
perhaps a bad idea for original out-of-camera jpgs as well).

You seem to use the hard links to avoid making a temporary copy. You have to
do that from outside Digikam, forcing Digikam to check the albums at startup
(which does slow down startup). And Digikam has no choice but to consider the
files independant, it has no notion of hard linking afaik.
Is the use of hard links really worth the hassle and savings on disk space,
even with your snapshot backups?

An alternative might be to mark the files you want to work on (green pick
label?) and then use the right-hand filters tab to show only those.

Regards,
Remco

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[John Stumbles]

On 16/05/12 05:07, Remco Viëtor wrote:

> I think it's actually the hard linking that causes the problem:
> Digikam sees that the file you want to copy to exists, asks for permission to
> overwrite, and then tries to open that file for writing, truncating the
> contents. But that file is also opened for reading through the other file
> name...

I just checked and dolphin has the same problem so it's really a bug in
kde. Maybe I'm missing something but overwriting - bitwise copying - one
file to another in order to move it seems like a really stupid thing to
do! I wonder why kde doesn't actually do a `mv` when the source and
destination are on the same filesystem? Maybe it was a bit of q&d coding
(saving having to check whether a mv is possible) that hasn't been
refined? (I know: it's open source, I should fix it myself instead of
complaining about it! But it's not like a little C file: there's a huge
learning curve to get up to start hacking on something like kde ...
would take me months to get going :-()


> You seem to use the hard links to avoid making a temporary copy.

One of the main ways I enjoy my photo collection is having my better
pictures displayed on kde's slideshow screen-saver when my PC is idle. I
acheive this by, when I've uploaded a set of photos containing some I'd
like to have on the screensaver, I `cp -rl` the album to a directory in
the tree the screensaver accesses; then I go through images in that
directory deleting poorer ones and tweaking the better ones. So now I
have one directory tree containing all my pictures, good and bad, and
another tree ("show") containing only good ones, including some edited
versions that aren't in the original tree. And since the "show" tree
contains a significant amount of images, using hard links saves a
worthwhile amount of space.

Now I've started using digikam I'd like to merge the "show" tree back
into the originals tree. I could just copy (or hard-link) images in each
album in the show tree back to their corresponding albums in the
original tree but, since the photos in the "show" tree are all good, I
can quickly star-rate them all and save myself having to re-rate all the
pictures in the new, merged, tree.


> Is the use of hard links really worth the hassle and savings on disk space,
> even with your snapshot backups?

Definitely with the snapshot backups: I've only got a 2.5TB drive
backing up a 3.5TB array! Obviously it's not backing up everything on
that array (I exclude DV videos for example) but even so I have to be
careful what I do back up. Of course a de-duplicating file system (e.g.
ZFS) would be the ideal: I did try setting something up but couldn't get
whatever it was (I forget now) working.



--
John Stumbles                                    http://stumbles.org.uk
:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

Hello,

I happened, at some occasions, to use hard links in a (maybe) similar way
as John does (have a folder with a bunch of images and some other folders
with selected images subsets).

I confirm Digikam doesn't recognize/handle links. When pathnames are
different, the images are seen as different even if it's the same file,
and database entries are different too.

What I used to do for final cleanup after tagging/rating/indexing was :
  - Run the « Find duplicates » function on the root folder (nearest root
    above linked images),
    As linked files are the same, obviously the two images are found to
    be duplicates,
  - In the duplicates search results view, set the display filter to
    show only unrated/untagged images,
  - Select all, Ctrl-A, then delete.
  - After that, move, reorganise the remainding (rated/tagged) images.

(This worked well. I must say I no longer work like that. Now I tend to
copy SD-card content into a new folder, then browse the images and tag
those I want to keep, then move them into a final folder, then work.
The original folder with remainding images is only there in case
I would feel remorse.)

Regards,
Jean-François


PS: I have an off-topic question, about something Remco wrote in his
answer :

> Also, storing metadata in the image files isn't recommended for RAW files
> (and perhaps a bad idea for original out-of-camera jpgs as well).

I've often read that kind of warning, on this list. From my naive point
of view, embedded metadata records, EXIF, XMP, sounds to be a nice thing
as one keeps all image information into one file. (And XMP sidecar files
doesn't seem to interoperate well betwen different applications.)

I'd be really interested in learning what could be potential problems and
why this seems to be a not recommended practice. Are there reasons related
to the libexiv2 to be not that stable and reliable, and writing metadata
could corrupt or destroy the file ? Or are there other reasons ?
Thanks for making that point become clearer to me.

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Remco Viëtor]

On Wednesday 16 May 2012 10:39:43 Jean-François Rabasse wrote:
...

> PS: I have an off-topic question, about something Remco wrote in his
> answer :
>
> > Also, storing metadata in the image files isn't recommended for RAW files
> > (and perhaps a bad idea for original out-of-camera jpgs as well).
>
> I've often read that kind of warning, on this list. From my naive point
> of view, embedded metadata records, EXIF, XMP, sounds to be a nice thing
> as one keeps all image information into one file. (And XMP sidecar files
> doesn't seem to interoperate well betwen different applications.)
>
> I'd be really interested in learning what could be potential problems and
> why this seems to be a not recommended practice. Are there reasons related
> to the libexiv2 to be not that stable and reliable, and writing metadata
> could corrupt or destroy the file ? Or are there other reasons ?
> Thanks for making that point become clearer to me.

For me, there are two reasons to avoid writing to RAW files:
- They are the original, irreplacable data that I like to keep in its original
state.  (I have backups, but still...)
- There is no such thing as _a_ raw format: every camera maker has his own
formats (yes, plural :( ) And those formats are poorly specified, especially
wrt to the 'makerdata' EXIF section, so it's very easy to corrupt something
when writing to such a file. The situation is getting better through a lot of
hard work from volunteers (not me, I hasten to add), but given the sheer
number of raw formats in existence, there will probably always be formats that
are not understood.

Also, RAW files in general only have EXIF data, part of which can be pure
binary data, and possibly encrypted (makernote..)

Regards,

Remco
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

On 16.05.2012 11:10, Remco Viëtor wrote:

> On Wednesday 16 May 2012 10:39:43 Jean-François Rabasse wrote:
> ...
>> PS: I have an off-topic question, about something Remco wrote in his
>> answer :
>>
>>> Also, storing metadata in the image files isn't recommended for RAW files
>>> (and perhaps a bad idea for original out-of-camera jpgs as well).
>>
>> I've often read that kind of warning, on this list. From my naive point
>> of view, embedded metadata records, EXIF, XMP, sounds to be a nice thing
>> as one keeps all image information into one file. (And XMP sidecar files
>> doesn't seem to interoperate well betwen different applications.)
>>
>> I'd be really interested in learning what could be potential problems and
>> why this seems to be a not recommended practice. Are there reasons related
>> to the libexiv2 to be not that stable and reliable, and writing metadata
>> could corrupt or destroy the file ? Or are there other reasons ?
>> Thanks for making that point become clearer to me.
>
> For me, there are two reasons to avoid writing to RAW files:
> - They are the original, irreplacable data that I like to keep in its original
> state.  (I have backups, but still...)
> ...

In reference to "and perhaps a bad idea for original
out-of-camera jpgs as well":
Writing metadata to JPGs works very well for me, for several
years now.

The only drawback I can think of is the mentioned "changing
the original".

But I decided for me, to _not keep_ the original
out-of-camera-jpgs. So I can live with this "issue". But
this is a decision everyone has to make him-/herself.

Regards,
        Peter
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

Hello,

On Wed, 16 May 2012, Remco Viëtor wrote:

> For me, there are two reasons to avoid writing to RAW files:
> - They are the original, irreplacable data that I like to keep in its
> original state. (I have backups, but still...)
> - There is no such thing as _a_ raw format: every camera maker has his
> own formats (yes, plural : ( ) And those formats are poorly specified,
> especially wrt to the 'makerdata' EXIF section, so it's very easy to
> corrupt something when writing to such a file. The situation is
> getting better through a lot of hard work from volunteers (not me,
> I hasten to add), but given the sheer number of raw formats in
> existence, there will probably always be formats that are not
> understood.

Thanks for the detailed explanation Remco. Sounds like it's a matter
of trustworthy and reliability of open source software wrt proprietary
formats.

But ok, this for raw files. And about JPEG files :

On Wed, 16 May 2012, Peter Albrecht wrote:

> In reference to "and perhaps a bad idea for original out-of-camera
> jpgs as well": Writing metadata to JPGs works very well for me,
> for several years now.
>
> The only drawback I can think of is the mentioned "changing the
> original".
>
> But I decided for me, to _not keep_ the original out-of-camera-jpgs.
> So I can live with this "issue". But this is a decision everyone has
> to make him-/herself.

Seems that both of you agree on « keep originals as they are » .
Should lead to the conclusion that the « zero risk » procedure is to
archive (e.g. USB drive) all out-of-camera data, then copy all or
selected images on main computer and work.
(And backup processed apart from original.)

Regards,

Jean-François

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users