Tagging deletes image-clueless

[ARKPhot]

Hi everybody,

I've been using DigiKam for years, it has always exceeded my expectations (and provided a lot of features that I have never even looked into)
Over time I have got used to tagging images either via the sidebar or by invoking the context menu. There are only 12 tags I use regularly.
A year ago I bought an old Lenovo T400 Notebook, which runs Fedora 20 flawlessly and fast from a Samsung SSD.

About a month ago I realized that the images I tagged were disappearing: Assigning a tag made the picture DISAPPEAR not only in DigiKam's database, but also systemwide! I tried every imaginable search throughout the whole system: Gone. This is reproducible.

I can only tag by using the editor, changing the image AND applying the tag. Just tagging in editor kills the image as well.

On April 18th I filed a bug with Bugzilla (Bug 333565, which appears in the devel section) but still seem to be the only one with that problem.

I'm still looking for clues, because I don't want to have to get used to another piece of software.

Anybody?

Thanks in advance,

Arne

[Gilles Caulier-4]

Tagging an image do not delete file.

At least it can change metadata contents through Exiv2, if you set
right options in config pannel.

I never reproduce this bug here. I use current implementation of course...

Gilles Caulier

2014-05-04 8:05 GMT+02:00 ARKPhot <[hidden email]>:

> Hi everybody,
>
> I've been using DigiKam for years, it has always exceeded my expectations
> (and provided a lot of features that I have never even looked into)
> Over time I have got used to tagging images either via the sidebar or by
> invoking the context menu. There are only 12 tags I use regularly.
> A year ago I bought an old Lenovo T400 Notebook, which runs Fedora 20
> flawlessly and fast from a Samsung SSD.
>
> About a month ago I realized that the images I tagged were disappearing:
> Assigning a tag made the picture DISAPPEAR not only in DigiKam's database,
> but also systemwide! I tried every imaginable search throughout the whole
> system: Gone. This is reproducible.
>
> I can only tag by using the editor, changing the image AND applying the tag.
> Just tagging in editor kills the image as well.
>
> On April 18th I filed a bug with Bugzilla (Bug 333565, which appears in the
> devel section) but still seem to be the only one with that problem.
>
> I'm still looking for clues, because I don't want to have to get used to
> another piece of software.
>
> Anybody?
>
> Thanks in advance,
>
> Arne
>
>
>
> --
> View this message in context: http://digikam.1695700.n4.nabble.com/Tagging-deletes-image-clueless-tp4669153.html
> Sent from the digikam-users mailing list archive at Nabble.com.
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[ARKPhot]

I thought so too, but then what happens?

The IMG just seem to drop out of existence, there's no entry in history about a changed file, nor any other hint about anything being done to it, it's just missing.

Funny that I should be the only one, I wonder what I must have done to the system that it behaves so cruelly...

Could that be related to the SSD in a way?

Thanks again.

Arne

[Gilles Caulier-4]

2014-05-04 9:32 GMT+02:00 ARKPhot <[hidden email]>:

> I thought so too, but then what happens?
>
> The IMG just seem to drop out of existence, there's no entry in history
> about a changed file, nor any other hint about anything being done to it,
> it's just missing.
>
> Funny that I should be the only one, I wonder what I must have done to the
> system that it behaves so cruelly...
>
> Could that be related to the SSD in a way?

It can be material failure...

To be sure, and to have the exact condition of dysfunction, i recommend :

1 to switch on/off metadata writing options to see the difference. If
no metadata are touch in image, file is not modified physically (only
DB). So file must still here as well.

2 You said that it's happen in editor, not album GUI. Both use same
interface to tags. There is no code difference. All is factored.

3 turn on debug statements in console through kdebugdialog (debug
space "digiKam", "KExiv2", etc...) and run digiKam from CLI to see all
debug message during dysfunction...

Gilles Caulier
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[ARKPhot]

I seem to not have made myself entirely clear:

the only way to tag a photo is by changing it in editor. Only a changed and rewritten image gets associated with a tag.

Tagging in all other views/via context menu/sidebar etc. results in "spontaneous massive existence failure" of the image. (at least to me...)

I just checked that I don't have that problem in my old install of Fedora 19 with DigiKam 3.4.0 on the spinning disc.

Here is what Debug says when I open enfuse-01.png then assign tag "Fanoe 2011". It takes a second for the data to get written to the file, then the file disappears. Any ideas which other process might be responsible?


Output:
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::setIptcKeywords: XXX/seals/enfuse-01.png  ==> Iptc Keywords:  Fanoe 2011
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::save: KExiv2::metadataWritingMode 3
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::save: Will write Metadata to file "enfuse-01.png"
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::save: Metadata for file "enfuse-01.png" written to file.
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/digikam (core) Digikam::DImg::load: "XXX/seals/enfuse-01.png"  : PNG file identified
digikam(4198)/digikam (core) Digikam::ImageScanner::commit: Scanning took 30 ms
digikam(4198)/digikam (core) Digikam::ImageScanner::~ImageScanner: Finishing took 21 ms
digikam(4198)/digikam (core) Digikam::DImg::load: "XXX/seals/enfuse-01.png"  : PNG file identified
digikam(4198)/digikam (core) Digikam::DMetadata::getImageHistory: Loading image history  ""
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::getImageDateTime: DateTime => Exif.Photo.DateTimeOriginal =>  QDateTime("Mo. Okt 7 23:39:35 2013")
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1
digikam(4198)/digikam (core) Digikam::AlbumWatch::rescanDirectory: Detected change, triggering rescan of directory "XXX/seals"
digikam(4198)/digikam (core) Digikam::CollectionScanner::itemsWereRemoved: Removed items: (5424) related items: ()
digikam(4198)/kio (Scheduler) KIO::SchedulerPrivate::doJob: KIO::SimpleJob(0x28b1110)
digikam(4198)/kio (KIOJob) KIO::TransferJob::slotFinished: KUrl("digikamalbums:/HuginTMP/seals/?albumRoot=%XXX&albumRootId=1&databaseType=QSQLITE&databaseName=%XXX%2Fdigikam4.db&connectOptions=&hostName=&userName=&password=")
digikam(4198)/kio (Scheduler) KIO::SchedulerPrivate::jobFinished: KIO::SpecialJob(0x28b1110) KIO::Slave(0x7b668c0)
digikam(4198)/digikam (core) Digikam::DImg::load: "XXX/seals/IMG_9779-IMG_9783_exposure_layers_0001.tif"  : TIFF file identified
digikam(4198)/KEXIV2 KExiv2Iface::KExiv2::getImageOrientation: Orientation => Exif.Image.Orientation =>  1

[Elle Stone-2]

On 05/04/2014 05:55 AM, ARKPhot wrote:
> I seem to not have made myself entirely clear:

> Tagging in all other views/via context menu/sidebar etc. results in
> "spontaneous massive existence failure" of the image. (at least to me...)

> Here is what Debug says when I open enfuse-01.png then assign tag "Fanoe
> 2011". It takes a second for the data to get written to the file, then the
> file disappears. Any ideas which other process might be responsible?
>

Are all the disappearing files png files? Or are jpeg and tiff files
similarly affected?


Elle Stone

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[ARKPhot]

I have experienced the problem with both png and jpg, wait a mo...yep tifs are affected too. Just tried with .jp2 and actually saw the tagged image in album view for a split second before it got lost.

Very strange

[Gilles Caulier-4]

It' abnormal.

I see seen this dysfunction here, and i manage a lots of pictures of
all type (JPEG, RAW, PNG)

You can image easily that something like this will be reported quickly
by other users... deleting a files is very serious, and personally
will be be crying immediately about (:-)))

Please check your hardware. Enable kernel log and look warnings and errors.

You can run hdparm CLI tool to check hardware log about your device too...

Note : here, for performances reasons, i use also SSD : one for OS,
one to host pictures. Data are saved in NAS for backup.

Gilles Caulier



2014-05-04 13:59 GMT+02:00 ARKPhot <[hidden email]>:

> I have experienced the problem with both png and jpg, wait a mo...yep tifs
> are affected too. Just tried with .jp2 and actually saw the tagged image in
> album view for a split second before it got lost.
>
> Very strange
>
>
>
> --
> View this message in context: http://digikam.1695700.n4.nabble.com/Tagging-deletes-image-clueless-tp4669153p4669162.html
> Sent from the digikam-users mailing list archive at Nabble.com.
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Elle Stone-2]

On 05/04/2014 07:59 AM, ARKPhot wrote:
> I have experienced the problem with both png and jpg, wait a mo...yep tifs
> are affected too. Just tried with .jp2 and actually saw the tagged image in
> album view for a split second before it got lost.
>
> Very strange
>

Just from curiosity, what happens if you write to an XMP side car file
instead of directly to the image?

"Settings/Configure/Metadata/Behavior/Reading and Writing Metadata",
check "Read from sidecar files"
and also check "Write to sidecar files",
and choose "Write to XMP sidecar files only"

> I tried every imaginable search throughout the whole
> system: Gone. This is reproducible.

Have you done a command line search to verify that the disappearing
images really are gone and not just invisible to digiKam/KDE?


Elle Stone
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Remco Viëtor]

On Sunday 04 May 2014 02:55:36 ARKPhot wrote:
(...).
>
> Here is what Debug says when I open enfuse-01.png then assign tag "Fanoe
> 2011". It takes a second for the data to get written to the file, then
the
> file disappears. Any ideas which other process might be responsible?

That name 'enfuse-01.png' sounds like it is an intermediate file, possibly
created by a process and deleted by that same process before exit.
The enfuse program comes to mind... (used in exposure stacking or panorama
stitching).
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Sveinn í Felli-2]

Þann sun  4.maí 2014 07:32, skrifaði ARKPhot:
>
> Could that be related to the SSD in a way?
>

I set up a SSD (OCZ Agility3 160Gb) couple of years ago;
during the first month it behaved great but then all kinds
of wierdness started to bug me. At that time there were
problems with implementation of the TRIM function which
affects filesystems. Some drives were more affected than
others, and since then the implementation in the Linux
kernel has also been modified.
Maybe this is worth exploring?

Sveinn í Felli

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Andrew Fritz-2]

I'm going to jump in and make a suggestion.

Start a brand new collection. Just set up a folder on your desktop or
something and copy a few images in. Test there in a VERY controlled
environment.

While testing, at the command line in a terminal, probably using sudo
(i.e. as root), look at the folder using "ls -alh" to make sure they
actually go away. Don't do this in a GUI file manager. Do it at the CLI.

Check dmesg if/when they go away and see if anything is being reported
in the kernel's log.

As a developer I can think of a couple of things that could happen in a
piece of software to "delete" a file when rewriting it, but all of those
would leave a 0 byte file with no contents (for example, some exception
in the code that rewrites the metadata after opening the file for
write). Without looking at that code in Digikam, I'd assume (based on
its wide user base and the fact that this isn't happening otherwise)
that that is NOT the case though. Or, if it is, it is something very
specific related to hardware or kernel issues on your specific system.

Andrew

Email: [hidden email]
Phone: [phone]
Andrew Fritz, Photographic - Weddings and Portraits
  http://www.fritztech.com
  https://www.facebook.com/AndrewFritzPhotographic
Arlington Real Estate Photography
  http://www.arlingtonrealestatephotography.com

On 05/04/2014 08:43 AM, Remco Viëtor wrote:

> On Sunday 04 May 2014 02:55:36 ARKPhot wrote:
> (...).
>> Here is what Debug says when I open enfuse-01.png then assign tag "Fanoe
>> 2011". It takes a second for the data to get written to the file, then
> the
>> file disappears. Any ideas which other process might be responsible?
> That name 'enfuse-01.png' sounds like it is an intermediate file, possibly
> created by a process and deleted by that same process before exit.
> The enfuse program comes to mind... (used in exposure stacking or panorama
> stitching).
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Andrew Fritz-2]

Another test: set up the new collection on a different drive (non-SSD),
maybe a external USB. Test again.

Try to isolate the factors.

Email: [hidden email]
Phone: [phone]
Andrew Fritz, Photographic - Weddings and Portraits
  http://www.fritztech.com
  https://www.facebook.com/AndrewFritzPhotographic
Arlington Real Estate Photography
  http://www.arlingtonrealestatephotography.com

On 05/04/2014 08:53 AM, Sveinn í Felli wrote:

> Þann sun  4.maí 2014 07:32, skrifaði ARKPhot:
>>
>> Could that be related to the SSD in a way?
>>
>
> I set up a SSD (OCZ Agility3 160Gb) couple of years ago; during the
> first month it behaved great but then all kinds of wierdness started
> to bug me. At that time there were problems with implementation of the
> TRIM function which affects filesystems. Some drives were more
> affected than others, and since then the implementation in the Linux
> kernel has also been modified.
> Maybe this is worth exploring?
>
> Sveinn í Felli
>
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[ARKPhot]

Yes it is an enfuse-file, but it is the end-result: It's my usual name for enfused files and resides with the source images in a special enfuse folder that I create for the task.
Thanks
Arne

[ARKPhot]

Thanks for the suggestions, will try that, but soonest tonight...

Arne

[ARKPhot]

This SSD is in use since October 2013, no problems until now, just this one in digikam...

Thanks,

Arne

[ARKPhot]

find does not see the lost files either... :(

Thanks,

Arne

[Andrew Fritz-2]

What file system are you using? Is it a "standard" (ext4 or 3) or
something strange? This sounds more and more like a weird interaction
between software, driver and hardware.

Email: [hidden email]
Phone: [phone]
Andrew Fritz, Photographic - Weddings and Portraits
  http://www.fritztech.com
  https://www.facebook.com/AndrewFritzPhotographic
Arlington Real Estate Photography
  http://www.arlingtonrealestatephotography.com

On 05/04/2014 10:17 AM, ARKPhot wrote:

> find does not see the lost files either... :(
>
> Thanks,
>
> Arne
>
>
>
> --
> View this message in context: http://digikam.1695700.n4.nabble.com/Tagging-deletes-image-clueless-tp4669153p4669172.html
> Sent from the digikam-users mailing list archive at Nabble.com.
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[ARKPhot]

FS is standard ext4

You could completely rule out a (userland) software bug (either in digikam or one of its libraries) by running digikam with strace and seeing if there are any calls to unlink/unlinkat.

Try something like

strace -o /tmp/output.log -f /usr/bin/digikam

then try to reproduce the bug. After you close digikam, check that /tmp/output.log file and look for any calls that reference the file to try to figure out what happened (there should be at least some for opening the file and writing the metadata). If anything deleted the file explicitly there should be a call to "unlink" or "unlinkat".  If it's not there, then is either a kernel error or a hardware problem.

HTH.

On Sun, May 4, 2014 at 8:57 AM, ARKPhot <[hidden email]> wrote:

FS is standard ext4



--
View this message in context: http://digikam.1695700.n4.nabble.com/Tagging-deletes-image-clueless-tp4669153p4669174.html

Sent from the digikam-users mailing list archive at Nabble.com.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users