[Bug 158377] New: digikam duplicates downloaded images while overwriting existing ones

[Thiago Jung Bauermann-3]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         
           Summary: digikam duplicates downloaded images while overwriting
                    existing ones
           Product: digikam
           Version: 0.9.3
          Platform: Debian testing
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: general
        AssignedTo: digikam-devel kde org
        ReportedBy: thiago.bauermann gmail com


Version:           0.9.3 (using KDE 3.5.8)
Installed from:    Debian testing/unstable Packages
OS:                Linux

When digikam is not able to download an image from the SD card at first (red X is shown in the image), I selected the failed images and used "download selected" to try to get them again. Digikam then says that it was able to download those, but in the process it also overwrote neighbour images with the downloaded ones!

I will upload some screenshots to illustrate the problem.

I was able to consistently reproduce the problem with the current images in my SD card, but at each time the set of images which digikam can't download is different. When started from the command line, no error is displayed by digikam, just this:

Found dcraw version: 8.81
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 8
Exif orientation tag set to: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1
Exif Orientation: 1

For now, I will keep the SD card as it is to guarantee I can reproduce the problem later if needed. But if I need to use it for some reason, I'll have to erase the card and thus loose my ability to reproduce the problem.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Thiago Jung Bauermann-3]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From thiago.bauermann gmail com  2008-02-25 03:22 -------
Created an attachment (id=23704)
 --> (http://bugs.kde.org/attachment.cgi?id=23704&action=view)
digikam window showing duplicated images
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Thiago Jung Bauermann-3]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From thiago.bauermann gmail com  2008-02-25 03:24 -------
Created an attachment (id=23705)
 --> (http://bugs.kde.org/attachment.cgi?id=23705&action=view)
download dialog with images for retry selected
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Thiago Jung Bauermann-3]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From thiago.bauermann gmail com  2008-02-25 03:29 -------
In the attached dialog screenshot, I had to retry downloading the selected images (img_3494.jpg and img_3497.jpg).

In the attached digikam screenshot, you can see that img_3492.jpg was overwritten with the contents of img_3494.jpg, and img_3496.jpg was overwritten with the contents of img_3497.jpg. You can see in the dialog screenshot that those images are originally very different.

Another weird thing is that what should be img_3496.jpg is called in digikam img_3495.jpg. And the real img_3495.jpg is missing. In the case of img_3492.jpg there was no such mess. It is img_3492.jpg itself that is missing.

I hope you realise how serious this bug is.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         
caulier.gilles gmail com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|general                     |Camera GUI
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From dani zubinet org  2008-02-25 11:23 -------
this is the same bug as 157681
it is a serious problem, I had to downgrade to 0.9.2
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From Klaus.Weidenbach gmx net  2008-02-25 16:07 -------
I mentioned this problem some time ago in digikam-devel mailinglist. The red cross marks the wrong image as failed. In fact that picture got downloaded successfully, but it is stored with the image name right before it and that one is silenty overwritten, or maybe that failed downloading. So when you redownload the red cross image it looks as everything is all right, but it is not. Didn't had time to investigate more on this, but it is a really nasty bug that really results in image loss.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[davidfraser]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         
davidf sjsoft com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |davidf sjsoft com



------- Additional Comments From davidf sjsoft com  2008-03-01 05:46 -------
I have lost photos on this bug and now it's happened again. I have kept Digikam open with the current situation to try and debug this...
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Arnd Baecker]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         
arnd.baecker web de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |critical
           Priority|NOR                         |VHI



------- Additional Comments From arnd.baecker web de  2008-03-01 07:36 -------
Bumping up priority and severity - data loss should not happen
(However I can't help, because I don't see the problem ..)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From waipengyip yahoo com  2008-03-01 16:40 -------
It is happening to me too, since I upgraded. To replicate this, get a card of images (not need to be full, but at least >50).

1. Open up the media in Camera - Browse Media - <your media>.
2. In Settings - File rename options - Camera file name - Leave as is.
3. In On the Fly Operation - Auto-rotate / Flip image
4. Select Download All.
5. Wait for the download to finish.
6. Look for the Red Cross / X. for a file (e.g. 317.jpg)
7. Take note of the file name of the previous file. Look for it in the downloaded folder. (e.g. 316.jpg)
8. Open that. You will find that it is actually 317.jpg. 316.jpg is not downloaded at all.

I hope this helps. If you need anything, please let me know!

I am running Fedora 8, Digikam 0.9.3 from package.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Loïc Brarda-2]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From loic.brarda21 fnac net  2008-03-02 23:15 -------
I have the same problem here. The messages in the terminal window were (I added some debug messages in cameracontroler.cpp):
digikam: Exif autorotate: pict7096.jpg using (/home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374)
digikam: mimetype = JPEG
Minolta Makernote Orientation: 72
digikam: ExifRotate: no rotation to perform: /home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374
digikam: File downloaded: pict7096.jpg using (/home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374)
digikam: Downloading: pict7097.jpg using (/home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374)
digikam: Exif autorotate: pict7097.jpg using (/home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374)
digikam: ExifRotate: file do not exist: /home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374
digikam: File downloaded: pict7097.jpg using (/home/loic/pictures/test/2008-02-25/.digikam-camera-tmp1-22374)

pict7097.jpg was downloaded and replaced pict7096.jpg. pict7097.jpg had the wrong download icon. What is strange is that ExifRotate does not find the temporary file. I'll try to have a longer loog tomorrow if I have time.

  Loïc
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From caulier.gilles gmail com  2008-03-03 08:09 -------
To be able to reproduce this problem, i need to see a screenshot of all pages from Camera GUI "Settings" right sidebar tab with all download configuration.

Also, let's me hear which camera driver is used in your case. Go to "Help" button and select "Camera Information".

Thanks in advance

Gilles Caulier
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From caulier.gilles gmail com  2008-03-03 09:41 -------
I cannot reproduce this problem with current svn implementation. In all cases, a confirmation dialog is launch to ping users about a possible overwriting in target album.

If in "File Renaming Options", "Camera Filenames" is used to download items, no items are overwrited by differents pictures. The same file name is used in target album than camera file name.

If "Customize" option is used instead, digiKam use KDE rename dialog to ping user about a possible overwritting. digiKam detect properly than a file with the same name already exist in target album (a preview of existing image and camera image is displayed), and propose to rename target file to download...

Gilles Caulier
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From caulier.gilles gmail com  2008-03-03 09:45 -------
Created an attachment (id=23769)
 --> (http://bugs.kde.org/attachment.cgi?id=23769&action=view)
file name rename dialog during download from camera to prevent overwritting
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From waipengyip yahoo com  2008-03-03 15:32 -------
I feel that the problem is less with the overwriting files then of files being silently failing and renamed.

To help with you reproducing the problem

1. In Settings - File rename options - Camera file name - Leave as is.
2. In On the Fly Operation - Auto-rotate / Flip image

As for "Camera Information":

Mounted Camera driver for USB/IEEE1394 mass storage cameras and Flash disk card readers.

Title: Images found in media:/sdb1
Model: directory browse
Port: Fixed
Path: /media/NIKON D50
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From caulier.gilles gmail com  2008-03-03 15:41 -------
Yip,

I use the same config here, using a mounted point on my HDD, not a real UMS camera connected to my computer (it's more easy to test).

Of course, the problem still un-reproductible...

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From caulier.gilles gmail com  2008-03-03 15:43 -------
Yip,

When i said "un-reproductible... ", i want mean than pictures are never silently overwritten. I have always a dialog to rename pictures when it's necessary...

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[davidfraser]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From davidf sjsoft com  2008-03-03 16:05 -------
I think I have traced what the problem is, by inspecting the source code.
The actual sequence (the important parts):
 A) Download to temporary file (ALWAYS WITH THE SAME NAME, based on the pid)
 B) send event saying download complete
 C) When event is received, rename temporary file to destination name

A) and B) take place in the CameraThread::run method. C) is handled in the CameraController::customEvent method.

Since these happen in separate threads, if the event handler ever gets behind the downloader, what will happen is:
 img1 step A) Download to temporary file
 img1 step B) Send event
 img2 step A) Download to temporary file (OVERWRITING img1 in the temporary file
 img2 step B) Send event
 img1 step C) Rename temporary file to dest/img1, CONTAINING img2!!!
 img2 step C) Try rename temporary file to dest/img2, FAILING, since temporary file is missing

The reason that I am sure that this is the problem, is that we know that a GPItemInfo::DownloadFailed must have been issued for img2, since the icon shows as a cross. Of the places emitting this signal, one is when the CameraEvent::gp_downloadFailed is issued. But this creates a dialog (as Gilles said), and no dialog is displayed. The other is if the file renaming fails (which it will if the temporary file has disappeared). In this case, no dialog is displayed - the signal is simply sent.

Simplest fix would be to generate a unique temporary file name for each file. Patch to follow...
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[davidfraser]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From davidf sjsoft com  2008-03-03 16:22 -------
Created an attachment (id=23770)
 --> (http://bugs.kde.org/attachment.cgi?id=23770&action=view)
Patch to include the filename in the source

Added patch that tries to include the filename in the source. Haven't tested
yet :-)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[davidfraser]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=158377         




------- Additional Comments From davidf sjsoft com  2008-03-03 16:59 -------
Tested and it at least works... I would recommend that this be used as a band-aid patch though.
The problem with the current system is that if the final rename fails, the picture has not been copied correctly, but there is no good way to recover this.
It seems that the rename was moved into the event handling section so that there could be GUI interaction to change the name.
Things that would improve this:
 * If the rename fails, a dialog should be created just like if the copy fails.(It is not expected to fail because the target file is checked, but as this error shows it can fail in other ways)
 * If the rename fails, then using "Download/Delete" should NOT delete images that weren't copied successfully... so the deleteAfter in slotDownload should ignore failed images
 * There should be options to "Select Failed" images and to "Delete successfully copied images"
 * There should be at least a sanity check when deleting to ensure that the target deleted image is the same file size etc as the source image, and big error dialogs should pop up if this is not the case :-)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel