[digikam] [Bug 320466] New: after a batch process lot of files are missing in preview

[Leo-2]

https://bugs.kde.org/show_bug.cgi?id=320466

            Bug ID: 320466
           Summary: after a batch process lot of files are missing in
                    preview
    Classification: Unclassified
           Product: digikam
           Version: 3.2.0
          Platform: Ubuntu Packages
                OS: Linux
            Status: UNCONFIRMED
          Severity: grave
          Priority: NOR
         Component: Preview
          Assignee: [hidden email]
          Reporter: [hidden email]

I processed (converted to jpg and resized) 96 files (some jpg some crw) crating
it in a new folder
Everything went fine but if I look in the new folder with dolphin I see all 96
files, but if I look in the main digikam gui, I find only 84!!!!

Reproducible: Always

Steps to Reproduce:
1. produce some file with a batch process into a new dir
2. diff the 2 dir
3. the number of files is different
I've tryed to do the same actions on the same file with different target dir...
Everytime I find ALL the files in the dirs with dolphin but the digikam gui
shows me differents files (some times 84, sometimes 86) but NEVER all the
files!!!! :(
Actual Results:  
not all the file produced are visible in digikam

Expected Results:  
all files should be visible

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from Gilles Caulier <[hidden email]> ---
How do you process RAW to JPEG ?

Can you run "ls -al" in album with JPEG files.
Also take a shot of icon view from digiKam with this album.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Preview                     |Albums GUI

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Leo-2]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #2 from Leo <[hidden email]> ---
Created attachment 80273
  --> https://bugs.kde.org/attachment.cgi?id=80273&action=edit
The origin

Here are the file requested.
A list of all the files in both dir, a snapshot of gui in the 2 dirs, and as
you can see in dolphin snapshot, the count of files is correct in both dirs,
but in DK some files in jpg are missing ;(

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

Nathan <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from Nathan <[hidden email]> ---
I have experienced this situation twice.  The first time is via a process
similar to the above steps, but I used dark table to do the conversion from
Olympus RAW to tiff (265 files, 265 preview images) and then after a bit more
processing and renaming from tiff to jpg (265 files, 186 preview images) in
digiKam.  This was first seen on digiKam 3.1 on Kubuntu 13.04.  I upgraded to
3.2 and still see the same issue.

In the second case, the setup is a bit different.  I took 459 files from
someone else and copied them to my computer.  I then opened the gallery in
digiKam and it shows only 47 files with 47 preview images.  These were taken in
camera as jpeg images and have not been converted at all.  I have only noticed
this on digiKam 3.2 on Kubuntu 13.04

The second one is not identical, but is close enough in effect that I thought I
should mention them both.  All files are owned by me with at least r&w
permissions for me.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #4 from Nathan <[hidden email]> ---
I have a work around in case anyone needs it.  In every case I have tried it,
importing the "missing" pictures causes them to show up.  So the steps for this
work around are as follows:

1) In digiKam, create a new gallery.  This is just temporary storage (call it
DKTemp).
2) Select all thumbnails and move them to the new temporary folder (DKTemp).
3) In some other program (not digiKam - shell, dolphin, etc.) move all of the
files that remain in the original folder (call it DKGallery) to some other
location - I used /tmp
4) In digiKam, move everything from DKTemp to DKGallery.
5) In digiKam, import everything from /tmp to DKGallery.
6) Delete everything in /tmp

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #5 from Gilles Caulier <[hidden email]> ---
digiKam 3.5.0 is out.

Can you give a fresh feedback about your report ? Problem still reproducible ?

Thanks in advance

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #6 from Nathan <[hidden email]> ---
I was able to reproduce it in 3.4.0, but I'll see if I can do the same in
3.5.0.

One negative on 3.4.0 was that my work around (the import process) didn't work
anymore.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.2.0                       |3.4.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #7 from Nathan <[hidden email]> ---
I can confirm that the issue still exists in 3.5.0

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.4.0                       |3.5.0

--- Comment #8 from Gilles Caulier <[hidden email]> ---
Are you sure that problem is not relevant of digiKam versioning settings ?
Check on setup dialog for details, especially the section Editing Images where
"Always show original image" and "Always show intermediate snapshot" options
can be a solution...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #9 from Nathan <[hidden email]> ---
I've thought about that and it isn't the case here.  When I do a batch process,
I am going to a new folder with a rename - so it shouldn't be a versioning
thing.  In addition, I have it set to destructive save, so I don't get
versions.  This is truly a new file/photo that just doesn't show in the main
thumbnail window no matter what I do.

One change on this version - this is the first time on this bug that I have
compiled digiKam from source.  I have historically waited until KUbuntu has it
in the repositories prior to upgrading.  I wanted to troubleshoot this more
though, so I compiled instead of waiting.  I'm happy to run tests or check
other things if needed!

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Faldrian]

https://bugs.kde.org/show_bug.cgi?id=320466

Faldrian <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #10 from Faldrian <[hidden email]> ---
I use digikam 3.5.0 on Gentoo and also have this bug.
When I use batch to resize pictures into another folder, there are sometimes 1
or 2 pictures missing in the target folder - at least digikam does not have
them in list. On Disk all files are there, so it's just an issue with the GUI
not showing them or not indexing them.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #11 from Nathan <[hidden email]> ---
ShowFoto works:

I just realized something interesting that might be a decent work around for
now.  If I open a folder in showFoto, it will display all of the pictures in
the folder window even for folders/galleries that do not display all of the
images in digiKam.  There must be a difference between the way these 2
applications generate their folder view thumbnails despite how closely related
they are.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #12 from Gilles Caulier <[hidden email]> ---
digiKam use DB to register file system items.
Showfoto do not use DB.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #13 from Nathan <[hidden email]> ---
It appears that the photos are being registered in the database correctly - as
in the insert is at least partially successful.  Details:

Look in the gallery and I see 10 thumbnails, but the Album pane shows there
should be 23.

Database queries:
select * from Images where album = 1154;  (23 rows - correct)

select *
from Images i
    inner join ImageMetadata im on i.id = im.imageid
where i.album = 1154; (8 rows - Every image here has a thumbnail, but so do 2
others)

select *
from Images i
    inner join ImageInformation ii on i.id = ii.imageid
where i.album = 1154; (23 rows - correct)

select *
from Images i
    inner join ImageHistory ih on i.id = ih.imageid
where i.album = 1154; (23 rows - correct)

I'll try and get the MySQL general log output of a batch editing run shortly.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #14 from Nathan <[hidden email]> ---
I experienced this bug on import of photos and turned on the general log at
that point.  Deleted and re-imported the files.  The same files were missing
and it appears that the reason it is always the same files is that the insert
into ImageInformation uses the previous row as the source for the new import.
Would it be possible for import to do it from scratch each time instead of
trying to be smart about it and failing consistently?

Line from the log:
INSERT INTO ImageInformation  (imageid, rating, creationDate, digitizationDate,
orientation,   width, height, format, colorDepth, colorModel) SELECT 224073,
rating, creationDate, digitizationDate, orientation,   width, height, format,
colorDepth, colorModel FROM ImageInformation WHERE imageid=224056

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Nathan]

https://bugs.kde.org/show_bug.cgi?id=320466

--- Comment #15 from Nathan <[hidden email]> ---
(In reply to comment #14)

> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
>
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

(In reply to comment #14)

> I experienced this bug on import of photos and turned on the general log at
> that point.  Deleted and re-imported the files.  The same files were missing
> and it appears that the reason it is always the same files is that the
> insert into ImageInformation uses the previous row as the source for the new
> import.  Would it be possible for import to do it from scratch each time
> instead of trying to be smart about it and failing consistently?
>
> Line from the log:
> INSERT INTO ImageInformation  (imageid, rating, creationDate,
> digitizationDate, orientation,   width, height, format, colorDepth,
> colorModel) SELECT 224073, rating, creationDate, digitizationDate,
> orientation,   width, height, format, colorDepth, colorModel FROM
> ImageInformation WHERE imageid=224056

I believe the missing row in "ImageInformation" is controlling as the query run
when viewing a gallery joins to that table.  Such as:
SELECT DISTINCT Images.id, Images.name, Images.album,      
ImageInformation.rating, Images.category,        ImageInformation.format,
ImageInformation.creationDate,        Images.modificationDate, Images.fileSize,
       ImageInformation.width, ImageInformation.height  FROM Images      
INNER JOIN ImageInformation ON Images.id=ImageInformation.imageid  WHERE
Images.status=1 AND Images.album = ?


I have yet to capture the initial insertion for the parent file into
ImageInformation for why it failed.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|after a batch process lot   |After to run BQM a lot of
                   |of files are missing in     |files are missing in
                   |preview                     |icon-view
          Component|Albums GUI                  |Database

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=320466

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|After to run BQM a lot of   |SCAN : after to run BQM a
                   |files are missing in        |lot of files are missing in
                   |icon-view                   |icon-view

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel