[Digikam-devel] [Bug 130525] New: Saving large (>5M) jpg's result in corrupt file

[J.Westveer]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         
           Summary: Saving large (>5M) jpg's result in corrupt file
           Product: digikam
           Version: 0.9.0-svn
          Platform: unspecified
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: general
        AssignedTo: digikam-devel kde org
        ReportedBy: jwest melaque com


Version:           0.9.0-svn (using KDE 3.5.3, Debian Package 4:3.5.3-1 (testing/unstable))
Compiler:          Target: i486-linux-gnu
OS:                Linux (i686) release 2.6.15-1-686-smp

When working with larger jpeg files (>5M) a save, or save as operation results in a corrupt output file.  This may be me, as I am using current svn compiled on a Debian (etch) system, but it is annoying.  If I fall back to 0.8.2 the file will save correctly.  In addition the file sizes are radically different (5.9M vs 4.4M)

digicam reports this during open and save:

igikam: /home/MyPhotos/wall/PICT5785.JPG : JPEG file identified
digikam: Exif color-space tag is sRGB. Using default sRGB ICC profile.
digikam: mimetypes=image/x-portable-bitmap image/x-pcx image/x-portable-greymap image/x-xbm image/x-targa image/png image/x-portable-pixmap image/jpeg image/x-xpm image/x-eps image/x-bmp image/jp2 image/x-rgb image/tiff
digikam: Saving to :/home/MyPhotos/wall/C2bXhadigikamtempfile.tmp (jpeg)
digikam: (800x531) JPEG image preview size: 110196 bytes
digikam: Dirty: /wall
digikam: Dirty: /
digikam: renaming to /home/MyPhotos/wall/x.JPG
digikam: ImageInfo::copyItem 554 PICT5785.JPG to 554 x.JPG
digikam: Dirty: /
digikam: Dirty: /wall
digikam: Dirty: /
digikam: mimetypes=image/x-portable-bitmap image/x-pcx image/x-portable-greymap image/x-xbm image/x-targa image/png image/x-portable-pixmap image/jpeg image/x-xpm image/x-eps image/x-bmp image/jp2 image/x-rgb image/tiff
digikam: Saving to :/home/MyPhotos/wall/spPR6adigikamtempfile.tmp (jpeg)
digikam: (800x531) JPEG image preview size: 110196 bytes
digikam: Dirty: /wall
digikam: Dirty: /
digikam: Dirty: /wall
digikam: renaming to /home/MyPhotos/wall/x1.JPG
digikam: ImageInfo::copyItem 554 x.JPG to 554 x1.JPG
digikam: Dirty: /
digikam: Dirty: /wall
digikam: Dirty: /
digikam: Cannot load metadata using Exiv2 (This does not look like a JPEG image)
digikam: Found JPEGLossless plugin
digikam: Cannot load metadata using Exiv2 (This does not look like a JPEG image)
digikam: Dirty: /wall
digikam: Dirty: /



I renamed the two files good/bad, and they are available here:
 http://www.melaque.com/bad.jpg
 http://www.melaque.com/good.jpg

jpeginfo command report on both files:

jwest fysm:~$ jpeginfo good.jpg
good.jpg 3008 x 2000 24bit Exif  N 597156

jwest fysm:~$ jpeginfo bad.jpg
bad.jpg  Corrupt JPEG data: 65519 extraneous bytes before marker 0xd9  JPEG datastream contains no image  [ERROR]


Apologies for not knowing how to debug this further.  
Thanks for any assistance.

jjw
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From caulier.gilles free fr  2006-07-09 23:42 -------
Hi,

I have already reproduce this problem in the past. I have suspected a bug in Exiv2 library used by digikam to manage metadata. But not sure...

Can you give me the Exiv2 release installed in your computer ? 0.10.0 ?

Here, to develop digiKam, i'm using current Exiv2 implementation from svn.

Gilles Caulier
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[J.Westveer]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From jwest melaque com  2006-07-10 16:37 -------
I currently have exiv2-0.10.tar.gz   loaded .

I just loaded exiv2 from cvs, made clean and recompiled.
No Joy.  Same results.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerhard Kulzer]

Am Montag, 10. Juli 2006 16:37 schrieb J.Westveer:

> ------- You are receiving this mail because: -------
> You are the assignee for the bug, or are watching the assignee.
>
> http://bugs.kde.org/show_bug.cgi?id=130525
>
>
>
>
> ------- Additional Comments From jwest melaque com  2006-07-10 16:37
> ------- I currently have exiv2-0.10.tar.gz   loaded .
>
> I just loaded exiv2 from cvs, made clean and recompiled.
> No Joy.  Same results.

I have the same problem, Gilles, with the latest exiv2 from svn, and the
problem existed at east for the last 5 revisions of exiv2.

Gerhard
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gerhard Kulzer]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From gerhard kulzer net  2006-07-10 17:58 -------
Am Montag, 10. Juli 2006 16:37 schrieb J.Westveer:
[bugs.kde.org quoted mail]
I have the same problem, Gilles, with the latest exiv2 from svn, and the
problem existed at east for the last 5 revisions of exiv2.

Gerhard
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From ahuggel gmx net  2006-07-10 18:20 -------
This is what exifprobe says for bad.jpg, note the "INVALID JPEG TAG":

 0x000614f=24911   :    <JPEG_APP13> 0xffed length 44828, Photoshop 3.0 - unknown format (not dumped: use -A)
-0x001106c=69740   :    </JPEG_APP13>
 0x001106d=69741   :    <TAG_0xb060> INVALID JPEG TAG
-0x04389e6=4426214 :  END OF FILE
 000000000=0       :  Start of JPEG (UNKNOWN JPEG compression) primary image [0x0] length 0 (CORRUPTED) (no image)

With the standalone exiv2 utility I can modify and write good.jpg (bot Exif and IPTC metadata) without problems.

Further, good.jpg (unmodified) starts with

00000000  ff d8 ff e1 d1 b1 45 78  69 66 00 00 4d 4d 00 2a  |ÿØÿáѱExif..MM.*|
00000010  00 00 00 08 00 0c 01 0e  00 02 00 00 00 1e 00 00  |................|

but bad.jpg starts with:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 00 00 01  |ÿØÿà..JFIF......|
00000010  00 01 00 00 ff e1 61 39  45 78 69 66 00 00 4d 4d  |....ÿáa9Exif..MM|

Exiv2 does not insert a JFIF segment, so this image was not written with Exiv2 (only).

Finally, with "dd skip=30 bs=1 if=bad.jpg of=bad.tif count=24881" I can extract the Exif data from bad.jpg and parse it with exiv2 -pt bad.tif and it looks ok.

Andreas
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From ahuggel gmx net  2006-07-10 18:20 -------
This is what exifprobe says for bad.jpg, note the "INVALID JPEG TAG":

 0x000614f=24911   :    <JPEG_APP13> 0xffed length 44828, Photoshop 3.0 - unknown format (not dumped: use -A)
-0x001106c=69740   :    </JPEG_APP13>
 0x001106d=69741   :    <TAG_0xb060> INVALID JPEG TAG
-0x04389e6=4426214 :  END OF FILE
 000000000=0       :  Start of JPEG (UNKNOWN JPEG compression) primary image [0x0] length 0 (CORRUPTED) (no image)

With the standalone exiv2 utility I can modify and write good.jpg (bot Exif and IPTC metadata) without problems.

Further, good.jpg (unmodified) starts with

00000000  ff d8 ff e1 d1 b1 45 78  69 66 00 00 4d 4d 00 2a  |ÿØÿáѱExif..MM.*|
00000010  00 00 00 08 00 0c 01 0e  00 02 00 00 00 1e 00 00  |................|

but bad.jpg starts with:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 00 00 01  |ÿØÿà..JFIF......|
00000010  00 01 00 00 ff e1 61 39  45 78 69 66 00 00 4d 4d  |....ÿáa9Exif..MM|

Exiv2 does not insert a JFIF segment, so this image was not written with Exiv2 (only).

Finally, with "dd skip=30 bs=1 if=bad.jpg of=bad.tif count=24881" I can extract the Exif data from bad.jpg and parse it with exiv2 -pt bad.tif and it looks ok.

Andreas
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From caulier.gilles free fr  2006-07-10 22:02 -------
Andreas,

I i understand what you mean, is this a bug in digiKam JPEGLoader implementation... right ?

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From ahuggel gmx net  2006-07-11 06:16 -------
Gilles,

This was just the information gathered from playing around with exiv2, without using digikam.
The question to further investigate is where does the INVALID TAG come from?
How does digikam write the IPTC segment and what does it do right after that?

Regards,
Andreas
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From caulier.gilles free fr  2006-07-11 14:01 -------
Andreas,

digiKam work like this :

- In editor, all image data/metadata are stored in memory using a DImg class instance.

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/dimgprivate.h?rev=545648&view=auto

- To save into a new JPEG file, a JEPGLoader::save() is called :

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/loaders/jpegloader.cpp?rev=557529&view=auto

JPEG image loader is derivated from generic DimgLoader loader class (a friend of DImg) :

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dimg/dimgloader.cpp?rev=557998&view=auto

- The JPEG Loader is simple and do not store the metadata in the new file. It create the the new and store the image data and the icc profile. The file is closed and the DImgLoader::saveMetadata() method is called to store Exif and IPTC (look at end of JPEGLoader::save() method.

- In fact the metadata are stored using Exiv2, by the famous DMetadata class (:=))). Do you remember ?

http://websvn.kde.org/trunk/extragear/graphics/digikam/libs/dmetadata/dmetadata.cpp?rev=556687&view=auto

I suspect a problem in DImgLoader::saveMetadata() calling DMetadata methods. Your viewpoint ?

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From kdebugs nacnud force9 co uk  2006-07-11 16:25 -------
For me, this appeared once I switched from Exiv2 0.9.1 to 0.10.  This would imply to me that it might be a change in how Exiv2 presents or writes some of the data (when integrated into digikam).
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From caulier.gilles free fr  2006-07-11 20:56 -------
Duncan report is interressing. I cannot test here to back on Exiv2 0.9.1. I'm not i'm home and my internet connection/time is limited during some day.

Somebody can confirm this point ?

Duncan, can you confirm this problem is relevant of the target JPEG file size ?

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From kdebugs nacnud force9 co uk  2006-07-12 15:38 -------
If I start with a 2.3 MB JPEG source, resize to 50% and re-save to a 385KB JPEG, the resulting JPEG is intact.

If I start with a 8.8 MB RAW file, and save to PNG, I get a 27 MB file with no tags at all (EXIF, Makernote or ITPC), but it's fine other than Exiv2 not liking it (digikam: Cannot load metadata using Exiv2 (/home/dhill/Images/2006/2006-07-10/z.png: The file contains data of an unknown image type))

If I start with a 8.8 MB RAW file, and save to JPEG with no changes, I get a 781 KB JPEG that's intact.

If I start with a -different- 8.8 MB RAW file, and save to JPEG with no changes, I get a corrupt JPEG.  If I resize to 50% I get a corrupt JPEG.

The correlation between corrupt and not corrupt appears to be the 'density' of colour data in an image.  The image that produces good JPEGs when saved is just a bowl of food.  The image that produces bad JPEGs when saved is a big yellow truck.

Resources to help test:
http://www.cricalix.net/~dhill/digikam/good_src.raw
http://www.cricalix.net/~dhill/digikam/good_src.jpg
http://www.cricalix.net/~dhill/digikam/bad_src.raw
http://www.cricalix.net/~dhill/digikam/bad_src.jpg
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From kdebugs nacnud force9 co uk  2006-07-12 16:06 -------
If I downgrade to Exiv2 0.9.1 using svn (by checking out the appropriate tag), and use the 'bad_src.raw' to generate a JPEG, I get a good JPEG.

0.10 is revision 846.
0.9.1 is revision 694.

I'll see if I can do a quick binary walk of the revisions to find where the problem kicks in (though anything related to compiling c++ is not quick, even on a Xeon).
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From kdebugs nacnud force9 co uk  2006-07-12 17:07 -------
Another data point - it's related to the embedding of ICC data into the JPEG.

Digikam          : 561569
Exiv2            : trunk 846
Source image     : bad_src.raw

Colour management: off
RAW to JPEG      : Intact, no corruption.
Colour management: on
RAW to JPEG      : Corrupted.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         
kdebugs nacnud force9 co uk changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kdebugs nacnud force9 co uk
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Duncan Hill-7]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From kdebugs nacnud force9 co uk  2006-07-12 17:45 -------
Digikam          : 561577
Exiv2            : trunk 750
Source image     : bad_src.raw
 
Colour management: off
RAW to JPEG      : Intact, no corruption.
Colour management: on
RAW to JPEG      : Corrupted.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         
marcel.wiesweg gmx de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
      everconfirmed|0                           |1



------- Additional Comments From marcel.wiesweg gmx de  2006-07-12 22:42 -------
I have tried some brute-force debugging on this one, using the "good.jpg" from above, which reproducably produces a broken JPEG here.

First, removing image->setIptcData in dmetaloader.cpp, line 145 would fix the bug.
Then (good guess) I removed meta.setImagePreview in dimginterface.cpp, line 518. So it seems it is related to the embedded image preview, at least reproducably on my machine with the given file.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[J.Westveer]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From jwest melaque com  2006-07-12 23:47 -------
ok. So it is not a "Size" related problem.  Now that you point it out, this file
was read in with digikam (svn) because it has IPTC data in it. If I would have just mounted and copied the file, and not modifyed it, it would NOT have had IPTC data.

So we have a jpg, that was created by loading with digikam, that CAN be read and printed by digicam and other programs, but for some reason it CAN NOT be saved, because something is wrong with the (good.jpg) file.

This does change the description of the bug, but a problem still exists if you load a file with digikam, and then later can not modify/re-save it.

Thoughts?
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Florencio Guimaraes-2]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=130525         




------- Additional Comments From florencio omeu com br  2006-07-13 06:16 -------
I think is a bug is Exiv2. Here, only when the IPTC image preview is larger than 65536 bytes the JPEG gets corrupted. As a (hack|workaround|test)?
changing line 1517 in
 
libs/dmetadata/dmetadata.cpp to:

preview.save(previewFile.name(), "JPEG", 50);

gives me always a smaller than 64K preview, and no corrupt JPEG.

And, BTW about comment #16 from J.Westveer: my original files doesn't have
any IPTC data.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel