[Digikam-devel] [Bug 134999] New: crash in exiv2 when searching for new images

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         
ahuggel gmx net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dseifert gmx de



------- Additional Comments From ahuggel gmx net  2006-12-04 02:20 -------
Daniel,
Please upgrade to Exiv2 0.12 and try again.
-ahu.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-04 08:44 -------
No difference with exiv2 0.12.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-2]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From caulier.gilles kdemail net  2006-12-04 09:14 -------
Take a care to uninstall properlly old Exiv2 0.11, clean up digiKam source code and recompile all (digiKam/DigiakmaImagePlugins/Kipi-plugins)... else you will see no difference. Of course, use RC1 source code tarball or better current implementation from KDE svn.

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From ahuggel gmx net  2006-12-04 09:38 -------
Ok thanks. Can you investigate further:

- what system are you runnning this on (32 or 64bit)?
- can you recreate this with a standalone 'exiv2 -pt *.jpg'?
- is the crash related to specific images?
- what cameras are the images from (what makernotes do they contain), all from Canon ixus?

-ahu.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-04 10:03 -------
Daniel, it seems you have the same problem. I haven't had the time yet to do a closer investigation (I really should since this should be fixed in 0.9 final in late December).

What I know now:
 - my filesystem (and glibc libraries) are ISO8859-1x. digiKam always crashes when it tries to scan folders that have non-ASCII characters.
 - it is not always the same folders that are scanned during the crash.
 - after the crash, 'free' function is not called and the kernel does not clear up the malloc'ed memory. I may lose several megabytes of free RAM with a single crash.
 - the crashes come quicker when there is less free RAM
 - the pictures than with Canon Ixus and transferred in Windows (then moved to Linux FS) may be a source of one problem
 - some pictures than with Canon Axx cameras and transferred via digiKam (all  versions since 0.8) are a cause for some malformed queries.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-04 10:53 -------
Gilles: I recompiled exiv2, digikam 0.9.0 RC1 and the plugins from scratch, so this was a clean install (Gentoo ebuilds). Still crashes, but read on.

Andreas:
 - I am running on 32bit.
 - I don't see a problem with exiv2 -pt * */* */*/* (my images are not all in the same directory), at least it does not end with a crash (I didn't really have the time to read through all the stdout (40MB), though). On stderr I didn't see anything related to a crash.
 - The crash is not related to specific images, at least the crash does not happen with the same images or even directories each time. In fact, I am now making progress in adding my images to digikam by copying one folder at a time (i.e. 10-50 images) and having digikam rescan them. I suspect that some images cause digikam to mess up something which manifests itself in a crash at a later time. I.e. I copied 8 directories with a total of 460 images into digikam's directory and digikam crashed when scanning them. When I copied+rescanned each directory individually, digikam did not crash.
 - The images are 99% from Canon Ixus (II and 400, I think)

The makernotes are a bit messed up, as I am switching from iPhoto (MacOS X) and when modifying the images in iPhoto the offsets become incorrect. I did add comments and keywords to most images using exiftool, but at the moment exiftool does not fix the offset (there should be an update soon).

The filenames and folders do not contain non-ascii characters (anymore).
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-04 13:05 -------
Update: after continuing to copy images in small steps to digikam, I got 7 of 8 GB to the database without problems. But now I am left with a number of images (many, but not all, copied over from Windows) that cause digiKam to output things like

digikam: WARNING: sqlite_step error: database disk image is malformed on query: REPLACE INTO Images ( caption , datetime, name, dirid )  VALUES ('64-Bit RAID-Karte','2004-08-09T16:05:37','IMG_0793.JPG',26)

and so on. Even adding one of these images now causes an error message, trying to add many will eventuall lead to a crash (see above). Removing the digikam3.db file and only adding one of these images does not give any warning, so it does not seem to be the image itself that causes the problem, it only seems to trigger another problem in the database.

This whole issue looks like at some point something is being messed up (in RAM and/or in the database) but only causes a problem later on. I don't have any ideas left what to do. If anyone wants me to try something, please let me know.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From marcel.wiesweg gmx de  2006-12-04 16:27 -------
Created an attachment (id=18781)
 --> (http://bugs.kde.org/attachment.cgi?id=18781&action=view)
Disable use of local codec

To test if the valgrind report about QLatin15Codec is the problem, please try
the attached patch, which disables the use of this codec.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-04 22:26 -------
I've compiled 0.9.0 rc1 from hand now (as opposed to using the ebuild) and added debug as well. I've applied your patch.

For the first test, I am running digikam in a directory with about 1700 images. Adding them in small steps works fine, there are no error messages regarding makernotes and no warnings/errors from sqlite on the console. Adding them all at once (i.e. removing digikam3.db and running digikam which will scan the directory) will reliably cause a crash with the unmodified 0.9.0 rc1 tarball.

When running the patched version on the same images, there is no crash.

Next, I copied all my images (6000 images, 8 GB) into digikam's directory, removed digicam3.db and started digikam. It took some time to scan, but there were neither sqlite warnings or errors about malformed disk images, and there is also no crash.

As far as I can tell, the patch solved the problem. Let me know if you want me to run further tests, but so far I am quite happy :-)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From ahuggel gmx net  2006-12-05 02:39 -------
Could this be related (reported for Qt 4.2.0 though)?

http://www.trolltech.com/developer/task-tracker/index_html?method=entry&id=135482

Does it make sense to try the workaround mentioned there to check?

-ahu.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From marcel.wiesweg gmx de  2006-12-05 22:42 -------
Created an attachment (id=18807)
 --> (http://bugs.kde.org/attachment.cgi?id=18807&action=view)
Use QLatin15codec code

Here is another patch (for debugging only). It uses the code of
QLatin15Codec::toUnicode to find out where exactly the crash happens. There is
also a debug output for each image, so expect large logs.
The question is, does the crash occur with this as well (the code is the same),
which line does valgrind point to, what is the debug output before the
crash/valgrind error.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-06 00:21 -------
Created an attachment (id=18809)
 --> (http://bugs.kde.org/attachment.cgi?id=18809&action=view)
Valgrind output

I've applied this change and ran digikam several times. I've attached a file
with the last few lines in each run's output. Please let me know if you want me
to do further tests.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-06 15:50 -------
Created an attachment (id=18816)
 --> (http://bugs.kde.org/attachment.cgi?id=18816&action=view)
another valgrind info with the latest QtLatin5codec patch

I patched digikam-0.9.0_rc1 with the latest debug patch and the --debug useflag
on Gentoo. Here is the output from valgrind digikam &>
digikam-QTpatch-valgrind.

The album path to scan is my whole photo collection (43 GB).
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-06 17:16 -------
GDB debug output with the patch over the collection is similar to what is already reported. digiKam scanned for about 10 minutes, then crashed. Here are the last lines:

digikam: WARNING: sqlite_step error: database disk image is malformed on query: REPLACE INTO Images ( caption , datetime, na
me, dirid )  VALUES ('','2005-09-22T09:34:55','IMG_9815.JPG',2243)
digikam: detectEncodingAndDecode 256 0
digikam: WARNING: sqlite_step error: database disk image is malformed on query: REPLACE INTO Images ( caption , datetime, na
me, dirid )  VALUES ('','2005-09-22T11:34:08','IMG_9816.JPG',2243)
digikam: detectEncodingAndDecode 256 0
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-06 17:31 -------
Created an attachment (id=18818)
 --> (http://bugs.kde.org/attachment.cgi?id=18818&action=view)
another valgrind debug

 $ time valgrind digikam &> digikam-QTpatch-valgrind2


real 1m57.633s
user 1m17.373s
sys 0m1.579s
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-06 17:46 -------
Created an attachment (id=18819)
 --> (http://bugs.kde.org/attachment.cgi?id=18819&action=view)
yet more debug info

I've removed digikam.db3 from the album root, but there are more digikam.db3s
inside some subfolders. Do they matter or should I remove them too? Below is
shown that digiKam hogs close to 200MBs of RAM that isn't freed. This is run
after a clean reboot.

 $ free
             total used    free     shared    buffers   cached
Mem:   1032996     506540  526456  0 27780   248400
-/+ buffers/cache:     230360  802636
Swap:    746980    0  746980
mikael planck ~ 18:41:22
 $ time valgrind digikam &> digikam-QTpatch-valgrind4

real 1m13.320s
user 1m9.043s
sys 0m1.056s
mikael planck ~ 18:42:38
 $ free
             total used    free     shared    buffers   cached
Mem:   1032996     683212  349784  0 28832   323008
-/+ buffers/cache:     331372  701624
Swap:    746980    0  746980
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From marcel.wiesweg gmx de  2006-12-06 20:14 -------
Thanks for your testing. All your debug logs seem to show the same problem.

What is happening before valgrind reports an error:

digikam: detectEncodingAndDecode 256 0
==28542==
==28542== Invalid write of size 2
==28542==    at 0x4325A3A: Digikam::DMetadata::detectEncodingAndDecode(std::string const&) (dmetadata.cpp:1225)

There is an image comment, a std::string object. The string says its length is 256, strlen says the length is 0, which means the first character is \0. This happens always before valgrind errors, for the "normal" comments, string::length() and strlen agree on the string length.
Then, in the code from QLatin15Codec, there seems to be a off-by-one problem. (It operates on ushort, 2 bytes). This code does not check the length parameter passed from digikam.
The Latin15 codec differs from Latin1 only in 8 of 256 values, so only if the bytes following the memory area have one of these values, they will be changed, and a crash seems to occur if some of glibc's memory management bytes are overwritten.

I will do some tests or think about what is the actual problem.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Mikael Lammentausta]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From mikael.lammentausta student savonia-amk fi  2006-12-06 21:47 -------
Here's additional information of my glibc, if this matters:

[U] sys-libs/glibc
     Installed versions:  2.3.4.20041102-r1(2.2)(20:50:47 19.01.2006)(-build -erandom -hardened -multilib nls -nomalloccheck -nptl -nptlonly -pic -selinux -userlocales)
     Homepage:            http://www.gnu.org/software/libc/libc.html
     Description:         GNU libc6 (also called glibc2) C library

 $ /lib/libc-2.3.4.so
GNU C Library 20041102 release version 2.3.4, by Roland McGrath et al.
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.3.5-20050130 (Gentoo 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie-8.7.7.1).
Compiled on a Linux 2.6.11 system on 2005-10-20.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        The C stubs add-on version 2.1.2.
        GNU Libidn by Simon Josefsson
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[dseifert]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From dseifert gmx de  2006-12-06 23:20 -------
Marcel, would it help you if I send you (where to?) an image which has this discrepancy in length? This image does not seem to have any comment.

I have other photos WITH comments that also cause this problem. They seem to be a bit "strange", sometimes not even displayed with exiv2, although after #26 they were imported just fine, it seems.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=134999         




------- Additional Comments From ahuggel gmx net  2006-12-07 02:30 -------
Please try this on top of Marcel's patch. It assumes c_str() only allocates memory up to the first 0 byte.

--- dmetadata.cpp       2006-12-07 09:05:02.000000000 +0800
+++ dmetadata2.cpp      2006-12-07 09:18:36.000000000 +0800
 @ -1195,8 +1195,8  @
     // Use whatever has the larger score, local or ASCII
     if (localScore >= 0 && localScore >= latin1Score)
     {
-        //return localCodec->toUnicode(value.c_str(), value.length());
-        const char *chars = value.c_str();
+        //return localCodec->toUnicode(value.data(), value.length());
+        const char *chars = value.data();
         uint len = value.length();

         // Code from QLatin15Codec::toUnicode

-ahu.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel