[Bug 307602] New: digiKam crashes, when starting up with minimum pic numerbs

[Axel Krebs]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #19 from Axel Krebs <[hidden email]> ---
Am 11.10.2012 20:54, schrieb Marcel Wiesweg:

> https://bugs.kde.org/show_bug.cgi?id=307602
>
> --- Comment #18 from Marcel Wiesweg <[hidden email]> ---
> All buffers are allocated in lines 461/2:
>             data.reset(new_failureTolerant(w * h * 4));
>             QScopedArrayPointer<uchar> strip(new_failureTolerant(w *
> rows_per_strip * 4));
> and checked later.
>
> Does it crash already on the first iteration, or sometime late during the
> decoding?
> Is it possible to get a valgrind trace?
>

Marcel,

you address to me. Unfortunatly, I am not able to analyse software. What
do you mean with "iteration"? I observed crashes, when digiKam was
idling som tim...


Axel

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|2.8.0                       |3.0.0

--- Comment #20 from Gilles Caulier <[hidden email]> ---
Marcel,

I will take a valgrind trace.

I will trying to put file on the web for you to download it. It's take time...

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #21 from Gilles Caulier <[hidden email]> ---
Marcel,

I shared with you in private tiff file through Google Drive web service. Do you
recieve the url ?

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #22 from Marcel Wiesweg <[hidden email]> ---
Axel: all this was for Gilles.
Gilles: Downloading the file

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Axel Krebs]

https://bugs.kde.org/show_bug.cgi?id=307602

Axel Krebs <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #23 from Axel Krebs <[hidden email]> ---
I've got a new camera D800 with 36 mio pixels.
It looks like large pixels-numbers can improve doing panoramas.

First observation: all takes huge space and _much_ more time.

Storing times in digiKam are _much_ longer, even when doing (only) small
modifictions in picture editor.
Maybe a new storage concept could overcome this potential handicap? I think
about
working with virtual pics-copies and when finishing work on pics, the
output will be written on disk- not before.

When working with tags (geotagging, e.g.), it feels _much_ slowlier than with
smaller pics (14, 3 as before). I do not understand why, as metadata should
scale constant (that is independent form pic size).

Crashes occour, when HUGIN as an external program finishes, as described
previously (bug #...?).

When finishing digiKam, I still get a crash event rather than simply closing
(bug #...?).

?? Do you need specific information, some minimal statistics, maybe?
Loadings times, storing times for small (7 mio)  medium size (14 mio)  and
large pics (30 mio)??

Have a nice Sunday-



Axel

My statistics right now:
digiKam version 2.8.0
Bilder:
BMP: 3
GIF: 99
JP2: 14
JPEG: 273
JPG: 118942
NIKON-NEF: 640
PGF: 2
PNG: 1608
RAW-CR2: 632
RAW-CRW: 11258
RAW-DNG: 7
RAW-NEF: 80755
TIFF: 1780
XCF: 1
Gesamt: 216014
:
:
Videos:
AVI: 60
MOV: 23
MPEG: 3
Gesamt: 86
:
:
Gesamtzahl der Einträge: 216100
Alben: 2222
Stichwörter: 109
Datenbanktreiber: QSQLITE

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #24 from Gilles Caulier <[hidden email]> ---
*** Bug 308572 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #25 from Marcel Wiesweg <[hidden email]> ---
==20316== Invalid write of size 1
==20316==    at 0x7C7DB9B: Digikam::TIFFLoader::load(QString const&,
Digikam::DImgLoaderObserver*) (tiffloader.cpp:548)
==20316==    by 0x7C5AA53: Digikam::DImg::load(QString const&, int,
Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) (dimg.cpp:449)
==20316==    by 0x7E2AD9D: Digikam::ThumbnailCreator::loadWithDImg(QString
const&, Digikam::IccProfile*) const (thumbnailcreator.cpp:561)
==20316==    by 0x7E2BED7:
Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect
const&, bool) const (thumbnailcreator.cpp:490)
==20316==    by 0x7E2F692: Digikam::ThumbnailCreator::load(QString const&,
QRect const&, bool) const (thumbnailcreator.cpp:260)
==20316==    by 0x7E304F1: Digikam::ThumbnailCreator::load(QString const&)
const (thumbnailcreator.cpp:199)
==20316==    by 0x7E3DEC6: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:170)
==20316==    by 0x7E1243D: Digikam::LoadSaveThread::run()
(loadsavethread.cpp:136)
==20316==    by 0x7E4FACD: Digikam::DynamicThread::DynamicThreadPriv::run()
(dynamicthread.cpp:186)
==20316==    by 0xB605A9C: QThreadPoolThread::run() (qthreadpool.cpp:107)
==20316==    by 0xB611E8A: QThreadPrivate::start(void*) (qthread_unix.cpp:307)
==20316==    by 0xBA6BE0D: start_thread (in /lib64/libpthread-2.15.so)
==20316==  Address 0xeb502a86 is 2 bytes after a block of size 903,703,108
alloc'd
==20316==    at 0x4C2A147: operator new[](unsigned long) (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==20316==    by 0x7C76E21: Digikam::DImgLoader::new_failureTolerant(unsigned
long) (dimgloader.h:147)
==20316==    by 0x7C7D1A7: Digikam::TIFFLoader::load(QString const&,
Digikam::DImgLoaderObserver*) (tiffloader.cpp:461)
==20316==    by 0x7C5AA53: Digikam::DImg::load(QString const&, int,
Digikam::DImgLoaderObserver*, Digikam::DRawDecoding const&) (dimg.cpp:449)
==20316==    by 0x7E2AD9D: Digikam::ThumbnailCreator::loadWithDImg(QString
const&, Digikam::IccProfile*) const (thumbnailcreator.cpp:561)
==20316==    by 0x7E2BED7:
Digikam::ThumbnailCreator::createThumbnail(Digikam::ThumbnailInfo const&, QRect
const&, bool) const (thumbnailcreator.cpp:490)
==20316==    by 0x7E2F692: Digikam::ThumbnailCreator::load(QString const&,
QRect const&, bool) const (thumbnailcreator.cpp:260)
==20316==    by 0x7E304F1: Digikam::ThumbnailCreator::load(QString const&)
const (thumbnailcreator.cpp:199)
==20316==    by 0x7E3DEC6: Digikam::ThumbnailLoadingTask::execute()
(thumbnailtask.cpp:170)
==20316==    by 0x7E1243D: Digikam::LoadSaveThread::run()
(loadsavethread.cpp:136)
==20316==    by 0x7E4FACD: Digikam::DynamicThread::DynamicThreadPriv::run()
(dynamicthread.cpp:186)
==20316==    by 0xB605A9C: QThreadPoolThread::run() (qthreadpool.cpp:107)

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Axel Krebs]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #26 from Axel Krebs <[hidden email]> ---
Am 17.10.2012 21:00, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
>
> --- Comment #24 from Gilles Caulier <[hidden email]> ---
> *** Bug 308572 has been marked as a duplicate of this bug. ***
>
How can bug #308572 be "duplicate of bug #307602", if the 1 GB pics have
been taken away from my pics collection?


Axel

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #27 from Gilles Caulier <[hidden email]> ---
Perhaps another image has same dysfunction. At least, it's always the same
libtiff error...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Axel Krebs]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #28 from Axel Krebs <[hidden email]> ---
Am 18.10.2012 10:28, schrieb Gilles Caulier:
> https://bugs.kde.org/show_bug.cgi?id=307602
>
> --- Comment #27 from Gilles Caulier <[hidden email]> ---
> Perhaps another image has same dysfunction. At least, it's always the same
> libtiff error...
>
> Gilles Caulier
>
If problem is understood, it should be possible to find suspicious files
which cause this type of errors inversely?

And, if, digiKam could (and should!!) avoid "touching those files". Right?


Axel

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #29 from Gilles Caulier <[hidden email]> ---
For me the crash appear when files are scanned. So files are not touched by
digiKam. Scan is read only.

If we found the problem in Tiff loader (used to scan file properties, load
thumbnail, and load image, we can avoid (corrupted ?) tiff items to prevent
future problem.

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #30 from Marcel Wiesweg <[hidden email]> ---
This was a tricky bug. The information from valgrind gives the clue: "a block
of size 903,703,108". In fact, the image needs about 5,9 GB of memory. This
memory is computed: w*h*4. And the TIFF loader uses a 32 bit type for storing
width and height!

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

Marcel Wiesweg <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
   Version Fixed In|                            |3.0.0
         Resolution|---                         |FIXED
      Latest Commit|                            |http://commits.kde.org/digi
                   |                            |kam/767a14c626384d646276972
                   |                            |c531e9a617e36face

--- Comment #31 from Marcel Wiesweg <[hidden email]> ---
Git commit 767a14c626384d646276972c531e9a617e36face by Marcel Wiesweg.
Committed on 18/10/2012 at 22:18.
Pushed by mwiesweg into branch 'master'.

Prevent integer overflows when calculating the memory necessary for loading an
image.

This can happen particularly for images requiring chunks of >4GB and when using
a 32 bit
variable to store width and height. Worst case, a smaller chunk of memory was
allocated,
resulting in a crash.
FIXED-IN: 3.0.0

M  +5    -1    NEWS
M  +10   -0    libs/dimg/loaders/dimgloader.cpp
M  +21   -0    libs/dimg/loaders/dimgloader.h
M  +2    -2    libs/dimg/loaders/jp2kloader.cpp
M  +1    -1    libs/dimg/loaders/jpegloader.cpp
M  +2    -2    libs/dimg/loaders/pgfloader.cpp
M  +2    -2    libs/dimg/loaders/pngloader.cpp
M  +1    -1    libs/dimg/loaders/qimageloader.cpp
M  +2    -2    libs/dimg/loaders/rawloader.cpp
M  +3    -3    libs/dimg/loaders/tiffloader.cpp

http://commits.kde.org/digikam/767a14c626384d646276972c531e9a617e36face

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

Marcel Wiesweg <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|FIXED                       |---

--- Comment #32 from Marcel Wiesweg <[hidden email]> ---
A sidenote: It's clear that digikam should not crash here, but it is not
designed with such large files in mind. It will probably work with a 64bit
Linux and sufficient physical memory, but we still allocate this as one
contiguous chunk, which can be a problem even with enough free memory. For
proper support of such large files, tiled memory allocation would be necessary
which we dont support.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #33 from Gilles Caulier <[hidden email]> ---
Marcel,

Qt API do not provide any storage container to :

1/ compress data to optimize memory allocation of RAW data ?
2/ tile access of data from a file stream ?

For 1/, if we use QByteArray into DImg to store image data instead a uchar
pointer, we can compress it :

http://qt-project.org/doc/qt-4.8/qbytearray.html#qCompress

QByteArray will also prevent memory allocation problem...

For 2/, i have no idea...

Gilles

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #34 from Marcel Wiesweg <[hidden email]> ---
If we change the data layout of DImg, we need to adapt everything access the
image data, most notably the image filters. It's a bit of work, but possible.
I believe the path to choose would be to encapsulate the added complexity in an
iterator class:
The underlying data storage in memory is intransparent through the DImg class.
The method returning an uchar* are removed, instead they return an Iterator
class. This iterator offers a few accessors:
uchar* row(int r), uchar* operator[](uint byte). This must be very carefully
designed as it has enormous performance impact. I believe Krita has adopted a
similar scheme, dont know how complex it is implemented.
Regarding in-memory compression, which also requires an extra access layer like
described above, there are well-suited algorithms available, most notably LZO:
http://www.oberhumer.com/opensource/lzo/
We would need to verify that this algorithm is also well-suited for our special
case of uncompressed, natural image data.

For both tiling and in-memory compression, we'd need to support inside the
implementation to use it or not use it for a DImg, depending on the use case
(small image vs. very large image, larger image but loaded to image editor,
smaller image but stored in cache etc.)

Interesting project, and needs benchmarking all along the way.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Christoph Feck]

https://bugs.kde.org/show_bug.cgi?id=307602

Christoph Feck <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #35 from Christoph Feck <[hidden email]> ---
LZ does not compress well with natural images; you can expect 90% of original
file size. They are fast to decompress, but only because they do "memcpy" 90%
of the time. For benchmarks of photo compression, see
http://www.imagecompression.info/gralic/LPCB.html

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #36 from Gilles Caulier <[hidden email]> ---
Thanks Christoph for this benchmark.

From my side, i have experimented some format. To compress image the best ratio
will be given by Wavelets algorithm in lossless :

- PGF : need to be improved around OpenMP support. Good candidate (GPL)
- Jasper : JPEG 2000 : good but very time consuming due to floating
computation. Not optimized and now, not maintained (GPL). Don't use...
- OpenJPEG : another JPEG 2000 implementation. Very well maintained, but not
yet optimized. In progress. (GPL).
- WebP : from Google, to replace JPEG on web. fast and Open-Source like
compatible, but patents still active ?
- JPEG_XR : from Microsoft. fast and optimized, but not open source compatible.
Re-usable only. 20 active patents... Note that it's have been selected by "JPEG
organization" to replace current JPEG in digital still cameras. The war is open
(:=)))... In all case, to ignore here...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #37 from Marcel Wiesweg <[hidden email]> ---
We are talking about in-memory compression, which means the uncompressed image
data is recompressed to save memory, but needs to be uncompressed each time the
image data is used (viewing, editing). That means the performance must be
suitable. Any sophisticated compression will be so slow that it cannot play a
role; LZO would be acceptable in speed, but obviously does not give any usable
benefit with our use case. Here comes ImageZero, which will give a benefit
regarding compression.
The performance number given is 180MB/s for compression or decompression.
A 25 MPx image in 16 bit needs roughly these 180 MB of memory.
There are real-life situations where applying this compression to images in our
in-memory cache could give us a benefit, but there are as many situations to be
expected where we win nothing.
Regarding the problem of extremely large images: Axel's example here would need
30 seconds for compression, and another 30 seconds for decompression.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Christoph Feck]

https://bugs.kde.org/show_bug.cgi?id=307602

--- Comment #38 from Christoph Feck <[hidden email]> ---
The "180 MB/s" figure is missleading; it should probably say "50 cycles per
pixel". If your pixels are 48 bit rather than 24 bit, you would get 300 MB/s
per core on a 2.5 gigacycles/s CPU.

On the other hand, 48 bit pixels are not as compressible as 24 bit pixels. You
probably would end up compressing only the higher 8 bits per component, and
store the lower 8 bits uncompressed. But I never did any tests with 48 bit, due
to lack of representative data.

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel