digiKam › digikam-users

Failing to load collection

Classic

List

Threaded

10 messages Options

Sigfrid Lundberg

Failing to load collection

Dear everybody,

I'd like first say, warmly, thanks for an excellent product. I've been a daily user since about 10 years.

I've a collection of close to 10000 raw images *.DNG, *.ORF and *.RW2 formats. DNG is about 75% of these. I've kept about 10% of the photos I've made. digikam has been essential in weeding and curating the collection (from about 100000 to 10000), and writing captions, applying tags/keywords and geo-tagging the all items.

I've always ensured that data is written back to the exif header and that there are xmp files in the file system. The collection is stored in a file system which is also a part of my Dropbox area.

I've share the same collection over more than one computer, but then had the sqlite database files in directories outside Dropbox. Worked without problem.

A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to 16.04 LTS, and I decided to allow that. When doing that, I felt that it would be time to go to digikam 5.5.0. It just didn't work.

I've

1. Installed the appimage (5.5.0 and 5.6.0)

2. Installed the digikam5 version for ubuntu according to tutorials on the Net.

3. I've compiled it from sources (been through all the cmake and dependency manage stuff), and got executable binaries.

finally I

4. Downgraded to digikam 4 again.

Sorry, but it doesn't work. There is no difference in having the database files inside our outside the collections. Not even downgrading helped.

What happens in all four cases is that digikam reads about 25% of the collection, and then it sighs deeply, printing to STDOUT what I've added below (seen people mentioning similar problems on mailing lists, but I've not found a clear answer).

The raw image in question is perfectly readable:

https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0

Apart from dropbox (which actually make the raw processing without problems), I have successfully read this image using gimp and rawtherapee, so the file shouldn't be corrupt. After failing on that one, the process won't recover. The corresponding xmp file is well formed according to xmllint.

Closing digikam, and doing dump and refresh of the database using sqlite3 command line tool doesn't help.
It still tries to reread the collection and ends at the same item. If I move this image outside the collection, remove the database files and start over it ends the same way for some other item.

Right now I'm desperate since my work-flow is gone.

Any clue, anyone?

Thanks in advance

Sigfrid

digikam.metaengine: DateTime (Exif digitalized): tors dec. 22 20:37:23 2011
digikam.metaengine: Orientation => Exif.Image.Orientation => 1
digikam.database: Scanning took 2 ms
digikam.database: Finishing took 0 ms
digikam.dimg: "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG" : RAW file identified
digikam.geoiface: ----
digikam.geoiface: ----
digikam.general: Using 4 CPU core to run threads
digikam.general: Stacked View Mode : 0
digikam.general: Action Thread run 1 new jobs
digikam.dbengine: Database is locked. Waited 0
digikam.geoiface: ----
digikam.dbengine: Database is locked. Waited 250
digikam.dbengine: Database is locked. Waited 500
digikam.dbengine: Database is locked. Waited 750

....

digikam.dbengine: Database is locked. Waited 9500
digikam.dbengine: Database is locked. Waited 9750
digikam.dbengine: Database is locked. Waited 10000
digikam.dbengine: Detected locked database file. There is an active transaction. Waited but giving up now.
digikam.dbengine: Failure executing query:
"SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
Error messages: "Unable to fetch row" "database table is locked: Albums" 6 1
Bound values: (QVariant(int, 1), QVariant(QString, "/200708-200712"))
digikam.general: Data From DBJobsThread is null: true
digikam.general: Cancel Main Thread
digikam.general: One job is done
^C

Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/

Gilles Caulier-4

Re: Failing to load collection

Hi,

I'm not sure if the DNG file identified is really the problem.

But it's clear that something is wrong with a file scanned to populate
the database.

I suspect a problem with Exiv2, which is mostly used to scan image
properties in background.

So the best way to identify the problem, especially the image file, is :

1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
2/ start a clean instance of digiKam from scratch.
3/ Import step by step albums. Not all in one step.
4/ When the album containing the image freeze digiKam, share the datat
somewhere on the web to test here in local.

Note : if there is a crash in the thread due to Exiv2, the crash will
not close the application, but we can catch it to identify the
problem. Run the AppImage from the console with the "debug" argument.
This will run digiKam in GDB. Run AppImage with "help" argument for
details.

Best

Gilles Caulier

2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg <[hidden email]>:

>
> Dear everybody,
>
> I'd like first say, warmly, thanks for an excellent product. I've been a
> daily user since about 10 years.
>
> I've a collection of close to 10000 raw images *.DNG, *.ORF and *.RW2
> formats. DNG is about 75% of these. I've kept about 10% of the photos I've
> made. digikam has been essential in weeding and curating the collection
> (from about 100000 to 10000), and writing captions, applying tags/keywords
> and geo-tagging the all items.
>
> I've always ensured that data is written back to the exif header and that
> there are xmp files in the file system. The collection is stored in a file
> system which is also a part of my Dropbox area.
> I've share the same collection over more than one computer, but then had the
> sqlite database files in directories outside Dropbox. Worked without
> problem.
>
> A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to 16.04
> LTS, and I decided to allow that. When doing that, I felt that it would be
> time to go to digikam 5.5.0. It just didn't work.
>
> I've
>
> 1. Installed the appimage (5.5.0 and 5.6.0)
> 2. Installed the digikam5 version for ubuntu according to tutorials on the
> Net.
> 3. I've compiled it from sources (been through all the cmake and dependency
> manage stuff), and got executable binaries.
>
> finally I
>
> 4. Downgraded to digikam 4 again.
>
> Sorry, but it doesn't work. There is no difference in having the database
> files inside our outside the collections. Not even downgrading helped.
>
> What happens in all four cases is that digikam reads about 25% of the
> collection, and then it sighs deeply, printing to STDOUT what I've added
> below (seen people mentioning similar problems on mailing lists, but I've
> not found a clear answer).
>
> The raw image in question is perfectly readable:
>
> https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>
> Apart from dropbox (which actually make the raw processing without
> problems), I have successfully read this image using gimp and rawtherapee,
> so the file shouldn't be corrupt. After failing on that one, the process
> won't recover. The corresponding xmp file is well formed according to
> xmllint.
>
> Closing digikam, and doing dump and refresh of the database using sqlite3
> command line tool doesn't help.
> It still tries to reread the collection and ends at the same item. If I move
> this image outside the collection, remove the database files and start over
> it ends the same way for some other item.
>
> Right now I'm desperate since my work-flow is gone.
>
> Any clue, anyone?
>
> Thanks in advance
>
> Sigfrid
>
>
>
>
> digikam.metaengine: DateTime (Exif digitalized): tors dec. 22 20:37:23 2011
> digikam.metaengine: Orientation => Exif.Image.Orientation => 1
> digikam.database: Scanning took 2 ms
> digikam.database: Finishing took 0 ms
> digikam.dimg: "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG" :
> RAW file identified
> digikam.geoiface: ----
> digikam.geoiface: ----
> digikam.general: Using 4 CPU core to run threads
> digikam.general: Stacked View Mode : 0
> digikam.general: Action Thread run 1 new jobs
> digikam.dbengine: Database is locked. Waited 0
> digikam.geoiface: ----
> digikam.dbengine: Database is locked. Waited 250
> digikam.dbengine: Database is locked. Waited 500
> digikam.dbengine: Database is locked. Waited 750
>
> ....
>
> digikam.dbengine: Database is locked. Waited 9500
> digikam.dbengine: Database is locked. Waited 9750
> digikam.dbengine: Database is locked. Waited 10000
> digikam.dbengine: Detected locked database file. There is an active
> transaction. Waited but giving up now.
> digikam.dbengine: Failure executing query:
> "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
> Error messages: "Unable to fetch row" "database table is locked: Albums" 6 1
> Bound values: (QVariant(int, 1), QVariant(QString, "/200708-200712"))
> digikam.general: Data From DBJobsThread is null: true
> digikam.general: Cancel Main Thread
> digikam.general: One job is done
> ^C
>
>
>
>
> --
> Sigfrid Lundberg, Ph.D., System developer
> Lund, Sweden
> http://sigfrid-lundberg.se/

Sigfrid Lundberg

Re: Failing to load collection

Thanks for rapid response... Starting with digikam-5.6.0-01-x86-64.appimage I did with debug option

1. I created an empty directory (~sigge/digikam) for the db

2 Tried with the very first directory inside my gallery area, which contains a some 120 JPGs.

I then got to 85% of that directory, with the following output from GDB and digikam:

digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG" : JPEG file identified
digikam.database: Adding new item "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal => QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:43:57 2012
digikam.metaengine: Orientation => Exif.Image.Orientation => 8
digikam.database: Scanning took 10 ms
digikam.database: Finishing took 0 ms
digikam.metaengine: Exiv2 ( 2 ) : Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.

digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG" : JPEG file identified
digikam.database: Adding new item "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal => QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:44:02 2012
digikam.metaengine: Orientation => Exif.Image.Orientation => 8
digikam.database: Scanning took 12 ms
digikam.database: Finishing took 0 ms
[Switching to Thread 0x7fffe1308700 (LWP 16323)]

Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown), 0x00007fffee1848bd in __cxa_throw ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb)

I still have it running, so if you have suggestions for next step, I can continue!

Sigge

On 6 June 2017 at 14:54, Gilles Caulier <[hidden email]> wrote:

Hi,

I'm not sure if the DNG file identified is really the problem.

But it's clear that something is wrong with a file scanned to populate
the database.

I suspect a problem with Exiv2, which is mostly used to scan image
properties in background.

So the best way to identify the problem, especially the image file, is :

1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
2/ start a clean instance of digiKam from scratch.
3/ Import step by step albums. Not all in one step.
4/ When the album containing the image freeze digiKam, share the datat
somewhere on the web to test here in local.

Note : if there is a crash in the thread due to Exiv2, the crash will
not close the application, but we can catch it to identify the
problem. Run the AppImage from the console with the "debug" argument.
This will run digiKam in GDB. Run AppImage with "help" argument for
details.

Best

Gilles Caulier

2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>
> Dear everybody,
>
> I'd like first say, warmly, thanks for an excellent product. I've been a
> daily user since about 10 years.
>
> I've a collection of close to 10000 raw images *.DNG, *.ORF and *.RW2
> formats. DNG is about 75% of these. I've kept about 10% of the photos I've
> made. digikam has been essential in weeding and curating the collection
> (from about 100000 to 10000), and writing captions, applying tags/keywords
> and geo-tagging the all items.
>
> I've always ensured that data is written back to the exif header and that
> there are xmp files in the file system. The collection is stored in a file
> system which is also a part of my Dropbox area.
> I've share the same collection over more than one computer, but then had the
> sqlite database files in directories outside Dropbox. Worked without
> problem.
>
> A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to 16.04
> LTS, and I decided to allow that. When doing that, I felt that it would be
> time to go to digikam 5.5.0. It just didn't work.
>
> I've
>
> 1. Installed the appimage (5.5.0 and 5.6.0)
> 2. Installed the digikam5 version for ubuntu according to tutorials on the
> Net.
> 3. I've compiled it from sources (been through all the cmake and dependency
> manage stuff), and got executable binaries.
>
> finally I
>
> 4. Downgraded to digikam 4 again.
>
> Sorry, but it doesn't work. There is no difference in having the database
> files inside our outside the collections. Not even downgrading helped.
>
> What happens in all four cases is that digikam reads about 25% of the
> collection, and then it sighs deeply, printing to STDOUT what I've added
> below (seen people mentioning similar problems on mailing lists, but I've
> not found a clear answer).
>
> The raw image in question is perfectly readable:
>
> https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>
> Apart from dropbox (which actually make the raw processing without
> problems), I have successfully read this image using gimp and rawtherapee,
> so the file shouldn't be corrupt. After failing on that one, the process
> won't recover. The corresponding xmp file is well formed according to
> xmllint.
>
> Closing digikam, and doing dump and refresh of the database using sqlite3
> command line tool doesn't help.
> It still tries to reread the collection and ends at the same item. If I move
> this image outside the collection, remove the database files and start over
> it ends the same way for some other item.
>
> Right now I'm desperate since my work-flow is gone.
>
> Any clue, anyone?
>
> Thanks in advance
>
> Sigfrid
>
>
>
>
> digikam.metaengine: DateTime (Exif digitalized): tors dec. 22 20:37:23 2011
> digikam.metaengine: Orientation => Exif.Image.Orientation => 1
> digikam.database: Scanning took 2 ms
> digikam.database: Finishing took 0 ms
> digikam.dimg: "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG" :
> RAW file identified
> digikam.geoiface: ----
> digikam.geoiface: ----
> digikam.general: Using 4 CPU core to run threads
> digikam.general: Stacked View Mode : 0
> digikam.general: Action Thread run 1 new jobs
> digikam.dbengine: Database is locked. Waited 0
> digikam.geoiface: ----
> digikam.dbengine: Database is locked. Waited 250
> digikam.dbengine: Database is locked. Waited 500
> digikam.dbengine: Database is locked. Waited 750
>
> ....
>
> digikam.dbengine: Database is locked. Waited 9500
> digikam.dbengine: Database is locked. Waited 9750
> digikam.dbengine: Database is locked. Waited 10000
> digikam.dbengine: Detected locked database file. There is an active
> transaction. Waited but giving up now.
> digikam.dbengine: Failure executing query:
> "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
> Error messages: "Unable to fetch row" "database table is locked: Albums" 6 1
> Bound values: (QVariant(int, 1), QVariant(QString, "/200708-200712"))
> digikam.general: Data From DBJobsThread is null: true
> digikam.general: Cancel Main Thread
> digikam.general: One job is done
> ^C
>
>
>
>
> --
> Sigfrid Lundberg, Ph.D., System developer
> Lund, Sweden
> http://sigfrid-lundberg.se/

Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/

Gilles Caulier-4

Re: Failing to load collection

Well it simple. digiKam is stopped in debugger due to a C++ exception.
You get the GDB prompt and it wait a command to hack.

Just enter "bt" for backtrace to catch the full backtrace, and
especially where the program is stopped in source code.

The notice is explained in details here :

https://www.digikam.org/contribute/

Note : I'm 90% sure that you have a problem with Exiv2 with a DNG
file... (:=)))...

Gilles Caulier

2017-06-06 15:43 GMT+02:00 Sigfrid Lundberg <[hidden email]>:

> Thanks for rapid response... Starting with digikam-5.6.0-01-x86-64.appimage
> I did with debug option
>
> 1. I created an empty directory (~sigge/digikam) for the db
> 2 Tried with the very first directory inside my gallery area, which contains
> a some 120 JPGs.
>
> I then got to 85% of that directory, with the following output from GDB and
> digikam:
>
> digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG" :
> JPEG file identified
> digikam.database: Adding new item
> "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
> digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
> QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
> digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:43:57 2012
> digikam.metaengine: Orientation => Exif.Image.Orientation => 8
> digikam.database: Scanning took 10 ms
> digikam.database: Finishing took 0 ms
> digikam.metaengine: Exiv2 ( 2 ) : Directory OlympusCs, entry 0x0101: Strip
> 0 is outside of the data area; ignored.
>
> digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG" :
> JPEG file identified
> digikam.database: Adding new item
> "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
> digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
> QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
> digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:44:02 2012
> digikam.metaengine: Orientation => Exif.Image.Orientation => 8
> digikam.database: Scanning took 12 ms
> digikam.database: Finishing took 0 ms
> [Switching to Thread 0x7fffe1308700 (LWP 16323)]
>
> Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
> 0x00007fffee1848bd in __cxa_throw ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (gdb)
>
>
> I still have it running, so if you have suggestions for next step, I can
> continue!
>
> Sigge
>
>
>
>
> On 6 June 2017 at 14:54, Gilles Caulier <[hidden email]> wrote:
>>
>> Hi,
>>
>> I'm not sure if the DNG file identified is really the problem.
>>
>> But it's clear that something is wrong with a file scanned to populate
>> the database.
>>
>> I suspect a problem with Exiv2, which is mostly used to scan image
>> properties in background.
>>
>> So the best way to identify the problem, especially the image file, is :
>>
>> 1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
>> 2/ start a clean instance of digiKam from scratch.
>> 3/ Import step by step albums. Not all in one step.
>> 4/ When the album containing the image freeze digiKam, share the datat
>> somewhere on the web to test here in local.
>>
>> Note : if there is a crash in the thread due to Exiv2, the crash will
>> not close the application, but we can catch it to identify the
>> problem. Run the AppImage from the console with the "debug" argument.
>> This will run digiKam in GDB. Run AppImage with "help" argument for
>> details.
>>
>> Best
>>
>> Gilles Caulier
>>
>>
>>
>> 2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>> >
>> > Dear everybody,
>> >
>> > I'd like first say, warmly, thanks for an excellent product. I've been a
>> > daily user since about 10 years.
>> >
>> > I've a collection of close to 10000 raw images *.DNG, *.ORF and *.RW2
>> > formats. DNG is about 75% of these. I've kept about 10% of the photos
>> > I've
>> > made. digikam has been essential in weeding and curating the collection
>> > (from about 100000 to 10000), and writing captions, applying
>> > tags/keywords
>> > and geo-tagging the all items.
>> >
>> > I've always ensured that data is written back to the exif header and
>> > that
>> > there are xmp files in the file system. The collection is stored in a
>> > file
>> > system which is also a part of my Dropbox area.
>> > I've share the same collection over more than one computer, but then had
>> > the
>> > sqlite database files in directories outside Dropbox. Worked without
>> > problem.
>> >
>> > A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to
>> > 16.04
>> > LTS, and I decided to allow that. When doing that, I felt that it would
>> > be
>> > time to go to digikam 5.5.0. It just didn't work.
>> >
>> > I've
>> >
>> > 1. Installed the appimage (5.5.0 and 5.6.0)
>> > 2. Installed the digikam5 version for ubuntu according to tutorials on
>> > the
>> > Net.
>> > 3. I've compiled it from sources (been through all the cmake and
>> > dependency
>> > manage stuff), and got executable binaries.
>> >
>> > finally I
>> >
>> > 4. Downgraded to digikam 4 again.
>> >
>> > Sorry, but it doesn't work. There is no difference in having the
>> > database
>> > files inside our outside the collections. Not even downgrading helped.
>> >
>> > What happens in all four cases is that digikam reads about 25% of the
>> > collection, and then it sighs deeply, printing to STDOUT what I've added
>> > below (seen people mentioning similar problems on mailing lists, but
>> > I've
>> > not found a clear answer).
>> >
>> > The raw image in question is perfectly readable:
>> >
>> > https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>> >
>> > Apart from dropbox (which actually make the raw processing without
>> > problems), I have successfully read this image using gimp and
>> > rawtherapee,
>> > so the file shouldn't be corrupt. After failing on that one, the process
>> > won't recover. The corresponding xmp file is well formed according to
>> > xmllint.
>> >
>> > Closing digikam, and doing dump and refresh of the database using
>> > sqlite3
>> > command line tool doesn't help.
>> > It still tries to reread the collection and ends at the same item. If I
>> > move
>> > this image outside the collection, remove the database files and start
>> > over
>> > it ends the same way for some other item.
>> >
>> > Right now I'm desperate since my work-flow is gone.
>> >
>> > Any clue, anyone?
>> >
>> > Thanks in advance
>> >
>> > Sigfrid
>> >
>> >
>> >
>> >
>> > digikam.metaengine: DateTime (Exif digitalized): tors dec. 22 20:37:23
>> > 2011
>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 1
>> > digikam.database: Scanning took 2 ms
>> > digikam.database: Finishing took 0 ms
>> > digikam.dimg: "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG"
>> > :
>> > RAW file identified
>> > digikam.geoiface: ----
>> > digikam.geoiface: ----
>> > digikam.general: Using 4 CPU core to run threads
>> > digikam.general: Stacked View Mode : 0
>> > digikam.general: Action Thread run 1 new jobs
>> > digikam.dbengine: Database is locked. Waited 0
>> > digikam.geoiface: ----
>> > digikam.dbengine: Database is locked. Waited 250
>> > digikam.dbengine: Database is locked. Waited 500
>> > digikam.dbengine: Database is locked. Waited 750
>> >
>> > ....
>> >
>> > digikam.dbengine: Database is locked. Waited 9500
>> > digikam.dbengine: Database is locked. Waited 9750
>> > digikam.dbengine: Database is locked. Waited 10000
>> > digikam.dbengine: Detected locked database file. There is an active
>> > transaction. Waited but giving up now.
>> > digikam.dbengine: Failure executing query:
>> > "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
>> > Error messages: "Unable to fetch row" "database table is locked: Albums"
>> > 6 1
>> > Bound values: (QVariant(int, 1), QVariant(QString, "/200708-200712"))
>> > digikam.general: Data From DBJobsThread is null: true
>> > digikam.general: Cancel Main Thread
>> > digikam.general: One job is done
>> > ^C
>> >
>> >
>> >
>> >
>> > --
>> > Sigfrid Lundberg, Ph.D., System developer
>> > Lund, Sweden
>> > http://sigfrid-lundberg.se/
>
>
>
>
> --
> Sigfrid Lundberg, Ph.D., System developer
> Lund, Sweden
> http://sigfrid-lundberg.se/

Sigfrid Lundberg

Re: Failing to load collection

That directory is JPG only, taken with an Olympus, no DNG

Exceptions I do understand. So close to java ;^)

GDB supports the exiv2 hypothesis, though. GDB claims the following

Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown), 0x00007fffee1848bd in __cxa_throw ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0 0x00007fffee1848bd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x0000003455b53b15 in Exiv2::ImageFactory::open (path=..., useCurl=<optimized out>)
    at /b/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/exiv2-trunk/src/image.cpp:848
#2 0x00007ffff6775c71 in Digikam::MetaEngine::load (this=this@entry=0x7fffd4063c60, filePath=...)
    at /b/dktemp/digikam-master/core/libs/dmetadata/metaengine.cpp:281
#3 0x00007ffff67be5c6 in Digikam::DMetadata::load (this=this@entry=0x7fffd4063c60, filePath=...)
    at /b/dktemp/digikam-master/core/libs/dmetadata/dmetadata.cpp:96
#4 0x00007fffee59a3fa in Digikam::ImageScanner::loadFromDisk (this=this@entry=0x7fffe13077e0)
    at /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:1563
#5 0x00007fffee59a590 in Digikam::ImageScanner::newFile (this=this@entry=0x7fffe13077e0, albumId=2)
    at /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:289
#6 0x00007fffee4dbb7e in Digikam::CollectionScanner::scanNewFile (this=this@entry=0x7fffe1307bf0,
    info=..., albumId=2)
    at /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1283
#7 0x00007fffee4dd488 in Digikam::CollectionScanner::scanAlbum (this=this@entry=0x7fffe1307bf0,
    location=..., album=...)
    at /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1106
#8 0x00007fffee4ddaf9 in Digikam::CollectionScanner::scanAlbumRoot (
    this=this@entry=0x7fffe1307bf0, location=...)
    at /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:835
#9 0x00007fffee4ddd26 in Digikam::CollectionScanner::completeScan (this=this@entry=0x7fffe1307bf0)
    at /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:488
#10 0x00007ffff74f485e in Digikam::ScanController::run (this=
    0x7ffff7dd4da0 <_ZZN7Digikam12_GLOBAL__N_113Q_QGS_creator13innerFunctionEvE6holder>)
    at /b/dktemp/digikam-master/core/libs/database/utils/scancontroller.cpp:708
#11 0x00000034890aef49 in ?? () from /tmp/.mount_sIj2qg/usr/lib/libQt5Core.so.5
#12 0x00007ffff5f1d6ba in start_thread (arg=0x7fffe1308700) at pthread_create.c:333
---Type <return> to continue, or q <return> to quit---
#13 0x00007fffed91582d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

On 6 June 2017 at 15:51, Gilles Caulier <[hidden email]> wrote:

Well it simple. digiKam is stopped in debugger due to a C++ exception.
You get the GDB prompt and it wait a command to hack.

Just enter "bt" for backtrace to catch the full backtrace, and
especially where the program is stopped in source code.

The notice is explained in details here :

https://www.digikam.org/contribute/

Note : I'm 90% sure that you have a problem with Exiv2 with a DNG
file... (:=)))...

Gilles Caulier

2017-06-06 15:43 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
> Thanks for rapid response... Starting with digikam-5.6.0-01-x86-64.appimage
> I did with debug option
>
> 1. I created an empty directory (~sigge/digikam) for the db
> 2 Tried with the very first directory inside my gallery area, which contains
> a some 120 JPGs.
>
> I then got to 85% of that directory, with the following output from GDB and
> digikam:
>
> digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG" :
> JPEG file identified
> digikam.database: Adding new item
> "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
> digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
> QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
> digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:43:57 2012
> digikam.metaengine: Orientation => Exif.Image.Orientation => 8
> digikam.database: Scanning took 10 ms
> digikam.database: Finishing took 0 ms
> digikam.metaengine: Exiv2 ( 2 ) : Directory OlympusCs, entry 0x0101: Strip
> 0 is outside of the data area; ignored.
>
> digikam.dimg: "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG" :
> JPEG file identified
> digikam.database: Adding new item
> "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
> digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
> QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
> digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:44:02 2012
> digikam.metaengine: Orientation => Exif.Image.Orientation => 8
> digikam.database: Scanning took 12 ms
> digikam.database: Finishing took 0 ms
> [Switching to Thread 0x7fffe1308700 (LWP 16323)]
>
> Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
> 0x00007fffee1848bd in __cxa_throw ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (gdb)
>
>
> I still have it running, so if you have suggestions for next step, I can
> continue!
>
> Sigge
>
>
>
>
> On 6 June 2017 at 14:54, Gilles Caulier <[hidden email]> wrote:
>>
>> Hi,
>>
>> I'm not sure if the DNG file identified is really the problem.
>>
>> But it's clear that something is wrong with a file scanned to populate
>> the database.
>>
>> I suspect a problem with Exiv2, which is mostly used to scan image
>> properties in background.
>>
>> So the best way to identify the problem, especially the image file, is :
>>
>> 1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
>> 2/ start a clean instance of digiKam from scratch.
>> 3/ Import step by step albums. Not all in one step.
>> 4/ When the album containing the image freeze digiKam, share the datat
>> somewhere on the web to test here in local.
>>
>> Note : if there is a crash in the thread due to Exiv2, the crash will
>> not close the application, but we can catch it to identify the
>> problem. Run the AppImage from the console with the "debug" argument.
>> This will run digiKam in GDB. Run AppImage with "help" argument for
>> details.
>>
>> Best
>>
>> Gilles Caulier
>>
>>
>>
>> 2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>> >
>> > Dear everybody,
>> >
>> > I'd like first say, warmly, thanks for an excellent product. I've been a
>> > daily user since about 10 years.
>> >
>> > I've a collection of close to 10000 raw images *.DNG, *.ORF and *.RW2
>> > formats. DNG is about 75% of these. I've kept about 10% of the photos
>> > I've
>> > made. digikam has been essential in weeding and curating the collection
>> > (from about 100000 to 10000), and writing captions, applying
>> > tags/keywords
>> > and geo-tagging the all items.
>> >
>> > I've always ensured that data is written back to the exif header and
>> > that
>> > there are xmp files in the file system. The collection is stored in a
>> > file
>> > system which is also a part of my Dropbox area.
>> > I've share the same collection over more than one computer, but then had
>> > the
>> > sqlite database files in directories outside Dropbox. Worked without
>> > problem.
>> >
>> > A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to
>> > 16.04
>> > LTS, and I decided to allow that. When doing that, I felt that it would
>> > be
>> > time to go to digikam 5.5.0. It just didn't work.
>> >
>> > I've
>> >
>> > 1. Installed the appimage (5.5.0 and 5.6.0)
>> > 2. Installed the digikam5 version for ubuntu according to tutorials on
>> > the
>> > Net.
>> > 3. I've compiled it from sources (been through all the cmake and
>> > dependency
>> > manage stuff), and got executable binaries.
>> >
>> > finally I
>> >
>> > 4. Downgraded to digikam 4 again.
>> >
>> > Sorry, but it doesn't work. There is no difference in having the
>> > database
>> > files inside our outside the collections. Not even downgrading helped.
>> >
>> > What happens in all four cases is that digikam reads about 25% of the
>> > collection, and then it sighs deeply, printing to STDOUT what I've added
>> > below (seen people mentioning similar problems on mailing lists, but
>> > I've
>> > not found a clear answer).
>> >
>> > The raw image in question is perfectly readable:
>> >
>> > https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>> >
>> > Apart from dropbox (which actually make the raw processing without
>> > problems), I have successfully read this image using gimp and
>> > rawtherapee,
>> > so the file shouldn't be corrupt. After failing on that one, the process
>> > won't recover. The corresponding xmp file is well formed according to
>> > xmllint.
>> >
>> > Closing digikam, and doing dump and refresh of the database using
>> > sqlite3
>> > command line tool doesn't help.
>> > It still tries to reread the collection and ends at the same item. If I
>> > move
>> > this image outside the collection, remove the database files and start
>> > over
>> > it ends the same way for some other item.
>> >
>> > Right now I'm desperate since my work-flow is gone.
>> >
>> > Any clue, anyone?
>> >
>> > Thanks in advance
>> >
>> > Sigfrid
>> >
>> >
>> >
>> >
>> > digikam.metaengine: DateTime (Exif digitalized): tors dec. 22 20:37:23
>> > 2011
>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 1
>> > digikam.database: Scanning took 2 ms
>> > digikam.database: Finishing took 0 ms
>> > digikam.dimg: "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG"
>> > :
>> > RAW file identified
>> > digikam.geoiface: ----
>> > digikam.geoiface: ----
>> > digikam.general: Using 4 CPU core to run threads
>> > digikam.general: Stacked View Mode : 0
>> > digikam.general: Action Thread run 1 new jobs
>> > digikam.dbengine: Database is locked. Waited 0
>> > digikam.geoiface: ----
>> > digikam.dbengine: Database is locked. Waited 250
>> > digikam.dbengine: Database is locked. Waited 500
>> > digikam.dbengine: Database is locked. Waited 750
>> >
>> > ....
>> >
>> > digikam.dbengine: Database is locked. Waited 9500
>> > digikam.dbengine: Database is locked. Waited 9750
>> > digikam.dbengine: Database is locked. Waited 10000
>> > digikam.dbengine: Detected locked database file. There is an active
>> > transaction. Waited but giving up now.
>> > digikam.dbengine: Failure executing query:
>> > "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
>> > Error messages: "Unable to fetch row" "database table is locked: Albums"
>> > 6 1
>> > Bound values: (QVariant(int, 1), QVariant(QString, "/200708-200712"))
>> > digikam.general: Data From DBJobsThread is null: true
>> > digikam.general: Cancel Main Thread
>> > digikam.general: One job is done
>> > ^C
>> >
>> >
>> >
>> >
>> > --
>> > Sigfrid Lundberg, Ph.D., System developer
>> > Lund, Sweden
>> > http://sigfrid-lundberg.se/
>
>
>
>
> --
> Sigfrid Lundberg, Ph.D., System developer
> Lund, Sweden
> http://sigfrid-lundberg.se/

Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/

Gilles Caulier-4

Re: Failing to load collection

Et voilà... an Exiv2 exception is generated with a specific file.

Now, try to identify which file create this dysfunction. It sound like
a file included in /home/sigge/Dropbox/galleri/2011-2012-freddy/

With the Exiv2 CLI tool, try to just read each file from this album to
see if something is wrong while scanning tag contents. Warning :
digiKam 5.6.0 Appimage use the last stable Exiv2 0.26, and previous
Exiv2 0.25 can not generate an exception).

When you found the right file, report the problem to Exiv2 bugzilla
for future investigations. Don't forget to share the problematic file.

From the digiKam side, it still a problem to wrap the exception from
Exiv2 without to lock the application. Typically a time out when the
item is scanned to a separated thread. The Exiv2 exception is catch
properly, but something is wrong somewhere to complete the scan even
if an error occur.

For this point, please open a file in KDE bugzilla, in
digiKam/metadata section with a link to the suspected file to hack
digiKam core implementation.

Gilles Caulier

2017-06-06 16:00 GMT+02:00 Sigfrid Lundberg <[hidden email]>:

> That directory is JPG only, taken with an Olympus, no DNG
> Exceptions I do understand. So close to java ;^)
>
> GDB supports the exiv2 hypothesis, though. GDB claims the following
>
> Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
> 0x00007fffee1848bd in __cxa_throw ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> (gdb) bt
> #0 0x00007fffee1848bd in __cxa_throw () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #1 0x0000003455b53b15 in Exiv2::ImageFactory::open (path=...,
> useCurl=<optimized out>)
> at
> /b/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/exiv2-trunk/src/image.cpp:848
> #2 0x00007ffff6775c71 in Digikam::MetaEngine::load
> (this=this@entry=0x7fffd4063c60, filePath=...)
> at /b/dktemp/digikam-master/core/libs/dmetadata/metaengine.cpp:281
> #3 0x00007ffff67be5c6 in Digikam::DMetadata::load
> (this=this@entry=0x7fffd4063c60, filePath=...)
> at /b/dktemp/digikam-master/core/libs/dmetadata/dmetadata.cpp:96
> #4 0x00007fffee59a3fa in Digikam::ImageScanner::loadFromDisk
> (this=this@entry=0x7fffe13077e0)
> at
> /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:1563
> #5 0x00007fffee59a590 in Digikam::ImageScanner::newFile
> (this=this@entry=0x7fffe13077e0, albumId=2)
> at /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:289
> #6 0x00007fffee4dbb7e in Digikam::CollectionScanner::scanNewFile
> (this=this@entry=0x7fffe1307bf0,
> info=..., albumId=2)
> at
> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1283
> #7 0x00007fffee4dd488 in Digikam::CollectionScanner::scanAlbum
> (this=this@entry=0x7fffe1307bf0,
> location=..., album=...)
> at
> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1106
> #8 0x00007fffee4ddaf9 in Digikam::CollectionScanner::scanAlbumRoot (
> this=this@entry=0x7fffe1307bf0, location=...)
> at
> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:835
> #9 0x00007fffee4ddd26 in Digikam::CollectionScanner::completeScan
> (this=this@entry=0x7fffe1307bf0)
> at
> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:488
> #10 0x00007ffff74f485e in Digikam::ScanController::run (this=
> 0x7ffff7dd4da0
> <_ZZN7Digikam12_GLOBAL__N_113Q_QGS_creator13innerFunctionEvE6holder>)
> at
> /b/dktemp/digikam-master/core/libs/database/utils/scancontroller.cpp:708
> #11 0x00000034890aef49 in ?? () from
> /tmp/.mount_sIj2qg/usr/lib/libQt5Core.so.5
> #12 0x00007ffff5f1d6ba in start_thread (arg=0x7fffe1308700) at
> pthread_create.c:333
> ---Type <return> to continue, or q <return> to quit---
> #13 0x00007fffed91582d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> (gdb)
>
>
> On 6 June 2017 at 15:51, Gilles Caulier <[hidden email]> wrote:
>>
>> Well it simple. digiKam is stopped in debugger due to a C++ exception.
>> You get the GDB prompt and it wait a command to hack.
>>
>> Just enter "bt" for backtrace to catch the full backtrace, and
>> especially where the program is stopped in source code.
>>
>> The notice is explained in details here :
>>
>> https://www.digikam.org/contribute/
>>
>> Note : I'm 90% sure that you have a problem with Exiv2 with a DNG
>> file... (:=)))...
>>
>> Gilles Caulier
>>
>> 2017-06-06 15:43 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>> > Thanks for rapid response... Starting with
>> > digikam-5.6.0-01-x86-64.appimage
>> > I did with debug option
>> >
>> > 1. I created an empty directory (~sigge/digikam) for the db
>> > 2 Tried with the very first directory inside my gallery area, which
>> > contains
>> > a some 120 JPGs.
>> >
>> > I then got to 85% of that directory, with the following output from GDB
>> > and
>> > digikam:
>> >
>> > digikam.dimg:
>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG" :
>> > JPEG file identified
>> > digikam.database: Adding new item
>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>> > QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
>> > digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:43:57
>> > 2012
>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 8
>> > digikam.database: Scanning took 10 ms
>> > digikam.database: Finishing took 0 ms
>> > digikam.metaengine: Exiv2 ( 2 ) : Directory OlympusCs, entry 0x0101:
>> > Strip
>> > 0 is outside of the data area; ignored.
>> >
>> > digikam.dimg:
>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG" :
>> > JPEG file identified
>> > digikam.database: Adding new item
>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>> > QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
>> > digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:44:02
>> > 2012
>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 8
>> > digikam.database: Scanning took 12 ms
>> > digikam.database: Finishing took 0 ms
>> > [Switching to Thread 0x7fffe1308700 (LWP 16323)]
>> >
>> > Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
>> > 0x00007fffee1848bd in __cxa_throw ()
>> > from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> > (gdb)
>> >
>> >
>> > I still have it running, so if you have suggestions for next step, I can
>> > continue!
>> >
>> > Sigge
>> >
>> >
>> >
>> >
>> > On 6 June 2017 at 14:54, Gilles Caulier <[hidden email]>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm not sure if the DNG file identified is really the problem.
>> >>
>> >> But it's clear that something is wrong with a file scanned to populate
>> >> the database.
>> >>
>> >> I suspect a problem with Exiv2, which is mostly used to scan image
>> >> properties in background.
>> >>
>> >> So the best way to identify the problem, especially the image file, is
>> >> :
>> >>
>> >> 1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
>> >> 2/ start a clean instance of digiKam from scratch.
>> >> 3/ Import step by step albums. Not all in one step.
>> >> 4/ When the album containing the image freeze digiKam, share the datat
>> >> somewhere on the web to test here in local.
>> >>
>> >> Note : if there is a crash in the thread due to Exiv2, the crash will
>> >> not close the application, but we can catch it to identify the
>> >> problem. Run the AppImage from the console with the "debug" argument.
>> >> This will run digiKam in GDB. Run AppImage with "help" argument for
>> >> details.
>> >>
>> >> Best
>> >>
>> >> Gilles Caulier
>> >>
>> >>
>> >>
>> >> 2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg
>> >> <[hidden email]>:
>> >> >
>> >> > Dear everybody,
>> >> >
>> >> > I'd like first say, warmly, thanks for an excellent product. I've
>> >> > been a
>> >> > daily user since about 10 years.
>> >> >
>> >> > I've a collection of close to 10000 raw images *.DNG, *.ORF and
>> >> > *.RW2
>> >> > formats. DNG is about 75% of these. I've kept about 10% of the photos
>> >> > I've
>> >> > made. digikam has been essential in weeding and curating the
>> >> > collection
>> >> > (from about 100000 to 10000), and writing captions, applying
>> >> > tags/keywords
>> >> > and geo-tagging the all items.
>> >> >
>> >> > I've always ensured that data is written back to the exif header and
>> >> > that
>> >> > there are xmp files in the file system. The collection is stored in
>> >> > a
>> >> > file
>> >> > system which is also a part of my Dropbox area.
>> >> > I've share the same collection over more than one computer, but then
>> >> > had
>> >> > the
>> >> > sqlite database files in directories outside Dropbox. Worked without
>> >> > problem.
>> >> >
>> >> > A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to
>> >> > 16.04
>> >> > LTS, and I decided to allow that. When doing that, I felt that it
>> >> > would
>> >> > be
>> >> > time to go to digikam 5.5.0. It just didn't work.
>> >> >
>> >> > I've
>> >> >
>> >> > 1. Installed the appimage (5.5.0 and 5.6.0)
>> >> > 2. Installed the digikam5 version for ubuntu according to tutorials
>> >> > on
>> >> > the
>> >> > Net.
>> >> > 3. I've compiled it from sources (been through all the cmake and
>> >> > dependency
>> >> > manage stuff), and got executable binaries.
>> >> >
>> >> > finally I
>> >> >
>> >> > 4. Downgraded to digikam 4 again.
>> >> >
>> >> > Sorry, but it doesn't work. There is no difference in having the
>> >> > database
>> >> > files inside our outside the collections. Not even downgrading
>> >> > helped.
>> >> >
>> >> > What happens in all four cases is that digikam reads about 25% of the
>> >> > collection, and then it sighs deeply, printing to STDOUT what I've
>> >> > added
>> >> > below (seen people mentioning similar problems on mailing lists, but
>> >> > I've
>> >> > not found a clear answer).
>> >> >
>> >> > The raw image in question is perfectly readable:
>> >> >
>> >> > https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>> >> >
>> >> > Apart from dropbox (which actually make the raw processing without
>> >> > problems), I have successfully read this image using gimp and
>> >> > rawtherapee,
>> >> > so the file shouldn't be corrupt. After failing on that one, the
>> >> > process
>> >> > won't recover. The corresponding xmp file is well formed according
>> >> > to
>> >> > xmllint.
>> >> >
>> >> > Closing digikam, and doing dump and refresh of the database using
>> >> > sqlite3
>> >> > command line tool doesn't help.
>> >> > It still tries to reread the collection and ends at the same item. If
>> >> > I
>> >> > move
>> >> > this image outside the collection, remove the database files and
>> >> > start
>> >> > over
>> >> > it ends the same way for some other item.
>> >> >
>> >> > Right now I'm desperate since my work-flow is gone.
>> >> >
>> >> > Any clue, anyone?
>> >> >
>> >> > Thanks in advance
>> >> >
>> >> > Sigfrid
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > digikam.metaengine: DateTime (Exif digitalized): tors dec. 22
>> >> > 20:37:23
>> >> > 2011
>> >> > digikam.metaengine: Orientation => Exif.Image.Orientation => 1
>> >> > digikam.database: Scanning took 2 ms
>> >> > digikam.database: Finishing took 0 ms
>> >> > digikam.dimg:
>> >> > "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG"
>> >> > :
>> >> > RAW file identified
>> >> > digikam.geoiface: ----
>> >> > digikam.geoiface: ----
>> >> > digikam.general: Using 4 CPU core to run threads
>> >> > digikam.general: Stacked View Mode : 0
>> >> > digikam.general: Action Thread run 1 new jobs
>> >> > digikam.dbengine: Database is locked. Waited 0
>> >> > digikam.geoiface: ----
>> >> > digikam.dbengine: Database is locked. Waited 250
>> >> > digikam.dbengine: Database is locked. Waited 500
>> >> > digikam.dbengine: Database is locked. Waited 750
>> >> >
>> >> > ....
>> >> >
>> >> > digikam.dbengine: Database is locked. Waited 9500
>> >> > digikam.dbengine: Database is locked. Waited 9750
>> >> > digikam.dbengine: Database is locked. Waited 10000
>> >> > digikam.dbengine: Detected locked database file. There is an active
>> >> > transaction. Waited but giving up now.
>> >> > digikam.dbengine: Failure executing query:
>> >> > "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
>> >> > Error messages: "Unable to fetch row" "database table is locked:
>> >> > Albums"
>> >> > 6 1
>> >> > Bound values: (QVariant(int, 1), QVariant(QString,
>> >> > "/200708-200712"))
>> >> > digikam.general: Data From DBJobsThread is null: true
>> >> > digikam.general: Cancel Main Thread
>> >> > digikam.general: One job is done
>> >> > ^C
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Sigfrid Lundberg, Ph.D., System developer
>> >> > Lund, Sweden
>> >> > http://sigfrid-lundberg.se/
>> >
>> >
>> >
>> >
>> > --
>> > Sigfrid Lundberg, Ph.D., System developer
>> > Lund, Sweden
>> > http://sigfrid-lundberg.se/
>
>
>
>
> --
> Sigfrid Lundberg, Ph.D., System developer
> Lund, Sweden
> http://sigfrid-lundberg.se/

Sigfrid Lundberg

Re: Failing to load collection

OK, I'll try to report to the relevant bug report systems. I'm not
sure my observations helps very much, though :^/

I've upgraded my exiv2 to the tar.gz I found on their site, 0.26 something.
Removed my previous one with apt-get remove.

Now 'exiv2 -V' gives 'exiv2 0.26 001a00 (64 bit build)'

The following produces exactly one warning per JPG, but not a single exception:

find 2011-2012-freddy/ -name '*.JPG' -exec exiv2 -pa {} > /dev/null \;

If I instead execs exiv2 for each file, regardoess, it throws
exceptions on AVI files, like

Exiv2 exception in print action for file 2011-2012-freddy/P1170859.AVI:
2011-2012-freddy/P1170859.AVI: The file contains data of an unknown image type

Likewise, if I run exiv2 on each file in the directory where it
stopped when I first contacted this list

find 201112-201112/ -type f -exec exiv2 -pa {} > /dev/null \;

Exiv2 exception in print action for file 201112-201112/R0010678.DNG.pp3:
201112-201112/R0010678.DNG.pp3: The file contains data of an unknown image type
Error: XMP Toolkit error 203: Duplicate property or field node
Warning: Failed to decode XMP metadata.

Where pp3 is a file left by rawtherapee. Then we haven't even come to
all the directories where I mix text image and sounds and whatever.
eps, pdf, ms, latex, html....

Is the combo exiv2 and digikam now expecting absolutely clean
directories untouched by text and video editors and no soundfiles in
the corners?

Sigge

On 6 June 2017 at 16:18, Gilles Caulier <[hidden email]> wrote:

> Et voilà... an Exiv2 exception is generated with a specific file.
>
> Now, try to identify which file create this dysfunction. It sound like
> a file included in /home/sigge/Dropbox/galleri/2011-2012-freddy/
>
> With the Exiv2 CLI tool, try to just read each file from this album to
> see if something is wrong while scanning tag contents. Warning :
> digiKam 5.6.0 Appimage use the last stable Exiv2 0.26, and previous
> Exiv2 0.25 can not generate an exception).
>
> When you found the right file, report the problem to Exiv2 bugzilla
> for future investigations. Don't forget to share the problematic file.
>
> From the digiKam side, it still a problem to wrap the exception from
> Exiv2 without to lock the application. Typically a time out when the
> item is scanned to a separated thread. The Exiv2 exception is catch
> properly, but something is wrong somewhere to complete the scan even
> if an error occur.
>
> For this point, please open a file in KDE bugzilla, in
> digiKam/metadata section with a link to the suspected file to hack
> digiKam core implementation.
>
> Gilles Caulier
>
> 2017-06-06 16:00 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>> That directory is JPG only, taken with an Olympus, no DNG
>> Exceptions I do understand. So close to java ;^)
>>
>> GDB supports the exiv2 hypothesis, though. GDB claims the following
>>
>> Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
>> 0x00007fffee1848bd in __cxa_throw ()
>> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> (gdb) bt
>> #0 0x00007fffee1848bd in __cxa_throw () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #1 0x0000003455b53b15 in Exiv2::ImageFactory::open (path=...,
>> useCurl=<optimized out>)
>> at
>> /b/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/exiv2-trunk/src/image.cpp:848
>> #2 0x00007ffff6775c71 in Digikam::MetaEngine::load
>> (this=this@entry=0x7fffd4063c60, filePath=...)
>> at /b/dktemp/digikam-master/core/libs/dmetadata/metaengine.cpp:281
>> #3 0x00007ffff67be5c6 in Digikam::DMetadata::load
>> (this=this@entry=0x7fffd4063c60, filePath=...)
>> at /b/dktemp/digikam-master/core/libs/dmetadata/dmetadata.cpp:96
>> #4 0x00007fffee59a3fa in Digikam::ImageScanner::loadFromDisk
>> (this=this@entry=0x7fffe13077e0)
>> at
>> /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:1563
>> #5 0x00007fffee59a590 in Digikam::ImageScanner::newFile
>> (this=this@entry=0x7fffe13077e0, albumId=2)
>> at /b/dktemp/digikam-master/core/libs/database/item/imagescanner.cpp:289
>> #6 0x00007fffee4dbb7e in Digikam::CollectionScanner::scanNewFile
>> (this=this@entry=0x7fffe1307bf0,
>> info=..., albumId=2)
>> at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1283
>> #7 0x00007fffee4dd488 in Digikam::CollectionScanner::scanAlbum
>> (this=this@entry=0x7fffe1307bf0,
>> location=..., album=...)
>> at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:1106
>> #8 0x00007fffee4ddaf9 in Digikam::CollectionScanner::scanAlbumRoot (
>> this=this@entry=0x7fffe1307bf0, location=...)
>> at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:835
>> #9 0x00007fffee4ddd26 in Digikam::CollectionScanner::completeScan
>> (this=this@entry=0x7fffe1307bf0)
>> at
>> /b/dktemp/digikam-master/core/libs/database/collection/collectionscanner.cpp:488
>> #10 0x00007ffff74f485e in Digikam::ScanController::run (this=
>> 0x7ffff7dd4da0
>> <_ZZN7Digikam12_GLOBAL__N_113Q_QGS_creator13innerFunctionEvE6holder>)
>> at
>> /b/dktemp/digikam-master/core/libs/database/utils/scancontroller.cpp:708
>> #11 0x00000034890aef49 in ?? () from
>> /tmp/.mount_sIj2qg/usr/lib/libQt5Core.so.5
>> #12 0x00007ffff5f1d6ba in start_thread (arg=0x7fffe1308700) at
>> pthread_create.c:333
>> ---Type <return> to continue, or q <return> to quit---
>> #13 0x00007fffed91582d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>> (gdb)
>>
>>
>> On 6 June 2017 at 15:51, Gilles Caulier <[hidden email]> wrote:
>>>
>>> Well it simple. digiKam is stopped in debugger due to a C++ exception.
>>> You get the GDB prompt and it wait a command to hack.
>>>
>>> Just enter "bt" for backtrace to catch the full backtrace, and
>>> especially where the program is stopped in source code.
>>>
>>> The notice is explained in details here :
>>>
>>> https://www.digikam.org/contribute/
>>>
>>> Note : I'm 90% sure that you have a problem with Exiv2 with a DNG
>>> file... (:=)))...
>>>
>>> Gilles Caulier
>>>
>>> 2017-06-06 15:43 GMT+02:00 Sigfrid Lundberg <[hidden email]>:
>>> > Thanks for rapid response... Starting with
>>> > digikam-5.6.0-01-x86-64.appimage
>>> > I did with debug option
>>> >
>>> > 1. I created an empty directory (~sigge/digikam) for the db
>>> > 2 Tried with the very first directory inside my gallery area, which
>>> > contains
>>> > a some 120 JPGs.
>>> >
>>> > I then got to 85% of that directory, with the following output from GDB
>>> > and
>>> > digikam:
>>> >
>>> > digikam.dimg:
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG" :
>>> > JPEG file identified
>>> > digikam.database: Adding new item
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080855.JPG"
>>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>>> > QDateTime(2012-01-08 16:43:57.000 CET Qt::TimeSpec(LocalTime))
>>> > digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:43:57
>>> > 2012
>>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 8
>>> > digikam.database: Scanning took 10 ms
>>> > digikam.database: Finishing took 0 ms
>>> > digikam.metaengine: Exiv2 ( 2 ) : Directory OlympusCs, entry 0x0101:
>>> > Strip
>>> > 0 is outside of the data area; ignored.
>>> >
>>> > digikam.dimg:
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG" :
>>> > JPEG file identified
>>> > digikam.database: Adding new item
>>> > "/home/sigge/Dropbox/galleri/2011-2012-freddy/P1080856.JPG"
>>> > digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal =>
>>> > QDateTime(2012-01-08 16:44:02.000 CET Qt::TimeSpec(LocalTime))
>>> > digikam.metaengine: DateTime (Exif digitalized): s�n jan. 8 16:44:02
>>> > 2012
>>> > digikam.metaengine: Orientation => Exif.Image.Orientation => 8
>>> > digikam.database: Scanning took 12 ms
>>> > digikam.database: Finishing took 0 ms
>>> > [Switching to Thread 0x7fffe1308700 (LWP 16323)]
>>> >
>>> > Thread 5 "Digikam::ScanCo" hit Catchpoint 1 (exception thrown),
>>> > 0x00007fffee1848bd in __cxa_throw ()
>>> > from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> > (gdb)
>>> >
>>> >
>>> > I still have it running, so if you have suggestions for next step, I can
>>> > continue!
>>> >
>>> > Sigge
>>> >
>>> >
>>> >
>>> >
>>> > On 6 June 2017 at 14:54, Gilles Caulier <[hidden email]>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I'm not sure if the DNG file identified is really the problem.
>>> >>
>>> >> But it's clear that something is wrong with a file scanned to populate
>>> >> the database.
>>> >>
>>> >> I suspect a problem with Exiv2, which is mostly used to scan image
>>> >> properties in background.
>>> >>
>>> >> So the best way to identify the problem, especially the image file, is
>>> >> :
>>> >>
>>> >> 1/ use only the last 5.6.0 pre-release AppImage from GDrive repository.
>>> >> 2/ start a clean instance of digiKam from scratch.
>>> >> 3/ Import step by step albums. Not all in one step.
>>> >> 4/ When the album containing the image freeze digiKam, share the datat
>>> >> somewhere on the web to test here in local.
>>> >>
>>> >> Note : if there is a crash in the thread due to Exiv2, the crash will
>>> >> not close the application, but we can catch it to identify the
>>> >> problem. Run the AppImage from the console with the "debug" argument.
>>> >> This will run digiKam in GDB. Run AppImage with "help" argument for
>>> >> details.
>>> >>
>>> >> Best
>>> >>
>>> >> Gilles Caulier
>>> >>
>>> >>
>>> >>
>>> >> 2017-06-06 14:44 GMT+02:00 Sigfrid Lundberg
>>> >> <[hidden email]>:
>>> >> >
>>> >> > Dear everybody,
>>> >> >
>>> >> > I'd like first say, warmly, thanks for an excellent product. I've
>>> >> > been a
>>> >> > daily user since about 10 years.
>>> >> >
>>> >> > I've a collection of close to 10000 raw images *.DNG, *.ORF and
>>> >> > *.RW2
>>> >> > formats. DNG is about 75% of these. I've kept about 10% of the photos
>>> >> > I've
>>> >> > made. digikam has been essential in weeding and curating the
>>> >> > collection
>>> >> > (from about 100000 to 10000), and writing captions, applying
>>> >> > tags/keywords
>>> >> > and geo-tagging the all items.
>>> >> >
>>> >> > I've always ensured that data is written back to the exif header and
>>> >> > that
>>> >> > there are xmp files in the file system. The collection is stored in
>>> >> > a
>>> >> > file
>>> >> > system which is also a part of my Dropbox area.
>>> >> > I've share the same collection over more than one computer, but then
>>> >> > had
>>> >> > the
>>> >> > sqlite database files in directories outside Dropbox. Worked without
>>> >> > problem.
>>> >> >
>>> >> > A few nights ago, my computer wanted to go from Ubuntu 14.04 LTS to
>>> >> > 16.04
>>> >> > LTS, and I decided to allow that. When doing that, I felt that it
>>> >> > would
>>> >> > be
>>> >> > time to go to digikam 5.5.0. It just didn't work.
>>> >> >
>>> >> > I've
>>> >> >
>>> >> > 1. Installed the appimage (5.5.0 and 5.6.0)
>>> >> > 2. Installed the digikam5 version for ubuntu according to tutorials
>>> >> > on
>>> >> > the
>>> >> > Net.
>>> >> > 3. I've compiled it from sources (been through all the cmake and
>>> >> > dependency
>>> >> > manage stuff), and got executable binaries.
>>> >> >
>>> >> > finally I
>>> >> >
>>> >> > 4. Downgraded to digikam 4 again.
>>> >> >
>>> >> > Sorry, but it doesn't work. There is no difference in having the
>>> >> > database
>>> >> > files inside our outside the collections. Not even downgrading
>>> >> > helped.
>>> >> >
>>> >> > What happens in all four cases is that digikam reads about 25% of the
>>> >> > collection, and then it sighs deeply, printing to STDOUT what I've
>>> >> > added
>>> >> > below (seen people mentioning similar problems on mailing lists, but
>>> >> > I've
>>> >> > not found a clear answer).
>>> >> >
>>> >> > The raw image in question is perfectly readable:
>>> >> >
>>> >> > https://www.dropbox.com/s/zwh1seo5gced12b/R0011071.DNG?dl=0
>>> >> >
>>> >> > Apart from dropbox (which actually make the raw processing without
>>> >> > problems), I have successfully read this image using gimp and
>>> >> > rawtherapee,
>>> >> > so the file shouldn't be corrupt. After failing on that one, the
>>> >> > process
>>> >> > won't recover. The corresponding xmp file is well formed according
>>> >> > to
>>> >> > xmllint.
>>> >> >
>>> >> > Closing digikam, and doing dump and refresh of the database using
>>> >> > sqlite3
>>> >> > command line tool doesn't help.
>>> >> > It still tries to reread the collection and ends at the same item. If
>>> >> > I
>>> >> > move
>>> >> > this image outside the collection, remove the database files and
>>> >> > start
>>> >> > over
>>> >> > it ends the same way for some other item.
>>> >> >
>>> >> > Right now I'm desperate since my work-flow is gone.
>>> >> >
>>> >> > Any clue, anyone?
>>> >> >
>>> >> > Thanks in advance
>>> >> >
>>> >> > Sigfrid
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > digikam.metaengine: DateTime (Exif digitalized): tors dec. 22
>>> >> > 20:37:23
>>> >> > 2011
>>> >> > digikam.metaengine: Orientation => Exif.Image.Orientation => 1
>>> >> > digikam.database: Scanning took 2 ms
>>> >> > digikam.database: Finishing took 0 ms
>>> >> > digikam.dimg:
>>> >> > "/home/sigge/Dropbox/galleri/201112-201112/R0011071.DNG"
>>> >> > :
>>> >> > RAW file identified
>>> >> > digikam.geoiface: ----
>>> >> > digikam.geoiface: ----
>>> >> > digikam.general: Using 4 CPU core to run threads
>>> >> > digikam.general: Stacked View Mode : 0
>>> >> > digikam.general: Action Thread run 1 new jobs
>>> >> > digikam.dbengine: Database is locked. Waited 0
>>> >> > digikam.geoiface: ----
>>> >> > digikam.dbengine: Database is locked. Waited 250
>>> >> > digikam.dbengine: Database is locked. Waited 500
>>> >> > digikam.dbengine: Database is locked. Waited 750
>>> >> >
>>> >> > ....
>>> >> >
>>> >> > digikam.dbengine: Database is locked. Waited 9500
>>> >> > digikam.dbengine: Database is locked. Waited 9750
>>> >> > digikam.dbengine: Database is locked. Waited 10000
>>> >> > digikam.dbengine: Detected locked database file. There is an active
>>> >> > transaction. Waited but giving up now.
>>> >> > digikam.dbengine: Failure executing query:
>>> >> > "SELECT id FROM Albums WHERE albumRoot=? AND relativePath=?;"
>>> >> > Error messages: "Unable to fetch row" "database table is locked:
>>> >> > Albums"
>>> >> > 6 1
>>> >> > Bound values: (QVariant(int, 1), QVariant(QString,
>>> >> > "/200708-200712"))
>>> >> > digikam.general: Data From DBJobsThread is null: true
>>> >> > digikam.general: Cancel Main Thread
>>> >> > digikam.general: One job is done
>>> >> > ^C
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Sigfrid Lundberg, Ph.D., System developer
>>> >> > Lund, Sweden
>>> >> > http://sigfrid-lundberg.se/
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Sigfrid Lundberg, Ph.D., System developer
>>> > Lund, Sweden
>>> > http://sigfrid-lundberg.se/
>>
>>
>>
>>
>> --
>> Sigfrid Lundberg, Ph.D., System developer
>> Lund, Sweden
>> http://sigfrid-lundberg.se/

--
Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/

Remco Viëtor

Re: Failing to load collection

On mardi 6 juin 2017 18:33:14 CEST Sigfrid Lundberg wrote:

> OK, I'll try to report to the relevant bug report systems. I'm not
> sure my observations helps very much, though :^/
>
> I've upgraded my exiv2 to the tar.gz I found on their site, 0.26 something.
> Removed my previous one with apt-get remove.
>
> Now 'exiv2 -V' gives 'exiv2 0.26 001a00 (64 bit build)'
>
> The following produces exactly one warning per JPG, but not a single
> exception:
>
> find 2011-2012-freddy/ -name '*.JPG' -exec exiv2 -pa {} > /dev/null \;
>
> If I instead execs exiv2 for each file, regardoess, it throws
> exceptions on AVI files, like
>
> Exiv2 exception in print action for file 2011-2012-freddy/P1170859.AVI:
> 2011-2012-freddy/P1170859.AVI: The file contains data of an unknown image
> type
>
> Likewise, if I run exiv2 on each file in the directory where it
> stopped when I first contacted this list
>
> find 201112-201112/ -type f -exec exiv2 -pa {} > /dev/null \;
>
> Exiv2 exception in print action for file 201112-201112/R0010678.DNG.pp3:
> 201112-201112/R0010678.DNG.pp3: The file contains data of an unknown image
> type Error: XMP Toolkit error 203: Duplicate property or field node
> Warning: Failed to decode XMP metadata.
>
> Where pp3 is a file left by rawtherapee. Then we haven't even come to
> all the directories where I mix text image and sounds and whatever.
> eps, pdf, ms, latex, html....
>
> Is the combo exiv2 and digikam now expecting absolutely clean
> directories untouched by text and video editors and no soundfiles in
> the corners?
>

I hope not...
But...
It's a bit disingenuous to launch the *executable* exiv2 /for each file in a
directory/, and then ask whether Digikam is going to require "absolutely clean
directories untouched by text and video editors"... Digikam decides which files
to read through the exiv2 *library*, and it will ignore certain file types.
'find' won't...

Remco.

P.S. Could you please delete unnecessary quoted material from your replies?

Sigfrid Lundberg

Re: Failing to load collection

Hi again,

Oops I've been using gmail's ways of doing things uncritically.

I tried to do what you asked for, and reported back those exceptions I
found and compared them with my experience from installing, upgrading
and scanning directories using digikam5.6. The image files didn't
throw exceptions, the videos and pp3 files did, and there wasn't
anything else in those directories.

Please bear with me if I'm a bit emotional, but I've had a work-flow
almost 10 years, until Saturday night. Now I'll have to figure
something out using whatever tools I have.

Thanks for you for your help, and suggestions.

Sigge

> I hope not...
> But...
> It's a bit disingenuous to launch the *executable* exiv2 /for each file in a
> directory/, and then ask whether Digikam is going to require "absolutely clean
> directories untouched by text and video editors"... Digikam decides which files
> to read through the exiv2 *library*, and it will ignore certain file types.
> 'find' won't...
>
> Remco.
>
> P.S. Could you please delete unnecessary quoted material from your replies?

--
Sigfrid Lundberg, Ph.D., System developer
Lund, Sweden
http://sigfrid-lundberg.se/

jensod

Re: Failing to load collection

Hi Sigge,

could you solve the issue in general or at least did you find a solution for
yourself?

I'm in the same situation, working with Digikam from the really beginning
promoting it among my friends and now being locked out, just before
Christmas time and the calendar preparation task!

Jens

--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html