Collection Scanning of Specific Directories

[Todd Goodman]

Hi,

I'm having problems with 1.3.0 (built from SVN.)

I have a large number of pictures spread out over a number of
directories.

When Digikam starts up with a new DB it wants to scan the entire
collection.

If I allow it to do so it performs the scan (over many hours) and then
when it looks like it's completed the scan, it gets a realloc (memory
corruption) error and hangs.

I have to kill Digikam and the next time I restart it still wants to
rescan everything.

If I cancel out of the collection scan dialog box then it ends up with
a QLIST out of memory error.

I was wondering if I could either:

1)  Just scan each directory individually so as not to run out of memory

or

2)  Perform some kind of incremental scan so that if it crashes during
the initial scan I could restart and pic up where it left off
(eventually getting past the out of memory problems.)

Thank you,

Todd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Johannes Wienke-2]

Gilles,

I remember there was a problem with the progress dialog for scanning
that consumed quadratic cpu and memory compared to the number of items.
Was that fixed or could it be related?

Johannes

Todd Goodman wrote:

> Hi,
>
> I'm having problems with 1.3.0 (built from SVN.)
>
> I have a large number of pictures spread out over a number of
> directories.
>
> When Digikam starts up with a new DB it wants to scan the entire
> collection.
>
> If I allow it to do so it performs the scan (over many hours) and then
> when it looks like it's completed the scan, it gets a realloc (memory
> corruption) error and hangs.
>
> I have to kill Digikam and the next time I restart it still wants to
> rescan everything.
>
> If I cancel out of the collection scan dialog box then it ends up with
> a QLIST out of memory error.
>
> I was wondering if I could either:
>
> 1)  Just scan each directory individually so as not to run out of memory
>
> or
>
> 2)  Perform some kind of incremental scan so that if it crashes during
> the initial scan I could restart and pic up where it left off
> (eventually getting past the out of memory problems.)
>
> Thank you,
>
> Todd
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> Hi,
>
> I'm having problems with 1.3.0 (built from SVN.)
>
> I have a large number of pictures spread out over a number of
> directories.

10^6? 10^7?

>
> When Digikam starts up with a new DB it wants to scan the entire
> collection.
>
> If I allow it to do so it performs the scan (over many hours) and then
> when it looks like it's completed the scan, it gets a realloc (memory
> corruption) error and hangs.
>
> I have to kill Digikam and the next time I restart it still wants to
> rescan everything.
>
> If I cancel out of the collection scan dialog box then it ends up with
> a QLIST out of memory error.

Can you give a backtrace? Would also be nice to have a backtrace for the first
crash, but I can understand that is not done in a minute.

>
> I was wondering if I could either:
>
> 1)  Just scan each directory individually so as not to run out of memory
>
> or
>
> 2)  Perform some kind of incremental scan so that if it crashes during
> the initial scan I could restart and pic up where it left off
> (eventually getting past the out of memory problems.)

The solution should be not to get out of memory at all ;-)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

2010/6/24 Johannes Wienke <[hidden email]>:
> Gilles,
>
> I remember there was a problem with the progress dialog for scanning
> that consumed quadratic cpu and memory compared to the number of items.
> Was that fixed or could it be related?

Not sure, but it can be. Anyway, we need to fix it. It's not very complicated.

The problem in this dialog is to host icons on treeview. In fact, we
don't need an history of progress jobs there.

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Todd Goodman]

* Marcel Wiesweg <[hidden email]> [100624 11:51]:
> > Hi,
> >
> > I'm having problems with 1.3.0 (built from SVN.)
> >
> > I have a large number of pictures spread out over a number of
> > directories.
>
> 10^6? 10^7?

About 5.5million

>
>
> >
> > When Digikam starts up with a new DB it wants to scan the entire
> > collection.
> >
> > If I allow it to do so it performs the scan (over many hours) and then
> > when it looks like it's completed the scan, it gets a realloc (memory
> > corruption) error and hangs.
> >
> > I have to kill Digikam and the next time I restart it still wants to
> > rescan everything.
> >
> > If I cancel out of the collection scan dialog box then it ends up with
> > a QLIST out of memory error.
>
> Can you give a backtrace? Would also be nice to have a backtrace for the first
> crash, but I can understand that is not done in a minute.

Yes for the QList out of memory problem if I let the scan take place
without cancelling the dialog box.  It's below.

I haven't been able to reproduce the realloc() problem

>
> >
> > I was wondering if I could either:
> >
> > 1)  Just scan each directory individually so as not to run out of memory
> >
> > or
> >
> > 2)  Perform some kind of incremental scan so that if it crashes during
> > the initial scan I could restart and pic up where it left off
> > (eventually getting past the out of memory problems.)
>
> The solution should be not to get out of memory at all ;-)

Yes, that would be ideal.  :-)

Thank you,

Todd

Application: digiKam (digikam), signal: Aborted
[Current thread is 0 (LWP 27729)]

Thread 2 (Thread 0xb27a9b90 (LWP 27731)):
[KCrash Handler]
#5  0xffffe424 in __kernel_vsyscall ()
#6  0xb5445c41 in raise () from /lib/libc.so.6
#7  0xb5447428 in abort () from /lib/libc.so.6
#8  0xb56d08f5 in qt_message_output () from /usr/lib/qt4/libQtCore.so.4
#9  0xb56d09aa in qFatal () from /usr/lib/qt4/libQtCore.so.4
#10 0xb56fd8c0 in QListData::realloc () from /usr/lib/qt4/libQtCore.so.4
#11 0xb56fd9a7 in QListData::append () from /usr/lib/qt4/libQtCore.so.4
#12 0xb57b9b57 in ?? () from /usr/lib/qt4/libQtCore.so.4
#13 0x084f30e8 in ?? ()
#14 0x0000002b in ?? ()
#15 0xb57b65cc in QCoreApplication::postEvent () from /usr/lib/qt4/libQtCore.so.4
#16 0xb57b667a in QCoreApplication::postEvent () from /usr/lib/qt4/libQtCore.so.4
#17 0xb57c74d8 in ?? () from /usr/lib/qt4/libQtCore.so.4
#18 0x085a8ce0 in ?? ()
#19 0x80196010 in ?? ()
#20 0x0859f690 in ?? ()
#21 0x0000000b in ?? ()
#22 0xb57c76db in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
#23 0xb57c7d44 in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
#24 0xb6da7980 in Digikam::DatabaseWatch::imageChange (this=0x859f690, _t1=@0x85a81d4, _t2=@0x85a81d8, _t3=@0xb27a915c) at /root/digikam-svn/graphics/digikam/build/digikam/moc_databasewatch.cpp:193
#25 0xb6da8152 in Digikam::DatabaseWatch::sendImageChange (this=0x859f690, cset=@0xb27a915c) at /root/digikam-svn/graphics/digikam/libs/database/databasewatch.cpp:241
#26 0xb6daea51 in Digikam::DatabaseBackendPrivate::sendToWatch (this=0x85a2058, changeset=@0xb27a915c) at /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:59
#27 0xb6daeb4a in Digikam::DatabaseBackendPrivate::ChangesetContainer<Digikam::ImageChangeset>::sendOut (this=0x85a20b8) at /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:94
#28 0xb6daec34 in Digikam::DatabaseBackendPrivate::transactionFinished (this=0x85a2058) at /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:112
#29 0xb70f4876 in Digikam::DatabaseCoreBackend::commitTransaction (this=0x85a05e0) at /root/digikam-svn/graphics/digikam/libs/database/databasecorebackend.cpp:1151
#30 0xb6daf53a in ~DatabaseTransaction (this=0xb27a9274) at /root/digikam-svn/graphics/digikam/libs/database/databasetransaction.cpp:57
#31 0xb6d932c6 in Digikam::CollectionScanner::completeScan (this=0xb27a92f8) at /root/digikam-svn/graphics/digikam/libs/database/collectionscanner.cpp:277
#32 0x08335a0f in Digikam::ScanController::run (this=0x8588f60) at /root/digikam-svn/graphics/digikam/digikam/scancontroller.cpp:541
#33 0xb56d7fc1 in ?? () from /usr/lib/qt4/libQtCore.so.4
#34 0x08588f60 in ?? ()
#35 0x00000000 in ?? ()

Thread 1 (Thread 0xb3bae700 (LWP 27729)):
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb567e8d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb56d739a in ?? () from /usr/lib/qt4/libQtCore.so.4
#3  0x0859fd80 in ?? ()
#4  0x0859fd68 in ?? ()
#5  0x00000000 in ?? ()
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> > > I have a large number of pictures spread out over a number of
> > > directories.
> >
> > 10^6? 10^7?
>
> About 5.5million
>

Ok, this explains it. That's a lot indeed.
The complete scan is performed within one database transaction, so the changes
become public only afterwards, and we need to store change notifications. That
works, but afterwards, with the current simple solution, notifications are
sent out one by one instead of collapsing them. This is when you run out of
memory.
I can try to optimize this situation.

Please note: These are a lot of pictures. It would be interesting where
digikam is hitting walls here ;-) Some operations require linear memory, e.g.
if you want to display all of your pictures in the view, the view will keep a
list of all entries in the collection, and the sort-filter model will also
need memory to do the sorting.

> > > 2)  Perform some kind of incremental scan so that if it crashes during
> > > the initial scan I could restart and pic up where it left off
> > > (eventually getting past the out of memory problems.)
> >
> > The solution should be not to get out of memory at all ;-)
>
> Yes, that would be ideal.  :-)
>
> Thank you,
>
> Todd
>
> Application: digiKam (digikam), signal: Aborted
> [Current thread is 0 (LWP 27729)]
>
> Thread 2 (Thread 0xb27a9b90 (LWP 27731)):
> [KCrash Handler]
> #5  0xffffe424 in __kernel_vsyscall ()
> #6  0xb5445c41 in raise () from /lib/libc.so.6
> #7  0xb5447428 in abort () from /lib/libc.so.6
> #8  0xb56d08f5 in qt_message_output () from /usr/lib/qt4/libQtCore.so.4
> #9  0xb56d09aa in qFatal () from /usr/lib/qt4/libQtCore.so.4
> #10 0xb56fd8c0 in QListData::realloc () from /usr/lib/qt4/libQtCore.so.4
> #11 0xb56fd9a7 in QListData::append () from /usr/lib/qt4/libQtCore.so.4
> #12 0xb57b9b57 in ?? () from /usr/lib/qt4/libQtCore.so.4
> #13 0x084f30e8 in ?? ()
> #14 0x0000002b in ?? ()
> #15 0xb57b65cc in QCoreApplication::postEvent () from
> /usr/lib/qt4/libQtCore.so.4 #16 0xb57b667a in QCoreApplication::postEvent
> () from /usr/lib/qt4/libQtCore.so.4 #17 0xb57c74d8 in ?? () from
> /usr/lib/qt4/libQtCore.so.4
> #18 0x085a8ce0 in ?? ()
> #19 0x80196010 in ?? ()
> #20 0x0859f690 in ?? ()
> #21 0x0000000b in ?? ()
> #22 0xb57c76db in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
> #23 0xb57c7d44 in QMetaObject::activate () from /usr/lib/qt4/libQtCore.so.4
> #24 0xb6da7980 in Digikam::DatabaseWatch::imageChange (this=0x859f690,
> _t1=@0x85a81d4, _t2=@0x85a81d8, _t3=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/build/digikam/moc_databasewatch.cpp:193
> #25 0xb6da8152 in Digikam::DatabaseWatch::sendImageChange (this=0x859f690,
> cset=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/libs/database/databasewatch.cpp:241 #26
> 0xb6daea51 in Digikam::DatabaseBackendPrivate::sendToWatch
> (this=0x85a2058, changeset=@0xb27a915c) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:59
> #27 0xb6daeb4a in
> Digikam::DatabaseBackendPrivate::ChangesetContainer<Digikam::ImageChangese
> t>::sendOut (this=0x85a20b8) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:94
> #28 0xb6daec34 in Digikam::DatabaseBackendPrivate::transactionFinished
> (this=0x85a2058) at
> /root/digikam-svn/graphics/digikam/libs/database/databasebackend_p.h:112
> #29 0xb70f4876 in Digikam::DatabaseCoreBackend::commitTransaction
> (this=0x85a05e0) at
> /root/digikam-svn/graphics/digikam/libs/database/databasecorebackend.cpp:1
> 151 #30 0xb6daf53a in ~DatabaseTransaction (this=0xb27a9274) at
> /root/digikam-svn/graphics/digikam/libs/database/databasetransaction.cpp:5
> 7 #31 0xb6d932c6 in Digikam::CollectionScanner::completeScan
> (this=0xb27a92f8) at
> /root/digikam-svn/graphics/digikam/libs/database/collectionscanner.cpp:277
> #32 0x08335a0f in Digikam::ScanController::run (this=0x8588f60) at
> /root/digikam-svn/graphics/digikam/digikam/scancontroller.cpp:541 #33
> 0xb56d7fc1 in ?? () from /usr/lib/qt4/libQtCore.so.4
> #34 0x08588f60 in ?? ()
> #35 0x00000000 in ?? ()
>
> Thread 1 (Thread 0xb3bae700 (LWP 27729)):
> #0  0xffffe424 in __kernel_vsyscall ()
> #1  0xb567e8d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0 #2  0xb56d739a in ?? () from
> /usr/lib/qt4/libQtCore.so.4
> #3  0x0859fd80 in ?? ()
> #4  0x0859fd68 in ?? ()
> #5  0x00000000 in ?? ()
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Todd Goodman]

* Marcel Wiesweg <[hidden email]> [100628 07:05]:

> > > > I have a large number of pictures spread out over a number of
> > > > directories.
> > >
> > > 10^6? 10^7?
> >
> > About 5.5million
> >
>
> Ok, this explains it. That's a lot indeed.
> The complete scan is performed within one database transaction, so the changes
> become public only afterwards, and we need to store change notifications. That
> works, but afterwards, with the current simple solution, notifications are
> sent out one by one instead of collapsing them. This is when you run out of
> memory.
> I can try to optimize this situation.
>
> Please note: These are a lot of pictures. It would be interesting where
> digikam is hitting walls here ;-) Some operations require linear memory, e.g.
> if you want to display all of your pictures in the view, the view will keep a
> list of all entries in the collection, and the sort-filter model will also
> need memory to do the sorting.

Thanks Marcel.  If I can help let me know.  It would be nice to perform
the transaction every N files.

I'll certainly keep an eye out for other things.  Once I'm up and
running I generally operate in subdirectories but will be happy to try
anything you'd like.

Thanks,

Todd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel