[digikam] [Bug 367853] New: Digikam hangs on 'Reading database' when stumbling across MacOS' Photos Library.photoslibrary of a huge size (>20GB)

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

            Bug ID: 367853
           Summary: Digikam hangs on 'Reading database' when stumbling
                    across MacOS' Photos Library.photoslibrary of a huge
                    size (>20GB)
           Product: digikam
           Version: 5.1.0
          Platform: Mac OS X Disk Images
                OS: OS X
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: Database-Scan
          Assignee: [hidden email]
          Reporter: [hidden email]

If installed on MacOSX (10.11.6 El Capitan) digikam seemed to freeze in the
process of reading the database after starting it for the first time with
default settings. This happens if digikam scans the default user's Pictures
folder AND in case one already has used the Apple application Photos before
that created a huge Photos Library.photoslibrary in the very same folder.
Photos Library.photoslibrary is a packaged db in itself. The size of this file
is in my  case 26 GB as I managed almost 10,000 photos with it.

One has to wait a looong time (about half an hour) to see digikam finish
reading the content of Photos Library.photoslibrary in its own database and
later on it seems to have some performance problems with so many photos or
maybe with this file that it represents as a folder structure.

It took me a long time to figure out that this is the reason for the apparent
freeze – at the beginning I just thought that digikam 5.0.1 is simply not
working on MacOSX El Capitan.

Reproducible: Always

Steps to Reproduce:
1. (for testing purpose one might) create a huge  Photos Library.photoslibrary
file in Pictures-folder (default) for many fotos (in my case almost 10.000)
with the Photos app of Apple MacOSX El Capitan  (it's done automatically if you
start it)
2. Install pre-compiled digikam app
3. Start digikam with default settings and let it scan the Pictures folder
(happens automatically)  with huge  Photos Library.photoslibrary in it

Actual Results:  
Digikam hangs on 'Reading database' during startup

Expected Results:  
At least the software should warn the user that it is going to take a very long
time to read a file of a certain size or package files like Photos
Library.photoslibrary and that it might cause performance problems later on and
give the opportunity to skip this file from being written to the databas/to
delete it/to move it elsewhere.

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Digikam hangs on 'Reading   |Digikam hangs on 'Reading
                   |database' when stumbling    |database' when stumbling
                   |across MacOS' Photos        |across MacOS' Photos
                   |Library.photoslibrary of a  |Library of a huge size
                   |huge size (>20GB)           |(>20GB)
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #1 from [hidden email] ---
Looking on my macbook pro how Apple iPhoto or Photos applications store the
data (image and metadata), it's look like a big mess, with a lots of redondant
subdirectories  relies on image, preview, thumbnails, sidecars for files and
folders, database, etc...

I think the best way to store images on my is to host all in dedicated
directory heirarchies, outside the extra data for application.

But of course, we need to be able to handle  pro photo applications.

On think that i know is the improvements to do in image scanner, to ignore some
king of files or directories, hidden or not. There is a non finalized patch
about this subject on this bugzilla entry :

https://bugs.kde.org/show_bug.cgi?id=123097

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #2 from [hidden email] ---
With current implementation we fix some lock with database on special
conditions discovered under Windows. I think this problem can also appears with
MacOS.

So i just build a new pre 5.2.0 installer for MacOS :

https://drive.google.com/file/d/0BzeiVr-byqt5RHU5LWRKaWh1YXM/view?usp=sharing

Please try with this version...

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

Lars Van Casteren <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from Lars Van Casteren <[hidden email]> ---
I encountered this problem also with a large Photos lib. Removed the folder
from the config to fix it.

I installed the new 5.2.1-01 and configured it with a local SQLite and pointed
it to $HOME\Pictures where my Photos lib is. It started importing, after a few
minutes it shot to 27% and incremented all the way up to 43% while the local
sqlite grew to 61Mb during that time.

After +- 15 min it stopped at 43% done, activity monitor reports it as 99%
usage & hanging / not responding and the sqlite db is no longer increasing in
size, stuck at 61Mb for > 5 minutes, the thumbnail db is at 76Kb. $iostat
doesn't show any significant disk access anymore so I guess it's dead.

Some stats on the Photos lib:
170.000 media files in 70.000 subfolders (that’s Masters,Previews & Thumbnails
dirs together)

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #4 from Andrea <[hidden email]> ---
Gilles,

 thank you very much for the new pre installer. I tried it. Unfortunately, the
problem remains. For the time being a hint like you proposed in the
releasenotes.html, a readme and the FAQ might suffice.

"The best way to store images on Mac Book Pro is to host them in dedicated
directory hierarchies, outside the extra data for other applications: Avoid
having e.g. iPhoto' or Photos' library in the same directory in which your
images are stored and scanned by digikam. "

Andrea

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #5 from Lars Van Casteren <[hidden email]> ---
I think the scanning actually went ok when seeing the console msg output.
The problem seems to appear after the folder scan, or during consequent startup
if you killed it.

This is the console after it finished scanning Photos, it seemed to actually
have finished.

digikam.database: Complete scan took: 1339440 msecs.
digikam.general: total scan value :  569565
digikam.database: Complete scan (file scanning deferred) took: 203521 msecs.
digikam.general: Event is dispatched to OSX desktop notifier
digikam.general: Camera XML data:  "/Users/wowbagger/Library/Application
Support/digikam/cameras.xml"
digikam.facedb: FaceDB SelectFaceSetting val ret =  0
digikam.facedb: FaceDB SelectFaceSetting val ret =  0
digikam.facedb: Face database: have a structure version  "2"
digikam.facesengine: Face database ready for use
digikam.facesengine: Face database ready for use
digikam.geoiface: "setting backend marble"
digikam.general: Stacked View Mode :  0
digikam.geoiface: "setting backend marble"
digikam.general: "browse_album"
digikam.general: "browse_tag"
digikam.general: "browse_labels"
digikam.general: "browse_date"
digikam.general: "browse_timeline"
digikam.general: "browse_search"
digikam.general: "browse_fuzzysearch"
digikam.general: "browse_gpssearch"
digikam.general: "browse_people"
digikam.widgets: Paths to color scheme :
("/opt/digikam/Applications/KF5/digikam.app/Contents/Resources//digikam/colorschemes")
digikam.widgets: ""  ::  ""
QFSFileEngine::open: No file name specified
digikam.dimg: Invalid variant value for device!
digikam.dimg: updating data
digikam.dimg: updating data
digikam.general: Using  4  CPU core to run threads
digikam.general: new search text settings:  "" : hasResult =  false , validRows
=  0
QFSFileEngine::open: No file name specified
digikam.geoiface: ----
digikam.geoiface: ----
digikam.general: Added root album called:  "Pictures"

This is a lldb bt:

* thread #1: tid = 0xd0980, 0x00000001065714ff QtCore`QHash<QString,
QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207, stop
reason = signal SIGSTOP
  * frame #0: 0x00000001065714ff QtCore`QHash<QString,
QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207
    frame #1: 0x000000010656f64b
QtCore`QtPrivate::QStringList_removeDuplicates(QStringList*) + 251
    frame #2: 0x0000000100df7a7d
libdigikamcore.5.2.0.dylib`___lldb_unnamed_function10472$$libdigikamcore.5.2.0.dylib
+ 29
    frame #3: 0x0000000100df7408
libdigikamcore.5.2.0.dylib`Digikam::ModelCompleter::slotDataChanged(QModelIndex
const&, QModelIndex const&) + 1560
    frame #4: 0x0000000100ea5e3f
libdigikamcore.5.2.0.dylib`___lldb_unnamed_function12112$$libdigikamcore.5.2.0.dylib
+ 207
    frame #5: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #6: 0x000000010675c69d
QtCore`QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex const&,
QVector<int> const&) + 61
    frame #7: 0x0000000100326621
libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::updateCount(Digikam::Album*)
+ 753
    frame #8: 0x00000001003266e5
libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::includeChildrenCount(QModelIndex
const&) + 117
    frame #9: 0x00000001003555bd
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13361$$libdigikamgui.5.2.0.dylib
+ 61
    frame #10: 0x000000010035548a
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13359$$libdigikamgui.5.2.0.dylib
+ 90
    frame #11: 0x0000000100355655
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13362$$libdigikamgui.5.2.0.dylib
+ 133
    frame #12: 0x00000001003562c3
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13381$$libdigikamgui.5.2.0.dylib
+ 51
    frame #13: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #14: 0x000000010675c88e
QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
QAbstractItemModel::QPrivateSignal) + 78
    frame #15: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() +
80
    frame #16: 0x00000001066590b6
QtCore`QSortFilterProxyModelPrivate::insert_source_items(QVector<int>&,
QVector<int>&, QVector<int> const&, QModelIndex const&, Qt::Orientation, bool)
+ 1430
    frame #17: 0x0000000106659b06
QtCore`QSortFilterProxyModelPrivate::source_items_inserted(QModelIndex const&,
int, int, Qt::Orientation) + 2070
    frame #18: 0x0000000106661d03
QtCore`QSortFilterProxyModel::qt_static_metacall(QObject*, QMetaObject::Call,
int, void**) + 1443
    frame #19: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #20: 0x000000010675c88e
QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
QAbstractItemModel::QPrivateSignal) + 78
    frame #21: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() +
80
    frame #22: 0x00000001003258ad
libdigikamgui.5.2.0.dylib`Digikam::AbstractAlbumModel::slotAlbumAdded(Digikam::Album*)
+ 93
    frame #23: 0x0000000100334cd7
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function12637$$libdigikamgui.5.2.0.dylib
+ 167
    frame #24: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #25: 0x000000010039174d
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::signalAlbumAdded(Digikam::Album*)
+ 77
    frame #26: 0x00000001003743b8
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::insertPAlbum(Digikam::PAlbum*,
Digikam::PAlbum*) + 328
    frame #27: 0x0000000100375782
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::scanPAlbums() + 4034
    frame #28: 0x0000000100374739
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::refresh() + 25
    frame #29: 0x000000010037402a
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::startScan() + 1578
    frame #30: 0x00000001000e8d71
libdigikamgui.5.2.0.dylib`Digikam::DigikamApp::DigikamApp() + 3953
    frame #31: 0x0000000100011b7f digikam`main + 7343
    frame #32: 0x00007fff9221b5ad libdyld.dylib`start + 1
    frame #33: 0x00007fff9221b5ad libdyld.dylib`start + 1

And this is the sqlite db:

sqlite> select count (*) from images;
120444

sqlite> select count(*) from albums;
106600

Maybe the Album parsing to create the tree make it hang/unresponsive due to the
very high amount of albums?

--
You are receiving this mail because:
You are the assignee for the bug.

[Gilles Caulier-4]

2016-08-27 13:50 GMT+02:00 Lars Van Casteren via KDE Bugzilla <[hidden email]>:

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #5 from Lars Van Casteren <[hidden email]> ---
I think the scanning actually went ok when seeing the console msg output.
The problem seems to appear after the folder scan, or during consequent startup
if you killed it.

This is the console after it finished scanning Photos, it seemed to actually
have finished.

digikam.database: Complete scan took: 1339440 msecs.
digikam.general: total scan value :  569565
digikam.database: Complete scan (file scanning deferred) took: 203521 msecs.
digikam.general: Event is dispatched to OSX desktop notifier
digikam.general: Camera XML data:  "/Users/wowbagger/Library/Application
Support/digikam/cameras.xml"
digikam.facedb: FaceDB SelectFaceSetting val ret =  0
digikam.facedb: FaceDB SelectFaceSetting val ret =  0
digikam.facedb: Face database: have a structure version  "2"
digikam.facesengine: Face database ready for use
digikam.facesengine: Face database ready for use
digikam.geoiface: "setting backend marble"
digikam.general: Stacked View Mode :  0
digikam.geoiface: "setting backend marble"
digikam.general: "browse_album"
digikam.general: "browse_tag"
digikam.general: "browse_labels"
digikam.general: "browse_date"
digikam.general: "browse_timeline"
digikam.general: "browse_search"
digikam.general: "browse_fuzzysearch"
digikam.general: "browse_gpssearch"
digikam.general: "browse_people"
digikam.widgets: Paths to color scheme :
("/opt/digikam/Applications/KF5/digikam.app/Contents/Resources//digikam/colorschemes")
digikam.widgets: ""  ::  ""
QFSFileEngine::open: No file name specified
digikam.dimg: Invalid variant value for device!
digikam.dimg: updating data
digikam.dimg: updating data
digikam.general: Using  4  CPU core to run threads
digikam.general: new search text settings:  "" : hasResult =  false , validRows
=  0
QFSFileEngine::open: No file name specified
digikam.geoiface: ----
digikam.geoiface: ----
digikam.general: Added root album called:  "Pictures"

This is a lldb bt:

* thread #1: tid = 0xd0980, 0x00000001065714ff QtCore`QHash<QString,
QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207, stop
reason = signal SIGSTOP
  * frame #0: 0x00000001065714ff QtCore`QHash<QString,
QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207
    frame #1: 0x000000010656f64b
QtCore`QtPrivate::QStringList_removeDuplicates(QStringList*) + 251
    frame #2: 0x0000000100df7a7d
libdigikamcore.5.2.0.dylib`___lldb_unnamed_function10472$$libdigikamcore.5.2.0.dylib
+ 29
    frame #3: 0x0000000100df7408
libdigikamcore.5.2.0.dylib`Digikam::ModelCompleter::slotDataChanged(QModelIndex
const&, QModelIndex const&) + 1560
    frame #4: 0x0000000100ea5e3f
libdigikamcore.5.2.0.dylib`___lldb_unnamed_function12112$$libdigikamcore.5.2.0.dylib
+ 207
    frame #5: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #6: 0x000000010675c69d
QtCore`QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex const&,
QVector<int> const&) + 61
    frame #7: 0x0000000100326621
libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::updateCount(Digikam::Album*)
+ 753
    frame #8: 0x00000001003266e5
libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::includeChildrenCount(QModelIndex
const&) + 117
    frame #9: 0x00000001003555bd
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13361$$libdigikamgui.5.2.0.dylib
+ 61
    frame #10: 0x000000010035548a
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13359$$libdigikamgui.5.2.0.dylib
+ 90
    frame #11: 0x0000000100355655
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13362$$libdigikamgui.5.2.0.dylib
+ 133
    frame #12: 0x00000001003562c3
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13381$$libdigikamgui.5.2.0.dylib
+ 51
    frame #13: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #14: 0x000000010675c88e
QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
QAbstractItemModel::QPrivateSignal) + 78
    frame #15: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() +
80
    frame #16: 0x00000001066590b6
QtCore`QSortFilterProxyModelPrivate::insert_source_items(QVector<int>&,
QVector<int>&, QVector<int> const&, QModelIndex const&, Qt::Orientation, bool)
+ 1430
    frame #17: 0x0000000106659b06
QtCore`QSortFilterProxyModelPrivate::source_items_inserted(QModelIndex const&,
int, int, Qt::Orientation) + 2070
    frame #18: 0x0000000106661d03
QtCore`QSortFilterProxyModel::qt_static_metacall(QObject*, QMetaObject::Call,
int, void**) + 1443
    frame #19: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #20: 0x000000010675c88e
QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
QAbstractItemModel::QPrivateSignal) + 78
    frame #21: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows() +
80
    frame #22: 0x00000001003258ad
libdigikamgui.5.2.0.dylib`Digikam::AbstractAlbumModel::slotAlbumAdded(Digikam::Album*)
+ 93
    frame #23: 0x0000000100334cd7
libdigikamgui.5.2.0.dylib`___lldb_unnamed_function12637$$libdigikamgui.5.2.0.dylib
+ 167
    frame #24: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*, int,
int, void**) + 742
    frame #25: 0x000000010039174d
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::signalAlbumAdded(Digikam::Album*)
+ 77
    frame #26: 0x00000001003743b8
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::insertPAlbum(Digikam::PAlbum*,
Digikam::PAlbum*) + 328
    frame #27: 0x0000000100375782
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::scanPAlbums() + 4034
    frame #28: 0x0000000100374739
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::refresh() + 25
    frame #29: 0x000000010037402a
libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::startScan() + 1578
    frame #30: 0x00000001000e8d71
libdigikamgui.5.2.0.dylib`Digikam::DigikamApp::DigikamApp() + 3953
    frame #31: 0x0000000100011b7f digikam`main + 7343
    frame #32: 0x00007fff9221b5ad libdyld.dylib`start + 1
    frame #33: 0x00007fff9221b5ad libdyld.dylib`start + 1

And this is the sqlite db:

sqlite> select count (*) from images;
120444

sqlite> select count(*) from albums;
106600

Maybe the Album parsing to create the tree make it hang/unresponsive due to the
very high amount of albums?

If it's the case, this can be a sqlite internal settings problem.

This will give the same result with Mysql internal to replace sqlite ? (not a remote Mysql server).

Mysql must be better to handle big set of data. Does it pass until the end of scan in this situation ?

Gilles Caulier

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #6 from [hidden email] ---
2016-08-27 13:50 GMT+02:00 Lars Van Casteren via KDE Bugzilla <
[hidden email]>:

> https://bugs.kde.org/show_bug.cgi?id=367853
>
> --- Comment #5 from Lars Van Casteren <[hidden email]> ---
> I think the scanning actually went ok when seeing the console msg output.
> The problem seems to appear after the folder scan, or during consequent
> startup
> if you killed it.
>
> This is the console after it finished scanning Photos, it seemed to
> actually
> have finished.
>
> digikam.database: Complete scan took: 1339440 msecs.
> digikam.general: total scan value :  569565
> digikam.database: Complete scan (file scanning deferred) took: 203521
> msecs.
> digikam.general: Event is dispatched to OSX desktop notifier
> digikam.general: Camera XML data:  "/Users/wowbagger/Library/Application
> Support/digikam/cameras.xml"
> digikam.facedb: FaceDB SelectFaceSetting val ret =  0
> digikam.facedb: FaceDB SelectFaceSetting val ret =  0
> digikam.facedb: Face database: have a structure version  "2"
> digikam.facesengine: Face database ready for use
> digikam.facesengine: Face database ready for use
> digikam.geoiface: "setting backend marble"
> digikam.general: Stacked View Mode :  0
> digikam.geoiface: "setting backend marble"
> digikam.general: "browse_album"
> digikam.general: "browse_tag"
> digikam.general: "browse_labels"
> digikam.general: "browse_date"
> digikam.general: "browse_timeline"
> digikam.general: "browse_search"
> digikam.general: "browse_fuzzysearch"
> digikam.general: "browse_gpssearch"
> digikam.general: "browse_people"
> digikam.widgets: Paths to color scheme :
> ("/opt/digikam/Applications/KF5/digikam.app/Contents/Resources//digikam/
> colorschemes")
> digikam.widgets: ""  ::  ""
> QFSFileEngine::open: No file name specified
> digikam.dimg: Invalid variant value for device!
> digikam.dimg: updating data
> digikam.dimg: updating data
> digikam.general: Using  4  CPU core to run threads
> digikam.general: new search text settings:  "" : hasResult =  false ,
> validRows
> =  0
> QFSFileEngine::open: No file name specified
> digikam.geoiface: ----
> digikam.geoiface: ----
> digikam.general: Added root album called:  "Pictures"
>
> This is a lldb bt:
>
> * thread #1: tid = 0xd0980, 0x00000001065714ff QtCore`QHash<QString,
> QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207,
> stop
> reason = signal SIGSTOP
>   * frame #0: 0x00000001065714ff QtCore`QHash<QString,
> QHashDummyValue>::insert(QString const&, QHashDummyValue const&) + 207
>     frame #1: 0x000000010656f64b
> QtCore`QtPrivate::QStringList_removeDuplicates(QStringList*) + 251
>     frame #2: 0x0000000100df7a7d
> libdigikamcore.5.2.0.dylib`___lldb_unnamed_function10472$$
> libdigikamcore.5.2.0.dylib
> + 29
>     frame #3: 0x0000000100df7408
> libdigikamcore.5.2.0.dylib`Digikam::ModelCompleter::
> slotDataChanged(QModelIndex
> const&, QModelIndex const&) + 1560
>     frame #4: 0x0000000100ea5e3f
> libdigikamcore.5.2.0.dylib`___lldb_unnamed_function12112$$
> libdigikamcore.5.2.0.dylib
> + 207
>     frame #5: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*,
> int,
> int, void**) + 742
>     frame #6: 0x000000010675c69d
> QtCore`QAbstractItemModel::dataChanged(QModelIndex const&, QModelIndex
> const&,
> QVector<int> const&) + 61
>     frame #7: 0x0000000100326621
> libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::
> updateCount(Digikam::Album*)
> + 753
>     frame #8: 0x00000001003266e5
> libdigikamgui.5.2.0.dylib`Digikam::AbstractCountingAlbumModel::
> includeChildrenCount(QModelIndex
> const&) + 117
>     frame #9: 0x00000001003555bd
> libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13361$$
> libdigikamgui.5.2.0.dylib
> + 61
>     frame #10: 0x000000010035548a
> libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13359$$
> libdigikamgui.5.2.0.dylib
> + 90
>     frame #11: 0x0000000100355655
> libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13362$$
> libdigikamgui.5.2.0.dylib
> + 133
>     frame #12: 0x00000001003562c3
> libdigikamgui.5.2.0.dylib`___lldb_unnamed_function13381$$
> libdigikamgui.5.2.0.dylib
> + 51
>     frame #13: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*,
> int,
> int, void**) + 742
>     frame #14: 0x000000010675c88e
> QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
> QAbstractItemModel::QPrivateSignal) + 78
>     frame #15: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows()
> +
> 80
>     frame #16: 0x00000001066590b6
> QtCore`QSortFilterProxyModelPrivate::insert_source_items(QVector<int>&,
> QVector<int>&, QVector<int> const&, QModelIndex const&, Qt::Orientation,
> bool)
> + 1430
>     frame #17: 0x0000000106659b06
> QtCore`QSortFilterProxyModelPrivate::source_items_inserted(QModelIndex
> const&,
> int, int, Qt::Orientation) + 2070
>     frame #18: 0x0000000106661d03
> QtCore`QSortFilterProxyModel::qt_static_metacall(QObject*,
> QMetaObject::Call,
> int, void**) + 1443
>     frame #19: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*,
> int,
> int, void**) + 742
>     frame #20: 0x000000010675c88e
> QtCore`QAbstractItemModel::rowsInserted(QModelIndex const&, int, int,
> QAbstractItemModel::QPrivateSignal) + 78
>     frame #21: 0x000000010663b740 QtCore`QAbstractItemModel::endInsertRows()
> +
> 80
>     frame #22: 0x00000001003258ad
> libdigikamgui.5.2.0.dylib`Digikam::AbstractAlbumModel::
> slotAlbumAdded(Digikam::Album*)
> + 93
>     frame #23: 0x0000000100334cd7
> libdigikamgui.5.2.0.dylib`___lldb_unnamed_function12637$$
> libdigikamgui.5.2.0.dylib
> + 167
>     frame #24: 0x00000001066c5796 QtCore`QMetaObject::activate(QObject*,
> int,
> int, void**) + 742
>     frame #25: 0x000000010039174d
> libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::
> signalAlbumAdded(Digikam::Album*)
> + 77
>     frame #26: 0x00000001003743b8
> libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::
> insertPAlbum(Digikam::PAlbum*,
> Digikam::PAlbum*) + 328
>     frame #27: 0x0000000100375782
> libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::scanPAlbums() + 4034
>     frame #28: 0x0000000100374739
> libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::refresh() + 25
>     frame #29: 0x000000010037402a
> libdigikamgui.5.2.0.dylib`Digikam::AlbumManager::startScan() + 1578
>     frame #30: 0x00000001000e8d71
> libdigikamgui.5.2.0.dylib`Digikam::DigikamApp::DigikamApp() + 3953
>     frame #31: 0x0000000100011b7f digikam`main + 7343
>     frame #32: 0x00007fff9221b5ad libdyld.dylib`start + 1
>     frame #33: 0x00007fff9221b5ad libdyld.dylib`start + 1
>
> And this is the sqlite db:
>
> sqlite> select count (*) from images;
> 120444
>
> sqlite> select count(*) from albums;
> 106600
>
> Maybe the Album parsing to create the tree make it hang/unresponsive due
> to the
> very high amount of albums?
>

If it's the case, this can be a sqlite internal settings problem.

This will give the same result with Mysql internal to replace sqlite ? (not
a remote Mysql server).

Mysql must be better to handle big set of data. Does it pass until the end
of scan in this situation ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #7 from Lars Van Casteren <[hidden email]> ---
I got it up & running with some major trickery using MariaDB 10.1.14  (lots of
manual config stuff & creating mysql. innoDB tables etc before DK would
continue) but in the end I get the same result when adding $HOME/Pictures
collection where the Photos library is located.
The scan hangs at the same 43% in DK frontend using the local MariaDB. Exactly
the same behaviour like SQLite.

MariaDB [(none)]> select count(*) from digikam.albums;
+----------+
| count(*) |
+----------+
|   106600 |
+----------+
1 row in set (0.02 sec)

106600 seemed like a strange round number to me for the amount of subfolder but
it checks out, $find . -type d | wc -l returns 106600 subfolders.

Another test I did was to create a large number of directories in a collection
to see what would happen.
*note: it might take a minute or 3 to create the 110000 subdirs...

$for ((i=1;i<=110000;i++)); do mkdir-p testdir/$i; done

I ran a clean DK instance with SQLite and point to testdir as a collection so
it would scan it and add the albums.
Same problem applies. It starts scanning but it hangs after a while and doesn't
seem to generate the (huge) tree although it’s just a single tree without
subfolders or images.

Rgds,
Lars

--
You are receiving this mail because:
You are the assignee for the bug.

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=367853

--- Comment #8 from [hidden email] ---
digiKam 5.4.0 pre-release is available in this repository :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Can you test with this version please ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.