[Bug 198063] New: Digikam startup is extremely slow
Locked
123
123
[Bug 198063] New: Digikam startup is extremely slow
 
|
https://bugs.kde.org/show_bug.cgi?id=198063
Summary: Digikam startup is extremely slow Product: digikam Version: unspecified Platform: Ubuntu Packages OS/Version: Linux Status: UNCONFIRMED Severity: normal Priority: NOR Component: general AssignedTo: [hidden email] ReportedBy: [hidden email] Version: (using KDE 4.2.2) Compiler: gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3 OS: Linux Installed from: Ubuntu Packages I have a collection of about 24000 images. Each time I start up digikam it takes approx. 2+ minutes to get past the initial splashscreen. The info displayed during the long wait is 'reading database'. I am running ubuntu 9.04 64bit on an AMD Phenom Quadcore @ 2.4Ghz with 8GB of RAM and a Seagate ST3500320AS 500GB SATA Drive with reiserfs 3 as filesystem. Subsequent starts of digikam are much faster, but this is only due to the fact that linux caches all the file information previously fetched. If I force the caches flushed, I always have the > 2min wait time. I have the option 'Scan for new items at startup' deselected, so there should be no reason for any filesystem search on startup. I did use strace to monitor the system calls, digikam performs during initialization and I discovered that digikam does a stat on each and every file in its database, thus effectively performing a scan. I doublechecked with the option 'Scan for new items at startup' turned on, and indeed, in this case the files get stat'ed 2 twice. This behaviour never happened with any digikam version < 0.10 and I am using digikam now from its earliest versions. Is there anything I can do to have my startup times reduced to a bearable time ? -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
https://bugs.kde.org/show_bug.cgi?id=198063
Gilles Caulier <[hidden email]> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|general |Database Version|unspecified |0.10.0 -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #1 from Martin Lubich <martin lubich gmx at> 2009-06-29 21:26:42 --- I just changed the filesystem where my collection resides from reiserfs 3 to ext3 and had an enormous increase in startup speed. It now takes just ~3 sec to pass the 'Reading database' stage. So this seems to be an issue related to reiserfs filesystems, allthough the stat system calls still are in place. ext3 just seems to handle them better. Might be interesting how other fs behave ... -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #2 from Gilles Caulier <caulier gilles gmail com> 2009-06-29 22:20:44 --- Martin, Where is stored you database file ? In Ext3 or reiserfs ? Gilles Caulier -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #3 from Martin Lubich <martin lubich gmx at> 2009-06-30 08:47:53 --- The database is stored in the same volume as my main photo collection, so it was on reiser and is now on ext3. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #4 from Gilles Caulier <caulier gilles gmail com> 2009-06-30 08:58:38 --- Very interesting. What's happen if Database file is back on reiserfs, image collection hosted by ext3 ? It still faster ? Gilles Caulier -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #5 from Martin Lubich <martin lubich gmx at> 2009-06-30 09:11:07 --- I'll check it out this evening. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #6 from Andi Clemens <andi clemens gmx net> 2009-06-30 09:22:45 --- For me reiserfs was always the slowest FS I ever tested, especially when it gets older. But this doesn't help you now :D If your reiserfs Partition isn't that big, maybe you can backup its data to an external drive, remove and re-create the reiserfs partition and copy the data back? This would defrag your system. I do this sometimes for my /var partition, the only one in my system with reiserfs on it. After that, I mostly have a speedup of 2-4. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #7 from Martin Lubich <martin lubich gmx at> 2009-06-30 09:33:20 --- The reiserfs partition I was using was indeed very old ~3-4 years. As I was planning to move to ext3/4 anyways, I have now moved my whole collection to the new fs, just as you suggested. That is where I noticed this massive speedup. I just think its maybe not a good idea to rely on the type or state of a fs to get decent startup times in digikam for huge collections. The typical use for these sort of collections is that they tend to sit very long on its fs and gets bigger all the time. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #8 from Andi Clemens <andi clemens gmx net> 2009-06-30 09:52:04 --- Sure, but even with 24000 images and more I have not such speed issues like you have. Not even with reiserfs. But note: I only tested different filesystems for a short time, not for over 4 years. My main FS was ext3 and now is ext4. If your reiserfs partition got so slow, it is due to fragmentation, which happens a lot on reiser3. What I want to say is that digiKam doesn't rely on the type of state of a filesystem, but it relies ON the filesystem. If your FS is slow and fragmented, any program will become slower, not only digiKam. What should we do about this? We use simply the Qt and KDE architecture provided to us, and that's it. We don't do FS specific things that slow down reiser but not the other ones :) Reiser was never optimum for something like an image or music collection, at least for me. It always was the slowest choice. If reiser gets fragmented, it's even worse. There is a besetting rumor that Linux FS don't fragment. This is just not true, but everybody will say it is so. Ext4 is the first FS that says it can fragment and delivers an online defragmenter, to avoid this. Now everybody goes like: "Uuuh, ext4 sucks, because it is fragmenting!" There is no FS in the world that does NOT fragment, but some do more, some do less. And windows does a lot and real quick ;-) This is why ext3, reiser and all the FS that don't provide defragmenting techniques, become slower over the time. I've seen this for many years, it just takes longer than with FAT32 or NTFS. What I wanted to say here is that digiKam can't do anything about this. The new version is starting up slower, that is true, but for me, all KDE4 apps do so. As you mentioned above, there seems to be a whole collection scan going on. I can confirm this, and I guess it is KDirWatch who is doing this. For me (50.000 images), it is not noticeable, but maybe on a heavily fragmented FS, it is. Marcel, is it possible that we still scan the collection, even though it is disabled? Andi -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #9 from Martin Lubich <martin lubich gmx at> 2009-06-30 10:12:29 --- Being a long time linux user and developer I am well aware of the misconception regarding fragmentation in linux fs, but thanks anyway for the sum up :) I suppose then, that the checked infrastructure kde4 now provides has a lot to do with this behaviour. Note that digikam 0.9.x did not show this problem and I used it with the same collection which gave me the slow startup now. Regarding the actually scan, digikam does respect the 'do not scan at startup' option. With this option enabled it just scans it twice. Its quite easy to see with a strace'd run. So this may really be a problem with the kde4 KDirWatch doing a stat, where the kde3 did not ( very oversimplified put :) ) -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #10 from Martin Lubich <martin lubich gmx at> 2009-06-30 22:51:20 --- I tested now with the collection residing on ext3 and the database on a reiser fs. The fast startup time is still there. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #11 from Andi Clemens <andi clemens gmx net> 2009-07-01 14:19:35 --- When using strace -c, you get some kind of "call graph". Sure it is not that accurate then using callgrind or gprof, but at least it give some useful information: % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 38.71 452.126188 10665 42393 508 stat64 15.53 181.369525 10659 17016 9687 read 14.35 167.568354 7106 23580 17458 access 5.01 58.565473 10666 5491 clock_gettime 4.08 47.689382 10647 4479 select 2.51 29.307504 10665 2748 241 lstat64 2.41 28.108221 10603 2651 writev 2.24 26.171927 10665 2454 gettimeofday 1.85 21.575460 10665 2023 399 open 1.61 18.791730 10665 1762 _llseek 1.57 18.293895 10667 1715 time 1.50 17.525235 10545 1662 close 1.14 13.309920 10665 1248 fstat64 1.06 12.403395 10665 1163 mmap2 0.99 11.562332 10666 1084 getdents 0.81 9.481194 10665 889 fcntl64 0.56 6.531191 10483 623 write 0.49 5.769765 10665 541 statfs 0.48 5.641834 10665 529 inotify_add_watch 0.48 5.641785 10665 529 inotify_rm_watch 0.45 5.303356 10481 506 munmap 0.38 4.490636 10769 417 poll 0.35 4.126428 10089 409 54 futex 0.30 3.512119 10675 329 ioctl 0.17 2.026389 10665 190 15 unlink 0.17 1.979691 10701 185 brk 0.10 1.141155 10665 107 mprotect 0.09 1.098495 10665 103 getcwd 0.08 0.938520 10665 88 uname 0.07 0.778545 10665 73 fchmod 0.07 0.767920 10666 72 link 0.07 0.767880 10665 72 fstatfs 0.03 0.383940 10665 36 clone 0.03 0.351945 10665 33 getuid32 0.02 0.255960 10665 24 socket 0.02 0.255960 10665 24 shmctl 0.01 0.170640 10665 16 getpeername 0.01 0.159975 10665 15 bind 0.01 0.159975 10665 15 listen 0.01 0.149310 10665 14 accept 0.01 0.149310 10665 14 getsockname 0.01 0.149310 10665 14 getsockopt 0.01 0.095985 10665 9 4 connect 0.01 0.095985 10665 9 semop 0.01 0.095985 10665 9 semctl 0.01 0.085320 10665 8 readlink 0.01 0.063990 10665 6 mlock 0.01 0.063990 10665 6 geteuid32 0.01 0.063990 10665 6 madvise 0.01 0.063990 10665 6 getdents64 0.01 0.063990 10665 6 shmat 0.01 0.063990 10665 6 shmdt 0.01 0.063990 10665 6 shmget 0.00 0.053325 10665 5 sched_get_priority_min 0.00 0.053325 10665 5 getgid32 0.00 0.042660 10665 4 kill 0.00 0.042660 10665 4 fdatasync 0.00 0.042660 10665 4 sched_get_priority_max 0.00 0.031995 10665 3 getegid32 0.00 0.031995 10665 3 clock_getres 0.00 0.031995 10665 3 semget 0.00 0.021330 10665 2 umask 0.00 0.021330 10665 2 rt_sigprocmask 0.00 0.010665 10665 1 execve 0.00 0.010665 10665 1 chmod 0.00 0.010665 10665 1 lseek 0.00 0.010665 10665 1 rename 0.00 0.010665 10665 1 1 mkdir 0.00 0.010665 10665 1 pipe 0.00 0.010665 10665 1 sched_getparam 0.00 0.010665 10665 1 sched_getscheduler 0.00 0.010665 10665 1 getrlimit 0.00 0.010665 10665 1 fchown32 0.00 0.010665 10665 1 set_thread_area 0.00 0.010665 10665 1 set_tid_address 0.00 0.010665 10665 1 inotify_init 0.00 0.010665 10665 1 set_robust_list 0.00 0.010665 10665 1 SYS_331 0.01 0.135977 9713 14 rt_sigaction 0.00 0.029327 7332 4 getpid ------ ----------- ----------- --------- --------- ---------------- 100.00 1168.071528 117481 28367 total I started digiKam and immediately closed it again. As you can see, stat64 took nearly 40% of all the executed calls. This would explain why the KDE4 version, although scanning is disabled, is much slower then the KDE3 version (as Martin already mentioned). KDE4 seems to do a lot more then KDE3 did here. I never really worked with the KDE3 code of digiKam so I don't know which techniques we used there. Also I'd like to mention that loading the plugins is slower then in KDE3, too. Maybe we could load them parallel? We only have 20 plugins, wouldn't it be possible to load them all at once by using threads? They don't depend on each other, so we actually don't need to wait before another plugin can be loaded. Or wouldn't this be possible? Andi -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #12 from Gilles Caulier <caulier gilles gmail com> 2009-07-01 14:28:21 --- Here, KDE4 loading plugin time is the same than KDE3. You can only see a difference if you compile with all debug symbol. In this case it's slower. Plugin interface is deleguate to KDE api. No way to use thread here. Scanning collection is really the problem. What's "stat64" exactly ? Gilles Caulier -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #13 from Andi Clemens <andi clemens gmx net> 2009-07-01 16:13:52 --- Hmm I just fired up my virtual machine with OpenSUSE 11.1 (which still has digiKam 0.9.4 in the repos) and run strace -c on it. I get (with the same collection) nearly the same result, still it starts a lot faster. Maybe it is the database that takes so long now? It has become bigger in KDE4 (even without the thumbsDB, fingerprints is really huge). I will backup my db now and reset the fingerprints table. It should be nearly the same size as the KDE3 version. Maybe it will start faster. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #14 from Andi Clemens <andi clemens gmx net> 2009-07-01 16:31:23 --- Without fingerprints data and without thumbsDB it is still slow, so it might not be an database loading issue. I guess we can't do anything about that if it is KDE related? -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #15 from Gilles Caulier <caulier gilles gmail com> 2009-07-01 16:39:36 --- And in this case what are strace results ? still stat64 on the top ? What's "stat64" exactly ? Gilles -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #16 from Gilles Caulier <caulier gilles gmail com> 2009-07-01 16:46:21 --- ok. look there : http://linux.die.net/man/2/stat64 If this function is called from QFileInfo or KDE api, we can do anything. If this function is called from Exiv2, this can optimized... I remember some fopen used in DImgLoader to check if file exist. I remember too that calls are redondant between DImg::load() and each DImgLoader::load(). Code must be factored to optimize. But why this function take a while ? This is certainly relevant of file system used. Right ? Gilles -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
--- Comment #17 from Andi Clemens <andi clemens gmx net> 2009-07-01 16:55:17 --- Yes, this might be filesystem relevant. On reiserfs for me digiKam startup is much slower then ext3/4. My collection I check here at the moment has 35.687 items in it. I just looked at the strace output without fingerprints and thumbsDB, it is the same. We stat every single item in the database, although "scan on startup" is turned off. When digiKam is actually shown, we get another load of stat64 calls, this is because we fill the album thumbnails and the iconview. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
[Bug 198063] Digikam startup is extremely slow
|
|
In reply to this post by Bugzilla from martin.lubich@gmx.at
https://bugs.kde.org/show_bug.cgi?id=198063
Marcel Wiesweg <[hidden email]> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |[hidden email] --- Comment #18 from Marcel Wiesweg <marcel wiesweg gmx de> 2009-07-01 18:49:43 --- callgrind won't give us information about time spent with filesystem access, but it tells us from where stat() is called. Results of a digikam startup (without collection scanning) and shutdown: 44754 calls of __xstat in libc.so of which 6547 from KStandardDirs 33288 from QFSFileEnginePrivate, called from QFileInfoPrivate::getFileFlags Now switching to the "All callers" view for getFileFlags: 94% of all costs spent in getFileFlags is spent when called from KDirWatch::addDir; indeed, KDirWatchPrivate::addEntry calls QDir::entryInfoList. It seems adding a directory to KDirWatch with the WatchSubDirs flag causes a stat for all contained files. -- Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug. _______________________________________________ Digikam-devel mailing list [hidden email] https://mail.kde.org/mailman/listinfo/digikam-devel |
«
Return to digikam-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |