digiKam › digikam-users

Digikam startup takes ~ 30 hours

Classic

List

Threaded

7 messages Options

Benjamin Harders

Digikam startup takes ~ 30 hours

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

For the last start-up of digikam I had to wait about 30 hours. *that is
not funny*
I was a little little bit not amused, when digikam crashed one hour
after the successful start, while I changed one word in the tag filter
field on the right bottom - obviously too quickly.
Digikam uses 80M RAM after a few minutes of start-up procedure. It
increases kilobyte by kilobyte until it comes to more than 400M RAM
after 30 hours. All the time there is a little network traffic ~
300kbit/s in both directions.

Some facts on my system:
Suse 11.4 64bit 4G RAM
NAS Synology 410 Gbit LAN
Mysql on that NAS
~ 360000 photos
Digikam 1.9.0-8.14

i need your help to find out something...

Thank you for some hints!

Ben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNyStyAAoJEJiq7tQPRElPCkMH/39YEE7oViYAgSHSIxMnjSNG
Y8WXmJmhwLX/g72xnLkk9L0nvt2IFf+cEslmdaJbBtR30SPppzR3OyEAcm5DS/mN
19cJWU4axmzNqlrelMsY4s0/XWQsq//QSmMKxD1gcaYBQnK12gCEiho9Bo5aGDyy
hVvn9Cb2c6Hx3pFG9RtPh9CFr2v2WzxQUCwU+31B9gKUxBWavZ3TBuL6MDkaezQn
n+Eg5Sr31mE0tt6fb7KvCvZCJkxO8H3IuKjLgbZN3ljRIW8y2FXzssvvbcSC7Nq/
Isv8iHbcuj+4HKzCg2jnFrIaIer5y4frPOVLdkEthrloftIxhDfXPuTH/3Sc1ls=
=ycVR
-----END PGP SIGNATURE-----
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

David Kerber-3

Re: Digikam startup takes ~ 30 hours

On 5/10/2011 8:11 AM, Benjamin Harders wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi!
>
> For the last start-up of digikam I had to wait about 30 hours. *that is
> not funny*
> I was a little little bit not amused, when digikam crashed one hour
> after the successful start, while I changed one word in the tag filter
> field on the right bottom - obviously too quickly.
> Digikam uses 80M RAM after a few minutes of start-up procedure. It
> increases kilobyte by kilobyte until it comes to more than 400M RAM
> after 30 hours. All the time there is a little network traffic ~
> 300kbit/s in both directions.
>
> Some facts on my system:
> Suse 11.4 64bit 4G RAM
> NAS Synology 410 Gbit LAN
> Mysql on that NAS
> ~ 360000 photos
> Digikam 1.9.0-8.14
>
> i need your help to find out something...
>
> Thank you for some hints!

Do you have a large picture library that it is indexing? My first
startup after adding a bunch of pics takes a lot longer, as it goes
through and indexes them. In my case, the pics are over a 100Mbps
network link, which probably slows it down a bit compared to being on a
local HD.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

David Kerber

Re: Digikam startup takes ~ 30 hours

In reply to this post by Benjamin Harders

On 5/10/2011 8:11 AM, Benjamin Harders wrote:

Martin Burnicki

Re: Digikam startup takes ~ 30 hours

David Kerber wrote:

> On 5/10/2011 8:11 AM, Benjamin Harders wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi!
> >
> > For the last start-up of digikam I had to wait about 30 hours. *that is
> > not funny*
> > I was a little little bit not amused, when digikam crashed one hour
> > after the successful start, while I changed one word in the tag filter
> > field on the right bottom - obviously too quickly.
> > Digikam uses 80M RAM after a few minutes of start-up procedure. It
> > increases kilobyte by kilobyte until it comes to more than 400M RAM
> > after 30 hours. All the time there is a little network traffic ~
> > 300kbit/s in both directions.

I have seen such behaviour before.

Have you run the function before which collects duplicates all over your
collection? The results are obviously saved in the database, and then loaded
again every time you start DK.

The only fix for me to get rid of this huge amount of search results was to
delete and re-create the data base. If you do so take care your metatadata
have been saved to the image files, otherwise they are lost when you delete
and recreate the database.

Martin
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Benjamin Harders

Re: Digikam startup takes ~ 30 hours

In reply to this post by David Kerber

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks for your replies!

David:
> Do you have a large picture library that it is indexing?

I do have a large library. The number of entries in my database are a
bit confusing:
e.g.:
780,941 ImageCopyright
636,169 ImageInformation
223,925 ImageProperties
615,427 Images
178,070 ImageTags
244,814 Thumbnails (3,5 GiB)
244,484 UniqueHashes
total: 3,8 GiB

360,000 files on my NAS
Actually I don't believe, that digikam is re-indexing. I didn't asked it
to do that. Indeed there are new pice added by the NAS, that is indexing
and creating five thumbnails of each photo. These thumbnails are stored
in @eadir named sub-directories for each photo. In the last hours of
start-up procedure I did not see any changes in the digikam database.
The first time when I had created the database with about 80,000 photos
it took digikam only a few hours to create the entries of the database.

Martin:
> I have seen such behaviour before.
> Have you run the function before which collects duplicates all over your
> collection? The results are obviously saved in the database, and then loaded
> again every time you start DK.

No, I did not use that function. But I planned to use it in the future.
The database's table Searches has three entries.
_Current_ ( Search_View | Time_Line | Map ) _Search_

Maybe i have to wait until the NAS finished indexing. Or I will drop the
table and let digikam re-index all folders again.

By the way: Since I am using mysql for digikam, every time I create a
new tag I have to wait five to ten minutes until digikam works again.
Writing an existing tag of my database to one picture, takes one second.

Thanks again,
Ben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNyWsxAAoJEJiq7tQPRElPvd4H/A2n3J5s6LCO/X6WEvJOhK5x
OoMoUDNA0e+Az4fvEU3V3g5fKPSr86mx1REF979MB+movhBdLuUgdYWnj+F3Q9JX
ageGv6iFdEMtysOlzlQ5rYseBG1Ml5EpHxXSpHMVXGxXTvvXI65P59A1ZkRg3utA
sRTRtRdh9H4wNZeCq18cFbh2P/q+E4TKIeaWWpI9BJTrs0oBNjGEG5/mEcGcCCJh
Zlg2AjRPcJ55i72YNdukX/iduGTQq2aIVYLIqMnY2HdNnHCQsvxlPA9BM0zNmvh8
toQLUwbDQ8sBNzW+axEoD6EHgih5ydrf2+h+8d9h0lbdN4pim9yNAW3LS+2TZ2w=
=LsGq
-----END PGP SIGNATURE-----
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Benjamin Harders

Re: Digikam startup takes ~ 30 hours

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Maybe i have to wait until the NAS finished indexing. Or I will drop the
> table and let digikam re-index all folders again.
>
> By the way: Since I am using mysql for digikam, every time I create a
> new tag I have to wait five to ten minutes until digikam works again.
> Writing an existing tag of my database to one picture, takes one second.

Maybe someone got the some probs. I deleted the database and let digikam
re-index all photos. After about one hundred thousand files (within a
few hours in total) it slowed down extremly to a few pics per minute. In
the sql-process list i found this entry with a run-time of about one minute:
REPLACE INTO TagsTree SELECT node.id, parent.pid FROM Tags AS node, Tags
AS ...
Something is wrong with this sql statement.
https://bugs.kde.org/show_bug.cgi?id=258409#c12
Is the whole TagsTree build new for every new photo?
This may explain my problem with the creation of new tags.

Now I removed the last ten thousand pics including meta tags from the
dir and let digikam finish the index on the other three hundred thousand
files without included meta tags. This took only about two hours. After
that digikam did not show up. I still can see the splash screen "reading
database". Obviously digikam shows some very slow (~300 KiB/s) activity
towards the NAS. Both systems do not fully use their processors (~20% -
30%). In the database In the database I have three sleeping connections
of digikam (13596 - 20671 sec); no other activity. Do you have an
explanation for this strange behavior, producing little network traffic
between my PC and the NAS? What is digikam supposed to do after indexing
new files? Nfsd on the NAS indicates some file system operations. But
for what?

I am praying, that this start-up will not take three days to succeed...

Ben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNy9jaAAoJEJiq7tQPRElPlDEIAIwxNYX1T3qDWR84BTOYLghh
Exf2u1Y1xwJ9ilHjWfed/VQ5g3KFHoDi6qDFjNXV3BZdgttNKr4hIKx78ezJkn7y
E4Zfhl5kHa/kuOUsHYeG/QWTTU50n7aH70FQHZ8ZmHXJ6zrkaXzQQTZ4uNfZE5bQ
rxXcVLSxJ2dRQHqT2KAhQmVFlKmnqHU3EGyY8fiG02R7LfFUgYcGy6PFBMpYf2WN
BBMcf5/xhaCOag6vYVLn9hk80ULsBkLp5Sz8+M1vJWlrvRoGCZF/PnrcP6g1NoD8
aORkmeDDiTdT4UYqIdonhWm4It3ekWhr414avGKgMdmAejEsVWvxSASweXcfDXA=
=znrP
-----END PGP SIGNATURE-----
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

Benjamin Harders

Re: Digikam startup takes ~ 30 hours

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

we are still having difficulties with digikam. It did re-index the whole
collection as I described some weeks ago. All database entries were set,
but I still had no digikam that showed up, only that nice splash screen
is "reading database". Having lousy network traffic of about 300 KiB/s
produced by dk I did a strace. It seems that there are a lot of timeouts
and it is searching and testing write access (?) thru the
sub-directories with the thumbnails produced by my NAS for each photo.
These files are owned by root and have chmod 777.

strace -p #:

[...]
select(11, [10], NULL, NULL, {0, 0}) = 0 (Timeout)
write(10, "X\0\1\0c\320\2\0N\0/home/fotos/date/2003-"..., 88) = 88
select(11, [10], NULL, NULL, {0, 0}) = 0 (Timeout)
lstat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG",
{st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG",
{st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
open("/home/fotos/date/2003-10/@eaDir/P1010287.JPG",
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 37
statfs("/home/fotos/date/2003-10/@eaDir/P1010287.JPG",
{f_type="NFS_SUPER_MAGIC", f_bsize=32768, f_blocks=59943779,
f_bfree=46741105, f_bavail=46737905, f_files=121798656,
f_ffree=120667253, f_fsid={0, 0}, f_namelen=255, f_frsize=32768}) = 0
getdents64(37, /* 7 entries */, 32768) = 288
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG/SYNOPHOTO:THUMB_S.jpg",
{st_mode=S_IFREG|0777, st_size=4269, ...}) = 0
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG/SYNOPHOTO:THUMB_XL.jpg",
{st_mode=S_IFREG|0777, st_size=263718, ...}) = 0
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG/SYNOPHOTO:THUMB_M.jpg",
{st_mode=S_IFREG|0777, st_size=19782, ...}) = 0
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG/SYNOPHOTO:THUMB_B.jpg",
{st_mode=S_IFREG|0777, st_size=42461, ...}) = 0
getdents64(37, /* 0 entries */, 32768) = 0
close(37) = 0
stat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG/SYNOPHOTO:THUMB_L.jpg",
{st_mode=S_IFREG|0777, st_size=106866, ...}) = 0
open("/etc/mtab", O_RDONLY) = 37
fstat(37, {st_mode=S_IFREG|0644, st_size=1275, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fd15b57b000
read(37, "/dev/sda2 / ext3 rw,acl,user_xat"..., 4096) = 1275
read(37, "", 4096) = 0
close(37) = 0
munmap(0x7fd15b57b000, 4096) = 0
lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/fotos", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/fotos/date", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/fotos/date/2003-10", {st_mode=S_IFDIR|0777, st_size=4096,
...}) = 0
lstat("/home/fotos/date/2003-10/@eaDir", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0
lstat("/home/fotos/date/2003-10/@eaDir/P1010287.JPG",
{st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
select(11, [10], NULL, NULL, {0, 0}) = 0 (Timeout)
write(10, "X\0\1\0d\320\2\0N\0/home/fotos/date/2003-"..., 88

It goes thru about 2000 files per hour. In one week it might be finished
with all of them.
I would like to understand, what dk is doing all the time.

Any clue or hint?

Thanks again.

Ben
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJN5OnzAAoJEJiq7tQPRElPV/UH/1Gpi+AvO+QV5rzN3ac6h1Ou
30dEZV/hDOUDQ+/v/uNLqrCbQFoiSmPHD65tQFWZlbOAvhtCfkris6qzPRpobZS1
kuyb0ZexMHgRloDWZYpcG1Wbd3pPccO1n1kjMXnjRwUulrZbgRu/2xbZL8MHZSVP
dscc8M4oqzazMXT13V7aP9pfpZkOw3uQDO6EBQmo5Izi1VijAoigajovPBOSKWyO
ceKfudS7D6O0HGSgnmsBKn4gr8Fyh0iMoNP+biPwEkZEyAxXotNuw65jHe1K5pfX
4gAfLp0lihQ0MKgSzRfufv0QuyuD46SC5YfKtwp6GU85AFubZYGXtxMcKR8pAzk=
=15wE
-----END PGP SIGNATURE-----
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users