[Bug 160966] New: some searches don't work properly with the new sqlite3-3.5.8

classic Classic list List threaded Threaded
72 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-25 22:05 -------
It is impossible for me to compile current SVN tree. When I come to the greycstoration-plugin, the compiler "hangs" and nothing is happening anymore. It just keeps on using 100% CPU and nothing actually happens. After 1 hour I canceled make and replayed CImg 1.2.7, but then compilation fails because some functions are not recognized anymore. But it doesn't hang, so maybe CImg is broken??
Has anybody the same problem compiling greycstoration? Or is there a way to skip it?
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
Hi Andi,

never had that type of problem.
It should be possible to disable that by
editing digikam/imageplugins/Makefile  and removing
`restoration` from the list of SUBDIRS

Good luck,

Arnd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From arnd.baecker web de  2008-04-25 22:39 -------
Hi Andi,

never had that type of problem.
It should be possible to disable that by
editing digikam/imageplugins/Makefile  and removing
`restoration` from the list of SUBDIRS

Good luck,

Arnd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-25 23:09 -------
No, it is still trying to compile greycstoration... I know that compilation of SVN worked for my 10 days ago (with the old CImg version), but the current tree is not compiling... grrrml... :-)

I just want to test the sql queries and now nothing is working anymore. My old digikam is not working because of the new SVN libs, and the current version is not compiling... hmmm... never had such problems with compiling software... this is so typical: you really need to compile a piece of software once only and right then all hell brakes loose... :-)
I will give it another try tomorrow...
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

Re: [Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
sorry, I was too quick in my previous reply, presumably it is
the one in digikam/libs/Makefile where  `greycstoration`
needs to be kicked out from SUBDIRS.

I am not sure, why things get so complicated for your system,
but I am optimistic that the compile will eventually work -
Thanks a lot for your patience and effort!

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From arnd.baecker web de  2008-04-25 23:19 -------
sorry, I was too quick in my previous reply, presumably it is
the one in digikam/libs/Makefile where  `greycstoration`
needs to be kicked out from SUBDIRS.

I am not sure, why things get so complicated for your system,
but I am optimistic that the compile will eventually work -
Thanks a lot for your patience and effort!
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-25 23:26 -------
Well I removed 'restoration' from digikam/imageplugins/Makefile AND 'greycrestoration' from digikam/libs/Makefile, but then showfoto is complaining that it needs libgreyc... or something like that. If I also remove showfoto from digikam/Makefile, my build system is REALLY getting angry :-)

Well if it is not working for me tomorrow, I will set up a little virtual machine and test if compilation works in there. If it does, well then I don't know what is wrong, but maybe I will be able to get to the actual testing needed for this bugreport :-)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 01:21 -------
AAAHH!! What's wrong now? I downloaded 0.9.4beta3 from sf.net because SVN is not working, but now I can't see the SQL-Query in the console. Am I doing something wrong? I compiled digikam using

./configure --prefix=/opt/kde --enable-debug=full

and added the line Arnd mentioned above:

        QStringList values;
        QString     errMsg;
        kdWarning() << "SQL Query " << sqlQuery << endl;
        if (!m_db.execSql(sqlQuery, &values, &errMsg))
        {
            error(KIO::ERR_INTERNAL, errMsg);
            return;
        }

But no output is printed to the console. Is this the wrong position or do I have to configure something else? I also changed kdWarning() to kdDebug() but still no output.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From arnd.baecker web de  2008-04-26 06:48 -------
Weird with the svn, anyway, you are getting closer.... ;-)
Note that there are two places in digikamsearch.cpp
with a m_db.execSql. Did you add the output to both?

The other possibility coming to my mind is that the
paths have to be set properly and a kdeinit has to be issued
(i.e., it could be that still the old kioslave is active,
but I am not sure on this, which I always considered to be a mess ... ;-):

export DIGIKAMDEST=here_the_path_used_on_configure.....
export KDEDIRS=$DIGIKAMDEST:/usr
export LD_LIBRARY_PATH=$DIGIKAMDEST/lib:$LD_LIBRARY_PATH
export LDPATH=$DIGIKAMDEST/lib:$LDPATH
export PATH=$DIGIKAMDEST/bin:$PATH
kdeinit

HTH, Arnd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 10:07 -------
I've added the line in the two places... yes... I also thought that the old kioslave could be in use, I even restarted my laptop, but still no output. But I will try your solution mentioned above...
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 10:48 -------
Hi,

after my compile odyssey I'm happy to present some results... finally.
I was not able to compile the SVN version, I don't know why but greycstoration just ignored my attempts to compile it.
So I downloaded the 0.9.4beta3 from sf.net and tried to compile this version, and IT WORKED!! Yeah!
Here is my little bugreport, hope it is somehow useful to you.

I did not build kipi-plugins and the other related libs by myself because I don't want to mess up my system too much.
I installed most of the packages from the testing repository of archlinux, only libkdcraw had to be build from SVN.
These are the libs and programs I'm using for the tests in this bugreport:
   
    exiv2-0.16
    libkexiv2-0.1.6
    kipi-plugins-0.1.5beta1
    libkipi-0.1.5
    dcraw-1.399
    libkdcraw (current SVN)
    sqlite3-3.5.8
    digikam-0.9.4beta3
    KDE-3.5.9
   
and some build tools used for compiling:
   
    autoconf-2.62
    automake-1.10.1
    gcc-4.3.0
    libtool-2.2
    m4-1.4.10
    make-3.81
    patch-2.5.4
   
Hope all important libs were mentioned above, if not just ask.

So first of all I did the sqlite analyze stuff but it did not fix the problem. I created the two dumps, but it doesn't seem to change a lot in the database structure. I don't know if it is useful but here is the diff:

=========================
before.dump -> after.dump
=========================

> INSERT INTO "Settings" VALUES('Scanned','2008-04-25T00:00:11');
> ANALYZE sqlite_master;
> INSERT INTO "sqlite_stat1" VALUES('Settings','sqlite_autoindex_Settings_1','4 1');
> INSERT INTO "sqlite_stat1" VALUES('Searches','sqlite_autoindex_Searches_1','4 1');
> INSERT INTO "sqlite_stat1" VALUES('ImageProperties','sqlite_autoindex_ImageProperties_1','5080 1 1');
> INSERT INTO "sqlite_stat1" VALUES('ImageTags','tag_index','8708 103');
> INSERT INTO "sqlite_stat1" VALUES('ImageTags','sqlite_autoindex_ImageTags_1','8708 2 1');
> INSERT INTO "sqlite_stat1" VALUES('Images','dir_index','5465 59');
> INSERT INTO "sqlite_stat1" VALUES('Images','sqlite_autoindex_Images_1','5465 2 1');
> INSERT INTO "sqlite_stat1" VALUES('TagsTree','sqlite_autoindex_TagsTree_1','207 3 1');
> INSERT INTO "sqlite_stat1" VALUES('Tags','sqlite_autoindex_Tags_1','92 1 1');
> INSERT INTO "sqlite_stat1" VALUES('Albums','sqlite_autoindex_Albums_1','111 1');
===============================================================================

After that I tried to compile the SVN version of digikam for KDE3, but failed miserably. I don't know why but it just didn't work out.

After compiling the 0.9.4beta3 source code from sf.net everything was up and I added debugging output to the digikamsearch-kioslave like Arnd mentioned before in this bugreport:

==============================================================
    kdWarning() << "SQL Query " << sqlQuery << endl;
    if (!m_db.execSql(sqlQuery, &values))
    {
        error(KIO::ERR_INTERNAL, errMsg);
        return;
    }
==============================================================

and

==============================================================
    kdWarning() << "SQL Query " << sqlQuery << endl;
    if (!m_db.execSql(sqlQuery, &values, &errMsg))
    {
        error(KIO::ERR_INTERNAL, errMsg);
        return;
    }
==============================================================

After installing digikam, no debugging infos were written to the console, even rebooting didn't do the trick, but Arnd's trick with restarting kdeinit did it, so I wrote a little startup script for running digikam with these new settings:

==============================================================
    #!/bin/bash
   
    export DIGIKAMDEST=/opt/kde
    export KDEDIRS=$DIGIKAMDEST:/usr
    export LD_LIBRARY_PATH=$DIGIKAMDEST/lib:$LD_LIBRARY_PATH
    export LDPATH=$DIGIKAMDEST/lib:$LDPATH
    export PATH=$DIGIKAMDEST/bin:$PATH
    kdeinit
    digikam
==============================================================

Now I was able to see the SQL queries that might be responsible for the slowdown. THIS query is my saved search called "top rated" that looks for images with a rating of 5 stars. It is NOT working properly anymore, I have to wait nearly 15 seconds to get a result, although I have only 6 images with a rating of 5:
       
    SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE (  (ImageProperties.value >= '5' and ImageProperties.property='Rating')  )  AND (Albums.id=Images.dirid);
       
The next query I wanted to check is the quick filter at the bottom of digikam, set to a rating of 5 stars, executed on a folder with 2000 images in it (recursively). Like I mentioned in my first post, this search is not slow. Unfortunately the quick search does not output a SQL query, so I can not give you any results.

The next step was to fire up sqlite and enter this query manually. So I pasted this query into sqlite and, as expected, it took nearly 15 seconds as well. So this must definitely be the problem!!
       
Hope somebody is still with me, this report is a little longer but I wanted to show all the steps taken to get to these results, so maybe another person might test it on a system with sqlite3-3.5.8
       
Andi
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 11:05 -------
One little note: I have another filter searching for images with the rating of 1 star. Those are images that are not completely tagged, because I was lazy and will do it later... :-)
This search also takes nearly 15 seconds to complete, although the result of this search is 768 images. So it doesn't seem to be important how many images are in the result... if this is of any use to you...
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Gilles Caulier-4
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From caulier.gilles gmail com  2008-04-26 11:06 -------
Marcel,

Take a look in Andi report about slowdown DB query. Thanks in advance

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Gilles Caulier-4
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From caulier.gilles gmail com  2008-04-26 11:09 -------
Andi,

I have just finalized the 0.9.4-beta4 tarball if you want to test with it. Look here :

http://digikam3rdparty.free.fr/0.9.x-releases/digikam-0.9.4-beta4.tar.bz2

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From arnd.baecker web de  2008-04-26 11:14 -------
Andi,

thanks a lot for your heroic effort in tracking down this bug - I think
we are really really getting closer now.
Would it be possible that you put a copy of your digikam3.db at a place
to download (or if it is not too big, i.e. below 5 MB, just mail it directly
to me)?
I would like to install sqlite 3.5.7 and 3.5.8 and compare the speed.
Most likely, we will have to pass this problem, together with your database,
on to the sqlite team...

Best, Arnd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 11:37 -------
Hi,

I was able to compile beta4... strange, SVN still is broken, even after deleting the whole folder and checking out the repo again. Anyway beta4 has this "bug", too. I will try to upload my database later.

Andi
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Bugzilla from andi.clemens@gmx.net
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From andi.clemens gmx net  2008-04-26 11:57 -------
Two more queries, maybe this is helpful.
The first query is very fast, it looks in a specific folder for images without a given tag:

SELECT Albums.url||'/'||Images.name FROM Images, Albums LEFT JOIN ImageProperties on Images.id = ImageProperties.imageid WHERE (  (Images.dirid = '39')  AND  (Images.id NOT IN    (SELECT ImageTags.imageid FROM ImageTags JOIN TagsTree on ImageTags.tagid = TagsTree.id     WHERE TagsTree.pid = '25' or ImageTags.tagid = '25' ))  )  AND (Albums.id=Images.dirid)  LIMIT 500;

The next one is incredibly slow, it took nearly 5 minutes!!!! It is a simple search looking for the family name "Saala":

SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE ( ( (Images.dirid IN   (SELECT id FROM Albums WHERE url LIKE '%Saala%'))  OR  (Images.name LIKE '%Saala%')  OR  (Images.id IN    (SELECT ImageTags.imageid FROM ImageTags JOIN TagsTree on ImageTags.tagid = TagsTree.id     WHERE TagsTree.pid = (SELECT id FROM Tags WHERE name LIKE 'Saala')     OR ImageTags.tagid = (SELECT id FROM Tags WHERE name LIKE 'Saala') ))  OR  (Images.dirid IN   (SELECT id FROM Albums WHERE caption LIKE '%Saala%'))  OR  (Images.dirid IN   (SELECT id FROM Albums WHERE collection LIKE '%Saala%'))  OR  (Images.caption LIKE '%Saala%')  OR  (ImageProperties.value LIKE '%Saala%' and ImageProperties.property='Rating') ) )  AND (Albums.id=Images.dirid);


So searching for ratings is not the only issue with the new sqlite version. Arnd, I will mail my DB to you (2.5 MB)
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         
arnd.baecker web de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
      everconfirmed|0                           |1



------- Additional Comments From arnd.baecker web de  2008-04-26 13:56 -------
Using Andi's query from #c26 and his database, I can reproduce the
problem with sqlite 3.5.8:

date
sqlite3 andi_digikam3.db
SELECT Images.id, Images.name, Images.dirid, Images.datetime, Albums.url FROM Images, Albums LEFT JOIN ImageProperties ON Images.id = Imageproperties.imageid WHERE (  (ImageProperties.value >= '5' and ImageProperties.property='Rating')  )  AND (Albums.id=Images.dirid);
.quit
date

this takes between 10-15s on my machine while with sqlite 3.3.8 it
is way below 1s.
I will try to contact the people from sqlite3.

Andi, again thanks a lot for your work ...

Arnd
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Arnd Baecker
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From arnd.baecker web de  2008-04-26 17:14 -------
After discussion on the #sqlite IRC, I filed a bug in their bug-tracker
http://www.sqlite.org/cvstrac/tktview?tn=3087

Presumably we leave this bug open, until the problem is solved
or a solution/work-around is known.

Should we put a warning somewhere, that we should not use sqlite 3.5.8.
in the meantime?
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 160966] some searches don't work properly with the new sqlite3-3.5.8

Gilles Caulier-4
In reply to this post by Bugzilla from andi.clemens@gmx.net
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
         
http://bugs.kde.org/show_bug.cgi?id=160966         




------- Additional Comments From caulier.gilles gmail com  2008-04-26 19:13 -------
Arnd,

There are few places for that :

- README
- digikam.org FAQ
- .configure log

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
1234