extragear/graphics/digikam

SVN commit 988322 by aclemens:

Removing this code again, it will not make things faster.

I did a lot of I/O performance testing now and at least with our sqlite3
solution, we really have reached the limit, it is just not performing
well on bigger collections.

My thumbsDB is 970MB big, and when the disk cache has not been already
filled, querying the database is painfully slow.
This is not digiKam's fault, it can be easily reproduced in the sqlite3
command line tool.

as root:
sync
echo 1 > /proc/sys/vm/drop_caches

Now your disk cache is cleared, as if the system just started up.
Run a simple query on a bigger thumbsDB like:

select count(*) from thumbnails where type = 0;

Even though this query delivers no results, it takes up to 20 seconds to
completely parse the table.

When used in digiKam, the application is not responding at all during
this operation, you get the impression that digiKam just hung up.

I have a large RAM here (4GB), but I don't know what happens if you run
digiKam 1.0.0 on an average system with 512MB or 1GB RAM.

For me the only solution is to go with embedded MySQL or even a real
MySQL server solution.
Because now we don't have just one, but two rather big databases.

CCMAIL:[hidden email]

 M  +0 -22     libs/database/thumbnaildb.cpp  
 M  +0 -1      libs/database/thumbnaildb.h  
 M  +0 -18     utilities/batch/batchthumbsgenerator.cpp  


--- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.cpp #988321:988322
@@ -150,28 +150,6 @@
     return filePaths;
 }
 
-QHash<QString, int> ThumbnailDB::getInvalidFilePaths()
-{
-    QSqlQuery query;
-    query = d->db->prepareQuery(QString("SELECT path, id "
-                                        "FROM FilePaths "
-                                        "   INNER JOIN Thumbnails ON FilePaths.thumbId=Thumbnails.id "
-                                        "WHERE type BETWEEN %1 AND %2;")
-                                .arg(DatabaseThumbnail::UndefinedType)
-                                .arg(DatabaseThumbnail::NoThumbnail));
-
-    if (!d->db->exec(query))
-        return QHash<QString, int>();
-
-    QHash <QString, int> filePaths;
-
-    while (query.next())
-    {
-        filePaths[query.value(0).toString()] = query.value(1).toInt();
-    }
-    return filePaths;
-}
-
 void ThumbnailDB::insertUniqueHash(const QString &uniqueHash, int fileSize, int thumbId)
 {
     d->db->execSql("REPLACE INTO UniqueHashes (uniqueHash, fileSize, thumbId) VALUES (?,?,?)",
--- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.h #988321:988322
@@ -104,7 +104,6 @@
     void replaceThumbnail(const DatabaseThumbnailInfo &info);
 
     QHash<QString, int> getValidFilePaths();
-    QHash<QString, int> getInvalidFilePaths();
 
 private:
 
--- trunk/extragear/graphics/digikam/utilities/batch/batchthumbsgenerator.cpp #988321:988322
@@ -121,26 +121,8 @@
 
 #ifdef USE_THUMBS_DB
 
-    /* Not working at the moment, it seems that the NoThumbnail type is not (yet) used, so no results
-     * are returned from getInvalidFilePaths()
-
     if (!d->rebuildAll)
     {
-        QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getInvalidFilePaths();
-
-        QStringList::iterator it = d->allPicturesPath.begin();
-        while (it != d->allPicturesPath.end())
-        {
-            if (!filePaths.contains(*it))
-                it = d->allPicturesPath.erase(it);
-            else
-                ++it;
-        }
-    }
-    */
-
-    if (!d->rebuildAll)
-    {
         QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getValidFilePaths();
 
         QStringList::iterator it = d->allPicturesPath.begin();
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

Marcel,

Perhaps we can try to mesure performance of thumbs DB using mysql Qt4
plugin ? This DB schema is simple and can be ported easily...

What do you think ?

Gilles


2009/6/27 Andi Clemens <[hidden email]>:

> SVN commit 988322 by aclemens:
>
> Removing this code again, it will not make things faster.
>
> I did a lot of I/O performance testing now and at least with our sqlite3
> solution, we really have reached the limit, it is just not performing
> well on bigger collections.
>
> My thumbsDB is 970MB big, and when the disk cache has not been already
> filled, querying the database is painfully slow.
> This is not digiKam's fault, it can be easily reproduced in the sqlite3
> command line tool.
>
> as root:
> sync
> echo 1 > /proc/sys/vm/drop_caches
>
> Now your disk cache is cleared, as if the system just started up.
> Run a simple query on a bigger thumbsDB like:
>
> select count(*) from thumbnails where type = 0;
>
> Even though this query delivers no results, it takes up to 20 seconds to
> completely parse the table.
>
> When used in digiKam, the application is not responding at all during
> this operation, you get the impression that digiKam just hung up.
>
> I have a large RAM here (4GB), but I don't know what happens if you run
> digiKam 1.0.0 on an average system with 512MB or 1GB RAM.
>
> For me the only solution is to go with embedded MySQL or even a real
> MySQL server solution.
> Because now we don't have just one, but two rather big databases.
>
> CCMAIL:[hidden email]
>
>  M  +0 -22     libs/database/thumbnaildb.cpp
>  M  +0 -1      libs/database/thumbnaildb.h
>  M  +0 -18     utilities/batch/batchthumbsgenerator.cpp
>
>
> --- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.cpp #988321:988322
> @@ -150,28 +150,6 @@
>     return filePaths;
>  }
>
> -QHash<QString, int> ThumbnailDB::getInvalidFilePaths()
> -{
> -    QSqlQuery query;
> -    query = d->db->prepareQuery(QString("SELECT path, id "
> -                                        "FROM FilePaths "
> -                                        "   INNER JOIN Thumbnails ON FilePaths.thumbId=Thumbnails.id "
> -                                        "WHERE type BETWEEN %1 AND %2;")
> -                                .arg(DatabaseThumbnail::UndefinedType)
> -                                .arg(DatabaseThumbnail::NoThumbnail));
> -
> -    if (!d->db->exec(query))
> -        return QHash<QString, int>();
> -
> -    QHash <QString, int> filePaths;
> -
> -    while (query.next())
> -    {
> -        filePaths[query.value(0).toString()] = query.value(1).toInt();
> -    }
> -    return filePaths;
> -}
> -
>  void ThumbnailDB::insertUniqueHash(const QString &uniqueHash, int fileSize, int thumbId)
>  {
>     d->db->execSql("REPLACE INTO UniqueHashes (uniqueHash, fileSize, thumbId) VALUES (?,?,?)",
> --- trunk/extragear/graphics/digikam/libs/database/thumbnaildb.h #988321:988322
> @@ -104,7 +104,6 @@
>     void replaceThumbnail(const DatabaseThumbnailInfo &info);
>
>     QHash<QString, int> getValidFilePaths();
> -    QHash<QString, int> getInvalidFilePaths();
>
>  private:
>
> --- trunk/extragear/graphics/digikam/utilities/batch/batchthumbsgenerator.cpp #988321:988322
> @@ -121,26 +121,8 @@
>
>  #ifdef USE_THUMBS_DB
>
> -    /* Not working at the moment, it seems that the NoThumbnail type is not (yet) used, so no results
> -     * are returned from getInvalidFilePaths()
> -
>     if (!d->rebuildAll)
>     {
> -        QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getInvalidFilePaths();
> -
> -        QStringList::iterator it = d->allPicturesPath.begin();
> -        while (it != d->allPicturesPath.end())
> -        {
> -            if (!filePaths.contains(*it))
> -                it = d->allPicturesPath.erase(it);
> -            else
> -                ++it;
> -        }
> -    }
> -    */
> -
> -    if (!d->rebuildAll)
> -    {
>         QHash<QString, int> filePaths = ThumbnailDatabaseAccess().db()->getValidFilePaths();
>
>         QStringList::iterator it = d->allPicturesPath.begin();
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel
>

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> Marcel,
>
> Perhaps we can try to mesure performance of thumbs DB using mysql Qt4
> plugin ? This DB schema is simple and can be ported easily...
>
> What do you think ?

I have know experience with MySQL (embedded), dont know if there is a single
file or a number of files etc and how this is controlled. We can test this of
course.

Generally, the dependency situation could be worse:
Amarok pulls in MySQL embedded as a dependency. I am not sure if there is a
Qt4 plugin linked to the embedded variant, but if not it should be a CMake
task to create one based on Qt code.
Akonadi will probably depend on full MySQL, at least it does currently. This
means KMail etc. will have MySQL as a dependency for approx. KDE 4.5.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

>  053 ==> 197641 : Can't delete exif x-default caption once set.
>  054 ==> 196437 : digiKam crashes at startup.
>  056 ==> 197868 : digiKam crashes on startup

055 was lost somewhere :)

m.

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

Fixed :D
Thanks!

Andi

On Monday 29 June 2009 23:16:40 Mikolaj Machowski wrote:

> >  053 ==> 197641 : Can't delete exif x-default caption once set.
> >  054 ==> 196437 : digiKam crashes at startup.
> >  056 ==> 197868 : digiKam crashes on startup
>
> 055 was lost somewhere :)
>
> m.
>
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

2009/6/28 Marcel Wiesweg <[hidden email]>:

>> Marcel,
>>
>> Perhaps we can try to mesure performance of thumbs DB using mysql Qt4
>> plugin ? This DB schema is simple and can be ported easily...
>>
>> What do you think ?
>
> I have know experience with MySQL (embedded), dont know if there is a single
> file or a number of files etc and how this is controlled. We can test this of
> course.
>
> Generally, the dependency situation could be worse:
> Amarok pulls in MySQL embedded as a dependency. I am not sure if there is a
> Qt4 plugin linked to the embedded variant, but if not it should be a CMake
> task to create one based on Qt code.

I cannot found MYSql embeded code in Amarok. only this code :

http://websvn.kde.org/trunk/extragear/multimedia/amarok/src/collection/mysqlecollection/MySqlEmbeddedCollection.cpp?revision=984916&view=markup

Look the commit comment.

Look also that #include <mysql.h> is used. sound like to use mysql
installed on the system, not Qt4 SQL plugin.

Gilles
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

But they use MySQL embedded:
http://amarok.kde.org/wiki/Development/MySQL_Embedded
:)

Embedded doesn't mean they are shipping it, it just means no server needs to
be started.

Andi

On Thursday 02 July 2009 08:56:22 Gilles Caulier wrote:

> 2009/6/28 Marcel Wiesweg <[hidden email]>:
> >> Marcel,
> >>
> >> Perhaps we can try to mesure performance of thumbs DB using mysql Qt4
> >> plugin ? This DB schema is simple and can be ported easily...
> >>
> >> What do you think ?
> >
> > I have know experience with MySQL (embedded), dont know if there is a
> > single file or a number of files etc and how this is controlled. We can
> > test this of course.
> >
> > Generally, the dependency situation could be worse:
> > Amarok pulls in MySQL embedded as a dependency. I am not sure if there is
> > a Qt4 plugin linked to the embedded variant, but if not it should be a
> > CMake task to create one based on Qt code.
>
> I cannot found MYSql embeded code in Amarok. only this code :
>
> http://websvn.kde.org/trunk/extragear/multimedia/amarok/src/collection/mysq
>lecollection/MySqlEmbeddedCollection.cpp?revision=984916&view=markup
>
> Look the commit comment.
>
> Look also that #include <mysql.h> is used. sound like to use mysql
> installed on the system, not Qt4 SQL plugin.
>
> Gilles
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> I cannot found MYSql embeded code in Amarok. only this code :
>
> http://websvn.kde.org/trunk/extragear/multimedia/amarok/src/collection/mysq
>lecollection/MySqlEmbeddedCollection.cpp?revision=984916&view=markup

As Andi already said, it relies on distributions shipping the database.
The full name is "MySQL Embedded Server"; the point is that you do not need a
full MySQL server process running, including all administrative stuff like
user access rights, but can run this from inside your application.

Please note that we can also take a look at Akonadi, who are running a
dedicated MySQL server process, but also probably without extra
administration.

>
> Look the commit comment.
>
> Look also that #include <mysql.h> is used. sound like to use mysql
> installed on the system, not Qt4 SQL plugin.

Yes, I think there is no Qt4 MySQL plugin that links to libmysqld.so, but only
a plugin that links to libmysqlclient.so.
Judging from the docs:

http://doc.trolltech.com/4.5/sql-driver.html#embedded-mysql-server

it means we have to take the source of the Qt MySQL plugin, compile it
ourselves just with the difference of linking to the embedded server libs
instead of the client libs.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel