[Bug 216895] New: Improve performance and reduce space with some sqlite3 pragma's

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 216895] New: Improve performance and reduce space with some sqlite3 pragma's

Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

           Summary: Improve performance and reduce space with some sqlite3
                    pragma's
           Product: digikam
           Version: unspecified
          Platform: Debian testing
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: NOR
         Component: general
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]


Version:           1.0.0beta6 (using KDE 4.3.2)
OS:                Linux
Installed from:    Debian testing/unstable Packages

You have an awesome program that I enjoy using. Just a couple of things you can
do to make it better. The default setup of sqlite uses a 1kb page size. This
can lead to a lot of wasted space in the large tuples that digikam uses.
Increasing the pagesize to 4kb reduces the total database size by a third. I
used 4kb because it matches the normal intel page size and larger page sizes
have marginal benefit (<1% space savings). I also suspect this page size comes
with some performance benefits.

For existing databases (and recent versions of sqlite3):
   'pragma page_size = 4096 ; vacuum;'

When creating a new database:
   'pragma page_size = 4096;' before the first table.

Also, in order to run in limited environments, sqlite only lets itself use a
default of at most of 2000 pages worth of data for caching or internal sorts. I
propose increasing this to 8k-16k pages (32-64mb). Modern machines have a lot
of RAM. I've not benchmarked if there is any speed improvement, but I cannot
see how giving sqlite a few more megabytes of memory can hurt.

Can be run on any existing or new database:
   'pragma default_cache_size=16000;'

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 216895] Improve performance and reduce space with some sqlite3 pragma's

Gilles Caulier-4
https://bugs.kde.org/show_bug.cgi?id=216895


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|general                     |Database
            Version|unspecified                 |1.0.0-beta6




--- Comment #1 from Gilles Caulier <caulier gilles gmail com>  2009-12-01 10:01:14 ---
Note : digiKam is under porting to MySQL...

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 216895] Improve performance and reduce space with some sqlite3 pragma's

Gilles Caulier-4
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|1.0.0-beta6                 |1.0.0




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 216895] Improve performance and reduce space with some sqlite3 pragma's

Gilles Caulier-4
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




--- Comment #2 from Gilles Caulier <caulier gilles gmail com>  2011-12-15 09:01:44 ---
Scott,

What's about this file using digiKam 2.x serie ?

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[Bug 216895] SQLITE : improve performance and reduce space with some pragma's

Gilles Caulier-4
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895


Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Improve performance and     |SQLITE : improve
                   |reduce space with some      |performance and reduce
                   |sqlite3 pragma's            |space with some pragma's




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 216895] SQLITE : improve performance and reduce space with some pragma's

Gilles Caulier-4
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

Gilles Caulier <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|Database                    |Database-Sqlite

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 216895] SQLITE : improve performance and reduce space with some pragma's

bugzilla_noreply
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #3 from [hidden email] ---
Swati,

Did you take a look about this file ?

Gilles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 216895] SQLITE : improve performance and reduce space with some pragma's

bugzilla_noreply
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

--- Comment #4 from [hidden email] ---
(In reply to caulier.gilles from comment #3)
> Swati,
>
> Did you take a look about this file ?

Yes. Seems like a good option.
The page and the cache size could be modified for SQLite.

But is there a file already, that contains statement specifically for SQLite?
Should the statement be added in XML schema file, under "Create DB" ?

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 216895] SQLITE : improve performance and reduce space with some pragma's

bugzilla_noreply
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

--- Comment #5 from [hidden email] ---
No specific file exist for sqlite settings.

XML file must be patch. create DB sound like the best pace for that. revision
must be increased of course.

Giles Caulier

--
You are receiving this mail because:
You are the assignee for the bug.
Reply | Threaded
Open this post in threaded view
|

[digikam] [Bug 216895] SQLITE : improve performance and reduce space with some pragma's

bugzilla_noreply
In reply to this post by Scott A Crosby
https://bugs.kde.org/show_bug.cgi?id=216895

Mario Frank <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Version Fixed In|                            |5.5.0
      Latest Commit|                            |https://commits.kde.org/dig
                   |                            |ikam/2f8ddd42ef62d7aea9e490
                   |                            |cdb05ffcc644810c81
         Resolution|---                         |FIXED
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #6 from Mario Frank <[hidden email]> ---
Git commit 2f8ddd42ef62d7aea9e490cdb05ffcc644810c81 by Mario Frank.
Committed on 22/02/2017 at 15:05.
Pushed by mfrank into branch 'master'.

Merged the current state of the garbage collection branch which improves the
database cleanup stage of the maintenance
and improves the reactiveness of the maintenance overall. We ported the way
items are processed to a queue based method
that can use the CPUs more effectively and does not create thousands of
threads.
Related: bug 283062, bug 374225, bug 351658, bug 362023, bug 329353
FIXED-IN: 5.5.0

M  +17   -12   NEWS

https://commits.kde.org/digikam/2f8ddd42ef62d7aea9e490cdb05ffcc644810c81

--
You are receiving this mail because:
You are the assignee for the bug.