Metadata

[Remco Viëtor]

On Sunday 30 June 2013 15:58:29 Anders Lund wrote:
...
>
> I don't know if "delayed" is any good.
>
> In my workflow, I typically import, select, tag, process and export
images.
> When exporting, having those metadata writte to the image is smart.
>
> Writing meta data to images, I assume, is a question of copying data from
the
> digikam db into the image or sidecar file. So everytime an images data
> changes, it should go into the queue, unless it is allready there. Now if
the
> actual data writing is done by a spearate process, what is the problem?
That
> process must then be throtteled, so that it does not take the system
down.
>  
> When exporting, it could be checked if the image is in the metadata
writing
> queue, so it can be processed there prior to exporting.
>
>
The problem can occur if you start a background job, and then, while it is
running, do something like:
- add or remove tags from a subset of the images in the queue (starting a
second background job)
- or process an image that hasn't had its tags written yet.
In both cases there are scenarios where the tags stored in the image can
end up being different from the tags as stored in the database.

There shouldn't be any need to throttle the background queue, that's why
you have task switching in any modern operating system. The queue might get
a lower priority ('nice' value under UNIX-likes).


_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Anders Lund]

On Søndag den 30. juni 2013 16:56:56, Remco Viëtor wrote:

> On Sunday 30 June 2013 15:58:29 Anders Lund wrote:
> ...
>
> > I don't know if "delayed" is any good.
> >
> > In my workflow, I typically import, select, tag, process and export
>
> images.
>
> > When exporting, having those metadata writte to the image is smart.
> >
> > Writing meta data to images, I assume, is a question of copying data from
>
> the
>
> > digikam db into the image or sidecar file. So everytime an images data
> > changes, it should go into the queue, unless it is allready there. Now if
>
> the
>
> > actual data writing is done by a spearate process, what is the problem?
>
> That
>
> > process must then be throtteled, so that it does not take the system
>
> down.
>
> > When exporting, it could be checked if the image is in the metadata
>
> writing
>
> > queue, so it can be processed there prior to exporting.
>
> The problem can occur if you start a background job, and then, while it is
> running, do something like:
> - add or remove tags from a subset of the images in the queue (starting a
> second background job)
> - or process an image that hasn't had its tags written yet.
> In both cases there are scenarios where the tags stored in the image can
> end up being different from the tags as stored in the database.

Hm, I figured it would be possible to know the state.

> There shouldn't be any need to throttle the background queue, that's why
> you have task switching in any modern operating system. The queue might get
> a lower priority ('nice' value under UNIX-likes).

Well, Depending on memory usage maybe, but I do know that akonadis nepomuk
feeder was able to make my and others PC unavailable to the owners.

--
Anders
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Boudewijn]

--- On Sun, 30/6/13, Elle Stone <[hidden email]> wrote:

> On 6/29/13, Boudewijn <[hidden email]>
> wrote:
> > What I had in mind is:
> > - write metadata to database only (leave the files as
> they are)
> > - when I'm done, tick "write meta data to file"
> > - chose the option to sync the database
> > - walk away from my computer
> I'm not sure exactly what Ander's goals are, but you just
> described
> exactly what I do when I'm tagging. I go to the
> settings/configure/metadata panel and *un*check write to the
> image, or
> sidecar in my case. Then I tag. The tags are only written to
> the
> database. When I'm done tagging I re-enable writing to the
> sidecar
> files (the image files are all read-only) and sync the
> sidecars with
> the database.

Thanks a lot! I will be doing just the same from now on, after reading the rest of the thread ;-)
(I actually use the camera-generated jpg's as sidecar file for the raw files, so that I still have a visual cue if the name of the file gets jumbled)

Best regards,

Boudewijn
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Marcel Wiesweg]

>
> What I had in mind is:
> - write metadata to database only (leave the files as they are)
> - when I'm done, tick "write meta data to file"
> - chose the option to sync the database
> - walk away from my computer

I plan to implement this one day. As discussed in this thread, it indeed
requires a more robust storage than "keep it in memory" and problems like
conflicts and subsequent edits must be dealt with, so it's not very easy to
do.

Apart from that, there's the very valid question why such edit processes bring
digikam performance down, with all the work exclusively being done in threads.
One answer lies in the fact that everything is database-based and while Sqlite
is very fast, writing anything to disk makes it do a hardware fsync and this
is very very very slow and is per default necessary quite often. And it blocks
all threads, including the main thread. For batch procedures the most
important optimization that we have is to group writing in transactions to
reduce the amount of fsyncs.
Possible fixes for the generaly problem:
1) Enabling SQlite's WAL mode and reducing the "synchronous" level of SQLite.
It did not yet dare the latter; maybe it's the key for now.
2) Accept the SQLite is not made for the heavily concurrent use that we make
of our db and go for a db system that runs as a daemon and solves the
synchronicity problem for us. I never tried MySQL but saw many problems with
our backend, which lowered my motivation.

Marcel
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users