Digikam + Kipi + Export and/or Sync to web services (and other such stuff)

[Colin Guthrie-6]

Hi,

First of all, I'm sorry this comes at such short notice. I had intended
to get this out over two weeks ago but I've just been swamped at work.

As some of you know I've had an interest in the above topic for some time.

I started off a while ago maintaining the Gallery Export plugin in Kipi
plugins and subsequently proposed (and started working on) a "sync"
plugin that was meant to unifiy the various Export plugins currently
available in kipi-plugins (Flickr export, Picassa export, Gallery export
etc.).

I abandoned the sync plugin quite a while back as I was not confident of
the architecture I had chosen which was separated away from the rest of
the desktop system and somewhat isolated to kipi which just felt
"wrong". I felt that there had to be a more appropriate way of
interacting with these services that would ultimately be more useful in
the rest of KDE and the desktop.

I considered OpenSync and have noted how systems such as FSpot permit
sync to Facebook via "conduit" which is built half on top of opensync
with some python bits chucked in. While I like the opensync approach, it
was just not in a state to be used when I looked and things are not much
better now! (/me tries to pretend he has not seen the 0.5 release)

In 2007 I attended Akademy in Glasgow. I sat in a talk on Akonadi and
although the talk was aimed at PIM data, I immediately thought that this
could be a very interesting system used for both Amarok and Digikam. Not
all that long ago, I asked the main developer of the "services"
framework of Amaork2 about whether he'd looked or was interested in
Akonadi, but due to the progress already made, he is currently not
interested in re-engineering it's core to build on top of it
(unsurprisingly!). While the concept has been mooted on the the
Digikam/Imaging MLs in the past, I don't think this is particularly
attractive an option right now to use Akonadi at the core of digikam
itself. This is all fair enough and I can't doubt the logic behind these
current decisions and I don't disagree that Akonadi may be overkill for
accessing a local collection of images. However, I firmly believe that
the constructs provided by Akonadi are ideally suited to interacting
with remote systems.

I'd recommend taking a brief wonder around the Akonadi site by way of a
teaser: http://pim.kde.org/akonadi/

As a brief overview, the Akonadi is built on top of a MySQL server and
runs as a KDE service. It is essentially a "framework" that represents
"Collections" and "Items" which maps pretty cleanly to Albums and Images
in our use case. Akonadi can be able to cache the remote Collections
meaning that interaction is both quick and responsive. Different
Collections can have different cache policies attached to them (so for
example a "View by Tag" collection could be cached less aggressively
than a "Album" collection). For each "Item" there are three levels of
granularity that can be affected the caching: Headers, Body and
Attachments. These are generic and thus can be manipulated for our
needs. In our case I'd say we'd map this to Metadata, Thumbnail and Full
Image. Our cache policy will probably be to store only Metadata and
Thumbnail initially but as the uses for this system expand over time
this may change (e.g. a screen saver that shows Facebook pictures!!).
Due to the way that Akonadi works, if part of the data for an item is
missing (e.g. the Thumbnail or the Full image) it will be fetched from
the remote system transparently.

This is just a very, very, very brief and hand-wavey description.

So, my (very rough) proposal is approximately as follows:
  0. Discuss the needs of the export and sync systems and decide whether
Akonadi is a good basis, assuming "yes" proceed.
  1. Read and absorb:
http://api.kde.org/4.x-api/kdepimlibs-apidocs/akonadi/html/index.html
  2. Take a look over the various web systems with regards to how they
represent their Collections and Metadata. Map this to a standard
representation and come up with a set of classes (inspired by Kipi
representation currently perhaps?)
  3. Write an Akonadi serializer plugin so that our representation can
be stored in the cache layer.
http://api.kde.org/4.x-api/kdepimlibs-apidocs/akonadi/html/classAkonadi_1_1ItemSerializerPlugin.html
  4. Write akonadi "Resources" for each service that implement the
individual APIs for the given service and map it to our structure. See
http://api.kde.org/4.x-api/kdepimlibs-apidocs/akonadi/html/classAkonadi_1_1ResourceBase.html
  5. Create a unified "export" plugin in kipi-plugins which will be an
Akonadi client.

The above is probably somewhat wrong as I don't yet know overly much
about Akonadi, but I feel this weekend will change that!

The bit I am very unsure about just now is "account" management. I'm not
sure how you use Akonadi from the perspective of different authenticated
"accounts". In the PIM context this would be different IMAP accounts,
but in our context it would be your flickr API key or your facebook
login etc. Till (for the befit of others, Till is one of the main
Akonadi developers, also CC'ed) if you have any info on this please feel
free to add it in :)


So what about Sync I hear you cry? Well, I think that ultimately the
correct horse to back here is OpenSync and that there will eventually be
a proper Sync solution that involves a standard synchronisation
application in KDE (think KitchenSync of old). When this happens,
integration with Akonadi will certainly be forth coming and
synchronising your collections and items will be come possible. When
this happens, I'd propose we create a layer that exposes the Digikam (or
rather Kipi Host) data via an Akonadi Resource and with this it will be
possible to "Sync" things via the standard system.

Overall I think using this framework will allow us to create a genuinely
useful set of interfaces that can be used elsewhere in the desktop and
also provide a solid foundation for utilising data.

There is definitely (IMO) scope for rebasing the whole KIPI API on top
of Akonadi but this is perhaps something that is best left for a future
occasion. Perhaps not, as I know the API breakage is up for discussion,
but I don't want to bite of more than can be comfortably chewed at the
moment!

By way of fortunate coincidence, the Akonadi team also have a sprint
planned for this weekend. So if we do decide to take this approach, all
the main people will be together and available at the same time as
ourselves. While I'm sure they have their own plans to get on with, I'm
sure they wont mind taking a little time out to help advise us on this
plan should we need any assistance!

I have included below my IRC conversation with Till. There is still a
huge amount I don't know about Akonadi, I just feel that it's an
appropriate framework to use. I hope other people feel the same :)

See you all on Thursday/Friday :)

Col




Oct 20 21:13:48 * Now talking on #akonadi
Oct 20 21:13:48 * Topic for #akonadi is: Akonadi 1.0.0 released:
http://download.akonadi-project.org/ | general info:
http://www.akonadi-project.org/ -
http://techbase.kde.org/Projects/PIM/Akonadi | FAQ:
http://techbase.kde.org/Projects/PIM/Akonadi#Akonadi_FAQ (includes "Why
are you using MySQL instead of $DB?" and "Do I need a MySQL server?")
Oct 20 21:13:48 * Topic for #akonadi set by vkrause at Wed Aug 13
15:55:47 2008
Oct 20 21:13:55 <coling> Hello
Oct 20 21:14:13 * bbroeksema has quit ("Lost terminal")
Oct 20 21:14:22 <coling> till, are you around just now? I would like to
ask a question and while I could mail you it would be nice to chat if
you've got a couple mins?
Oct 20 21:14:35 <till> Sure.
Oct 20 21:14:40 <coling> Great :)
Oct 20 21:14:53 <coling> I saw your talk on Akondi a few years back at
Akademy in Glasgow
Oct 20 21:15:02 * bbroeksema
(n=[hidden email]) has joined #akonadi
Oct 20 21:15:23 <coling> I asked about where you saw the scope *ending*
for akonadi as it seemed like a good idea and I mentioned perhaps using
it for e.g. Amarok or Digikam etc.
Oct 20 21:15:53 <coling> Well, I do some coding for digikam (or rather
kipi-plugins) and we're going to be having a coding sprint soon in Genoa
(two weeks time)
Oct 20 21:16:27 <coling> One of the things we want to do is establish a
framework for syncronising images from digikam to web-based services
like facebook, flicker, gallery2 etc.
Oct 20 21:17:16 <coling> We already have a collection of independant
plugins that do this job for individial services (e.g. there are
"flicker export" and "gallery export" etc. plugins.
Oct 20 21:17:39 <coling> But the problem is the UI is different each
time and the "account managment" is reimplemented each time too.
Oct 20 21:17:47 <till> Yep.
Oct 20 21:18:09 <coling> What I want to do is create a unified system
for "syncing" (not just export although it could be used for that) with
these external systems.
Oct 20 21:18:45 <coling> Where adding a new system is fairly trivial, so
adding in support for e.g. Picassaweb or whatever would be pretty quick
and painless.
Oct 20 21:19:27 <coling> So I've always had akonadi at the back of my
mind as a framework for doing this.
Oct 20 21:19:57 <coling> I've also considered trying to incorporate
opensync somewhere into the mix but as it seems in permanent flux I'm
not overly confident!
Oct 20 21:20:07 <till> Hm, sounds a bit like opensync is more what you
are looking for, to be honest.
Oct 20 21:21:05 <coling> Well that's kinda what I was wanting to quiz
you about as I see the latest paper on the akondi website mentioned
opensync in the title, but I didn't really gather much from the content
in relation to opensync and how it will work with Akonadi.
Oct 20 21:21:47 <till> Akonadi is good at caching stuff, providing
local, fast access to it, keeping track of local changes, etc.
Oct 20 21:22:20 <till> it has providers/agents/resources to get stuff
from/to other formats or endpoints.
Oct 20 21:22:26 <till> So in that case it does some of what you want.
Oct 20 21:22:53 <till> But then, you already have a local store, you
don't really need change management, Iguess, nor fine grained caching.
Oct 20 21:23:15 <till> opensync is about keeping things/files in sync
between several endpoints.
Oct 20 21:23:29 <till> where endpoitns can be local apps, mobile devices
or webservices.
Oct 20 21:23:49 <till> In that sense Akonadi would be one of many
opensync endpoints.
Oct 20 21:24:34 <till> There's some overlap, between the two.
Oct 20 21:24:55 <coling> Indeed I am beginning to see that :)
Oct 20 21:24:58 <till> Whether you want akonadi or not kind of depends
on whether the local store/cache features make sense to you.
Oct 20 21:25:15 <coling> I certainly like the idea of the local store
and cache features of your remote accounts.
Oct 20 21:25:46 <coling> Although I'm not sure if it's a full blown
absolute must.
Oct 20 21:26:55 <till> do people want to have several different backends
with different images, or keep themall in sync?
Oct 20 21:27:21 <coling> well the idea would be that you could define
several different backends, e.g. your facebook and your flicker etc.
Oct 20 21:27:32 <coling> And then do one of two things:
Oct 20 21:27:57 <coling> 1) Manually "Publish" on a per-photo (or
per-album) basis as a "one-time" operation.
Oct 20 21:28:26 <coling> 2) Define some "sync" settings e.g. publish
this saved search to facebook as the album "foo"
Oct 20 21:29:08 <coling> (perhaps not the saved search bit - that's
probably over complicating thiings for now! but sync this album with
facebook)
Oct 20 21:29:51 <coling> What would be good tho' is the ability to pull
in comments or tags left by users on these remote systems so you can use
the metadata locally.
Oct 20 21:32:34 <till> Hm, I think Akonadi can help quite a bit with that.
Oct 20 21:32:36 <coling> So I guess what (for me) would be nice is if I
could define a standard structure for working with this kind of data.
Oct 20 21:32:59 <coling> You say Akonadi has providers, agents and resources
Oct 20 21:33:08 <till> Yep.
Oct 20 21:33:17 <coling> So would I be correct in saying that the
resources would be the "images"
Oct 20 21:33:40 <coling> and the build in support for collections would
represent albums or tags etc.
Oct 20 21:34:09 <coling> A provider would be a sort of bridging layer
that joins e.g. facebook to this standard system?
Oct 20 21:34:20 <coling> And another provider would bridge flickr to it?
Oct 20 21:34:29 <coling> And then the agents would be what??
Oct 20 21:34:39 <coling> (or is this all waaaay off base? :))
Oct 20 21:34:39 <till> No, resources are transports, thigns that feed
data into the store, and write it back to backends.
Oct 20 21:34:54 <till> Think kresources, of old.
Oct 20 21:35:22 <till> agents are processes that operate on the store to
do things like tagging, manipulation, data mining.
Oct 20 21:36:01 <till> providers don't exist, I just used that term as a
analogous one, in case you don't recognize the others. :)
Oct 20 21:36:09 <till> provides == resources
Oct 20 21:36:13 <coling> Ahh OK,  So we'd write a resource for each
separate system we want to use? a Facebook resource and a Flickr
resource etc.
Oct 20 21:36:15 <coling> :p
Oct 20 21:36:19 <till> Yep.
Oct 20 21:36:40 <coling> And agents would not really be mandatory.
Oct 20 21:36:41 <till> Each a small process, for robustness reasons.
Oct 20 21:37:08 <till> No, only if you have things that work with the
data in the background, o extract info from it, annotate it, etc.
Oct 20 21:37:22 <till> Your app can do taht too, it would be an agent,
in that sense.
Oct 20 21:37:31 <till> Nomenclature is like this:
Oct 20 21:37:40 <till> clients: all things that read/write the store
Oct 20 21:37:49 <coling> Cool, I think I've kinda getting a vague handle
on it thus far!
Oct 20 21:37:53 <till> agents: non-user facing clients, background
tasks, etc
Oct 20 21:38:10 <till> resources: special feeder agents that conect to
backends
Oct 20 21:38:22 <coling> OK, I think I'm seeing this working quite
nicely thus far.
Oct 20 21:38:39 <coling> It certainly "fits" model wise IMO.
Oct 20 21:38:55 <till> Yes.
Oct 20 21:39:12 <till> The nice thing is that the API is fully typed.
Oct 20 21:39:36 <till> YOu add support for yor mimetype and the C++ type
for it, and then you can use the API natively with that.
Oct 20 21:39:39 <till> All templatized.
Oct 20 21:39:57 <coling> Right so I'd just have to define what I want an
"image" to represent?
Oct 20 21:40:08 <till> Yes.
Oct 20 21:40:27 <till> And write a plugin that can go from/to
QByteArray, for serialization into the db tables.
Oct 20 21:40:44 <till> Mostly code for that is already somewhere.
Oct 20 21:40:51 <till> For images it's easy, Qt already has that code.
Oct 20 21:41:52 <coling> cool. And would you recommend storing the full
images or perhaps just a thumb?
Oct 20 21:42:24 <till> depends on the use acses
Oct 20 21:42:30 <coling> While storing the full image would be nice,
should we really be forcing that much data into the db?
Oct 20 21:42:40 <till> You can have cache policies, we added that for
online/offline IMAP
Oct 20 21:42:52 <till> The user can decide, per folder or per resource.
Oct 20 21:43:06 <coling> Ahh sweet.
Oct 20 21:43:13 <till> Sometimes they want to keep just headers,
sometimes full mails, sometimes attachments, sometimes not.
Oct 20 21:43:22 <coling> So a different policy could be added for e.g.
your own image vs. those of your friends etc.
Oct 20 21:43:25 <till> On your phone, you want things lightweight, on
the desktop, everythign.
Oct 20 21:43:29 <till> Right.
Oct 20 21:43:40 <till> What is not there locally is fetched by the
resource online.
Oct 20 21:43:45 <till> Transparently.
Oct 20 21:43:50 <coling> \o/
Oct 20 21:43:53 <coling> Perfect.
Oct 20 21:44:21 <coling> OK so the only thing left really is to worry
about the sync'ing aspect of it.
Oct 20 21:44:36 <till> The granularity we have is "headers" "body"
attachments"
Oct 20 21:44:53 <till> YOu can map that how you like, maybe to name,
thumbnail, full image
Oct 20 21:45:17 <till> For syncing, yes, that goes into the realm of
opensync, potentially
Oct 20 21:45:24 <coling> Yeah, comments, tags etc. could be headers I
presume?
Oct 20 21:45:33 <till> Right.
Oct 20 21:45:36 <coling> Sounds ideal.
Oct 20 21:45:40 <coling> I think from the import/publication part of it,
akonadi sounds perfect.
Oct 20 21:45:54 <coling> And the syncing part of it perhaps it can go to
Opensync
Oct 20 21:45:58 <till> Could well be.
Oct 20 21:46:03 <coling> What may be better is to do this:
Oct 20 21:46:17 <coling> 1 Define the resources to work on facebook etc.
etc.
Oct 20 21:46:30 <coling> 2 Define another resource that can connect to
digikam
Oct 20 21:46:44 <coling> 3 Glue together these end points in opensync
Oct 20 21:46:55 <coling> Would that be possible?
Oct 20 21:46:59 <till> Yes, for example.
Oct 20 21:47:55 <coling> So the actual sync part of it is not part of
the actual stuff we do but we get it for "free" if the templated
serialization can be exposed to opensync correctly.
Oct 20 21:49:43 <till> In theory, yes.
Oct 20 21:49:53 <till> OpenSync as a whole is quite in flux, atm.
Oct 20 21:49:58 <till> No pun intended.
Oct 20 21:50:17 <coling> Yeah that's what puts me off using opensync
directly atm too.
Oct 20 21:51:19 <coling> I've tried to follow the dev for the last 6
months or so as a lurker but I haven't got a great feel for the current
state of affairs and from various discussions with colleagues who fiddle
with it, I don't think I can realistically develop specifically for it
right now.
Oct 20 21:51:32 * bbroeksema has quit ("Lost terminal")
Oct 20 21:52:12 <coling> But I think doing this work under the structure
of akonadi, that when opensync support comes up to speed, it should be
relatively easy to expand things to support it.
Oct 20 21:52:31 <coling> At least that would be my hope!
Oct 20 21:53:08 <till> yes, ours as well
Oct 20 21:53:14 <coling> :)
Oct 20 21:53:20 <till> we're still hoping our phones will magically
start working ;)
Oct 20 21:53:28 <coling> lol
Oct 20 21:53:30 <coling> So, just to summarise:
Oct 20 21:54:10 <coling> Do you think this is a fairly legitimate use of
akonadi? Or do you think I'm trying to squeeze a square peg into a round
hole by wanting to use it in this scenario?
Oct 20 21:56:07 <till> No, it seems to fit nicely.
Oct 20 21:56:10 <till> Worth a try.
Oct 20 22:03:04 <coling> Cool. OK, thanks for the advice. I'll try and
draft up a proposal so that we can start work on it. I'll be posting it
to kde-imaging@ ML and if you don't mind I'll CC you in. You don't have
to reply but if I say anything too stupid/incorrect perhaps you'd be
kind enough to gently point that out ;) I think the list is OK to post
to without subscription so no hassle there, but I'm happy to act as
go-between if needs be anywa
Oct 20 22:03:05 <coling> y :)
Oct 20 22:03:38 <coling> Thanks again for your advice.
Oct 20 22:05:48 <till> yw

--

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
   Mandriva Linux Contributor [http://www.mandriva.com/]
   PulseAudio Hacker [http://www.pulseaudio.org/]
   Trac Hacker [http://trac.edgewall.org/]

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel