Off-line Collection Browsing & DB cleanup

[nianderson]

Ive been using digikam for a while now. In general as a library
management tool it works well for me. There are two functions that I
don't understand.

Collections:
(I recently wiped out my digikam db and started fresh and have not
re-connected my remote collections yet so perhaps this has changed.)

The notion of remote or removable collections. Both the thumbnails and
the metadata are stored in the database correct? Whenever I launch
digikam and my network collections are not available I cant use digikam
to browse the disconnected collections. So in order to find an image I
have to have that collection connected? Since all the information is
available shouldn't I be able to search and browse the collection, but
be given some message about attaching the collection before being able
to view a full size image?

DB:
The reason I wiped out my db is because I rename and sort images with
some custom scripts. At one point some images could no longer be found
with digikam but they were inside the collections, and I did some db
spelunking and saw they were there too, I also noticed references to
files that I had moved outside of digikam. How do I "cleanup" that
database cruft? Is that what "rebuild fingerprints" is for? It wasnt
clear in the documentation what that function actually did.

I would appreciate any clarification and insight anyone can provide.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Peter Albrecht]

Hello Nick,

maybe this post at "Scribbles and Snaps" may help you clean
up your database (if it is broken again):

> http://scribblesandsnaps.wordpress.com/2011/12/19/check-and-optimize-digikams-databases/

As far as I understood, the fingerprints (referenced by
"rebuild fingerprints") are used for

"Browse" -> "Fuzzy Searches":
 - find similar images
 - find by sketch
 - find duplicates

I have not build fingerprints for all my images and digiKam
always asks me to do so, when I click on "Fuzzy Searches"
(to explore them).

Regards,
        Peter

On 10.05.2012 16:59, Nick Anderson wrote:

> Ive been using digikam for a while now. In general as a
> library management tool it works well for me. There are two
> functions that I don't understand.
>
> Collections:
> (I recently wiped out my digikam db and started fresh and
> have not re-connected my remote collections yet so perhaps
> this has changed.)
>
> The notion of remote or removable collections. Both the
> thumbnails and the metadata are stored in the database
> correct? Whenever I launch digikam and my network
> collections are not available I cant use digikam to browse
> the disconnected collections. So in order to find an image I
> have to have that collection connected? Since all the
> information is available shouldn't I be able to search and
> browse the collection, but be given some message about
> attaching the collection before being able to view a full
> size image?
>
> DB:
> The reason I wiped out my db is because I rename and sort
> images with some custom scripts. At one point some images
> could no longer be found with digikam but they were inside
> the collections, and I did some db spelunking and saw they
> were there too, I also noticed references to files that I
> had moved outside of digikam. How do I "cleanup" that
> database cruft? Is that what "rebuild fingerprints" is for?
> It wasnt clear in the documentation what that function
> actually did.
>
> I would appreciate any clarification and insight anyone can
> provide.
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users
>

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[nianderson]

On 05/10/2012 12:48 PM, Peter Albrecht wrote:
> Hello Nick,
>
> maybe this post at "Scribbles and Snaps" may help you clean
> up your database (if it is broken again):
>
>> http://scribblesandsnaps.wordpress.com/2011/12/19/check-and-optimize-digikams-databases/

Thanks, I was using a mysql backend running on my local machine when i
discovered the cruft mentioned but t was never completely broken. I
think the two issues with the database were separate.

First some images would not display in the browser (when it was present
in the collection). I found something on the mailing list that I tweaked
in the db to restore the specific images that i knew were not
displaying, but that wouldn't cover any images that weren't displaying
that i did not know about. Ultimately thats why I decided to start with
a fresh DB, it was never completely broken.

As part of that investigation I found the references to pictures that no
longer existed or had been moved outside of digikam. Curious what the
best way to clean those up was. Doesn't seem to be a clear function to
clean references. Maybe remove the collection and re-add it? Does
removing a collection remove all database entries related to that
collection?
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Thu, 10 May 2012, Nick Anderson wrote:

> Ive been using digikam for a while now. In general as a library
> management tool it works well for me. There are two functions that I
> don't understand.
>
> Collections:
> (I recently wiped out my digikam db and started fresh and have not
> re-connected my remote collections yet so perhaps this has changed.)
>
> The notion of remote or removable collections. Both the thumbnails and
> the metadata are stored in the database correct? Whenever I launch
> digikam and my network collections are not available I cant use
> digikam to browse the disconnected collections. So in order to find an
> image I have to have that collection connected? Since all the
> information is available shouldn't I be able to search and browse the
> collection, but be given some message about attaching the collection
> before being able to view a full size image?

Hello Nick,

A few comments, as I have exactly the same problem, i.e. how to manage
images collections with DK, when images are off-line.

I agree with you, given the database and images thumbnails, DK should be
able to handle all tasks that don't require the effective presence of
original files; tagging, indexing, metadata handling, browsing,
searching, etc.

But this is the theory, and DK isn't an images collections management
program but a « all in one » program, with also a lot of images
processing functions, raw conversion, edition, etc., not just
collections management. And the design doesn't split those two functions
families, so to use DK, whatever you plan to do, your images need to be
present even if of no use at the moment.



Don't know if this can help, as I don't known exactly the way you work,
but here is my personal usage and the way I work around that problem :

1. I use off-line collections because I want my images to be on
pluggable USB hard drives. (I have several of them including security
backups.)
My main reason is to spare space on my desktop fixed drives, the other
reasons are to be able to access my images from different machines,
desktop, laptop, and be able to move with them; an USB drive holds in
a pocket, a desktop computer no.
(Maybe I have too small pockets on my clothes:-)

2. On my desktop fixed drive, I have a complete directories tree of all
of my folders, albums, subalbums, but without the images files.
Instead, original images are replaced by small versions.

(This built is automatized with shell scripts, browsing an images set
and building small versions with command line tools as the ImageMagick
convert program. My typical convert settings are (-resize '300x300>'
-quality 70), so small sized but readable images, and low quality but
high compression. Typical reduced version is 40 Kbytes, instead of the
original 6 to 8 Mbytes, thus a disk space requirement reduced by a
factor of 200 !)

3. And I run DK on that collections tree. I can do all what doesn't
require the original images, browsing, tagging, looking at geolocation,
searching, even searching for duplicates, etc.
As long as DK has an image file, it is happy, even if this isn't the
genuine one.

4. The only thing I can't do is original images edition, of course.
But I don't see this as a problem, because I mostly work on new images
and only in some very rare occasions on archived images. I process my
new images on my standard disk, geotagging, cropping, adjustments, etc.,
and when all is Ok, I move the images set on an USB drive and build the
local reduced versions for DK.

5. And, obviously, when I want, after searching my folders, to get some
original images files to do something with, prints, web album, other,
I need to plug my external drive(s) and access files by pathnames, as
it's the same albums structure with only a different mount point.



I agree this may seem a bit weird, but I have no better idea to work
with off-line collections and, as for me, I get stuck to the idea of
working with removeable media. (In a few years from now, perhaps all of
our images will be « in the clouds » :-)

Maybe some other users could feed this issue with ideas and hints.

Cheers,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[John Stumbles]

On 13/05/12 14:34, Jean-François Rabasse wrote:

> 2. On my desktop fixed drive, I have a complete directories tree of all
> of my folders, albums, subalbums, but without the images files.
> Instead, original images are replaced by small versions.

...

> 5. And, obviously, when I want, after searching my folders, to get some
> original images files to do something with, prints, web album, other,
> I need to plug my external drive(s) and access files by pathnames, as
> it's the same albums structure with only a different mount point.

...

> I agree this may seem a bit weird, but I have no better idea to work
> with off-line collections and, as for me, I get stuck to the idea of
> working with removeable media. (In a few years from now, perhaps all of
> our images will be « in the clouds » :-)
>
> Maybe some other users could feed this issue with ideas and hints.

Sounds a good idea: I might set that up on my laptop which doesn't have
space for all my pictures.

I guess you could do the swapping between reduced-size collection and
full-size collection with symlinks (on non-Windows systems) e.g. have
your digikam root directory be a symlink to either your reduced-size
image tree or the mount-point of your external drive. (You'd have to
arrange things so your digikam database isn't in the dk root directory)

--
John Stumbles                                    http://stumbles.org.uk
:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Sun, 13 May 2012, John Stumbles wrote:

> Sounds a good idea: I might set that up on my laptop which doesn't have space
> for all my pictures.
>
> I guess you could do the swapping between reduced-size collection and
> full-size collection with symlinks (on non-Windows systems) e.g. have your
> digikam root directory be a symlink to either your reduced-size image tree or
> the mount-point of your external drive. (You'd have to arrange things so your
> digikam database isn't in the dk root directory)

Hi John,

If I follow your idea, you mean to trick Digikam with presentation of
images collection either as the full-size version, or reduce-size version ?
Well, could be a great idea if it works, but probably some point should be
tested before :
I don't know exactly how DK does decide, when scanning collections, what is
an existing image and what is a new image. If it's by full pathname check,
the idea would work. But if it uses also some image file specific signature
such as a MD5 or SHA digest, as files are different all images may be
considered as new ones, and all previous images as removed ones.
And the DB would then be reinit accordingly, loosing tags et al.

So, to be checked and if it works it could be a nice hint, especialy for
hotplug/udev gurus that could implement that simlinks swap upon plugging
and unplugging of USB devices.

I must admit I didn't investigate that point, for in most cases I use
both files trees at the same time; the reduced version with DK for browsing
and searching, and possibly the full-size pluggable drive with a files
browser, e.g. Dolphin, when images access is needed.

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[nianderson]

On 05/13/2012 08:34 AM, Jean-François Rabasse wrote:

> 2. On my desktop fixed drive, I have a complete directories tree of all
> of my folders, albums, subalbums, but without the images files.
> Instead, original images are replaced by small versions.
>
> (This built is automatized with shell scripts, browsing an images set
> and building small versions with command line tools as the ImageMagick
> convert program. My typical convert settings are (-resize '300x300>'
> -quality 70), so small sized but readable images, and low quality but
> high compression. Typical reduced version is 40 Kbytes, instead of the
> original 6 to 8 Mbytes, thus a disk space requirement reduced by a
> factor of 200 !)

Thanks for the response, it would  be great if you could share your scripts.

Your workaround seems like it would work ok but generating all the
thumbnails and keeping them updated seems a pain.

Hopefully the developers see this as a useful feature and stub it in
soon. Even read only browsing access would be better than not showing at
all. It would be great to have some kind of offline metadata editing
with some kind of synchronization feature but I would be content with
just read only.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Sun, 13 May 2012, Nick Anderson wrote:

> Thanks for the response, it would  be great if you could share your scripts.

No problem. I've attached the script I use to this e-mail, in case you
or others would wish to play with it. It's really simple stuff,
just get it, read it, try it, hack it to suit your personal usage.

> Your workaround seems like it would work ok but generating all the
> thumbnails and keeping them updated seems a pain.

Yes, could be. I should point out that this is just a trick, a hack,
not a regular way of full working with DK. It only provides a possible
solution to the original question « How can I browse my full collections
even if my original images aren't available at the moment ? »
Nothing more.
And clearly, images files processing/updating is off topic.

Using that kind of trick requires to clearly make a distinction between
images in processing state and images in archived state.
And it's possible to work in a « hybrid » mode, with a DK tree made of
mini images (as placeholders for off-line archived images) and regular
images files, full size, editable, processable.
New subfolders with new images can stay on a local disk several days or
weeks. When processing is considered as done, images go on a removeable
media and are replaced with reduced versions. But reduced versions are
the same images so DK thumbnails and fingerprints remain the same.
(And when, in some occasions, several archived images get edited,
it's always possible to rebuild the reduced versions then ask DK to
rebuild thumbnails and fingerprints on that subset.)

Well actually, I don't think there could be a definitive answer or
workflow. Each individual, each of us, has her/his own way to work and
own usage, and the best way to go is certainly empirical.

> Hopefully the developers see this as a useful feature and stub it in soon.
> Even read only browsing access would be better than not showing at all. It
> would be great to have some kind of offline metadata editing with some kind
> of synchronization feature but I would be content with just read only.

I'd guess it's up to DK developers to answer...
My personal opinion is that it's not a simple work. Digikam is a
personal images processing software *with* images collections
management functions, browse, tag, search.
It is not an images database management software, with possible
local images processing functions, and this makes the difference.
An images bank would require a different model (from a database point
of view), should support temporay off-line data, should also accomodate
for permament off-line images (e.g. only known via a web URL), and many
other things.
Moving from one software model to another is a huge task, IMHO.

Also, how many Digikam users get concerned with this off-line issue ?
If we are only a few percent, I doubt this would be worth the task:-)

Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[nianderson]

On 05/14/2012 05:26 AM, Jean-François Rabasse wrote:
>
> On Sun, 13 May 2012, Nick Anderson wrote:
>
>> Thanks for the response, it would be great if you could share your
>> scripts.
>
> No problem. I've attached the script I use to this e-mail, in case you
> or others would wish to play with it. It's really simple stuff,
> just get it, read it, try it, hack it to suit your personal usage.

Great, Ill take a look at it. I ended up writing a really simple bash
script last night to do similar. Ill clean it up and share it if anyone
else is interested.

>> Your workaround seems like it would work ok but generating all the
>> thumbnails and keeping them updated seems a pain.

> Well actually, I don't think there could be a definitive answer or
> workflow. Each individual, each of us, has her/his own way to work and
> own usage, and the best way to go is certainly empirical.

Yeah everyones work-flow is different and in general I don't think DK
should hamper whatever work-flow you prefer. This comes up for me
because I basically have two trees. An import tree where I am tagging
and renaming thats located on my laptop. I do local edits on those files
with ASP/Bibble. When I am done I put them into the processed tree which
is stored on my NAS.

> An images bank would require a different model (from a database point
> of view), should support temporay off-line data, should also accomodate
> for permament off-line images (e.g. only known via a web URL), and many
> other things.
> Moving from one software model to another is a huge task, IMHO.

I would agree that moving from one model to another is a huge task, but
to me it doesn't seem that it would be very hard to add the off-line
browsing/searching since the thumbs and metadata are already stored
separately from the picture. A bunch of functions would need to be
disabled for those but basic searching and browsing shouldn't require a
ton of work. It would be nice to support the permanent off-line paths as
you suggest but that kind of feature doesn't already exist.

> Also, how many Digikam users get concerned with this off-line issue ?
> If we are only a few percent, I doubt this would be worth the task:-)

I agree if its a very small portion. Perhaps my own habits are tainting
my perception. I have not used a desktop for nearly 7 years now. Higher
end laptops have been powerful enough for me. In the last several years
I have transitioned from monstrous spinning rust to smaller SSD drives
in my laptops. They are usually a bit better on battery consumption and
significantly faster. Of course they are smaller as well so if I had not
been already for disk redundancy, I would have been moving images to
some other storage location. I have been using a NAS for years, I
imagine USB hard drives are still quite popular with the price point. I
know several photographers who until recently had been burning DVDs.

It isn't a huge issue since I am using a NAS and I have all my pictures
there. But just thinking about someone who has a large collection of
DVDs or CDs trying to hunt down an image pains me. I suppose there is
separate software to catalog an optical disc collection but it seems
like DK already has all the needed stuff minus exposing it in the UI.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jean-François Rabasse]

On Mon, 14 May 2012, Nick Anderson wrote:

> ...
> I know several photographers who until recently had been burning DVDs.
>
> It isn't a huge issue since I am using a NAS and I have all my pictures
> there. But just thinking about someone who has a large collection of DVDs or
> CDs trying to hunt down an image pains me. I suppose there is separate
> software to catalog an optical disc collection but it seems like DK already
> has all the needed stuff minus exposing it in the UI.

Many thanks Nick, this is the perfect example of the usefulness of a
regular off-line data support.

1. Images collections on CD are always off-line. If it may be common for
someone to have a few dozens of CD/DVD, it's unlikely to have also a few
dozens of CD readers plugged onto the computer.

2. Images on CD aren't expected to be editable/processable
(even on CD/RW:-), so physical presence of the file isn't mandatory
and any built thumbnail would be up to date forever.

So, being able to read/index a CD, get images pathnames, read metadata,
build thumbnails, etc., then put back the CD into its box, could be
really helpful for future indexing, browsing, searching, along with
other images on the local disk.


Regards,
Jean-François
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users