Import from iPhoto 9.5, XMP sidecar format

[Jens Benecke-2]

Hello everybody,

I have - finally - decided to try to move my 41000 images, movies and
edits from iPhoto to Digikam and am currently writing an import script
to automate as much of the import as possible. I have already succeeded
in converting a lot automatically by creating XMP sidecar files for
every image, but some bits are still missing.

As a starting point I used the "iphoto2xmp.rb" script from here:
https://gist.github.com/lpar/2191225

So far this script can convert

- Original photos only (no modified versions, ie. crops, edits,
optimizations)
- iPhoto's Events as folders (= Albums in Digikam),
- Photo names (= filenames), descriptions, ratings
- Face names (as keywords)
- Locations (as GPS coordinates)

The following is missing and I plan to implement it. If anybody has a
script or info about how exactly to export this from iPhotos mess of
SQLite databases, please do speak up! Unfortunately these bits are
missing exactly because they are not contained in the AlbumData.xml file
which iPhoto creates for interoperability with other software.

- Import modified photos as a second version of the original
   ... how do I specify within the XMP sidecar that an image is a second
version of another image?
- iPhotos keywords (from Database/Keywords.plist and
Database/apdb/Library.apdb::RKKeyword)
- iPhotos Albums as "album" keywords (from AlbumData.xml "List of
Albums" key)
- iPhotos Folders as "folder" keywords (Database/Library.apdb::RKFolder)
- Location names (found in Database/Properties.apdb as a "RKPlace"
SQlite table) with radius
- Hidden and Starred flags (as appropriate keywords)
- Face rectangles within images
   ... how do I make Digikam's recognition engine learn from an existing
confirmed set of named Faces?

Can anybody shed some light on the open questions above or maybe even
share a script that does things like the above already?
Is the XMP sidecar format that Digikam accepts and expects - and the
list of parameters that it understands - documented somewhere?


Thank you!
Jens
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Alan Pater]

I can't claim a lot of expertise, but perhaps xmpMM DerivedFrom might
be what you are looking for. However, I am not sure what digikam uses
for image history.

       http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf#G4.1133171

       http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#xmpMM

For your Face rectangles, the MWG Region properties are used by digikam.

       http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf#page=51

       http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html#Regions

Hopefully this is a bit of help on getting the conversion further along.

Cheers!

On Mon, Aug 17, 2015 at 4:24 PM, Jens Benecke
<[hidden email]> wrote:

> - Import modified photos as a second version of the original
>   ... how do I specify within the XMP sidecar that an image is a second
> version of another image?

> - Face rectangles within images
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Alan Pater]

For the others, I would suggest following the MWG Guidelines where
ever applicable, as digikam generally follows those guidelines.
_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jens Benecke-2]

Hello,

thank you! I browsed through the documentation but the number of
specific examples is very low (i.e. zero).
Is there an example XMP sidecar file that

- Digikam can fully import,
- contains image versions, caption, description/comment, ratings,
"hidden" and "flagged" attributes, faces (rectangles, IDs and names),
GPS location data + names and so on?

Is there any property that Digikam supports which can NOT be imported
using sidecar files?

Thanks!

Jens Benecke


Am 19.08.2015 um 16:53 schrieb Alan Pater:

> I can't claim a lot of expertise, but perhaps xmpMM DerivedFrom might
> be what you are looking for. However, I am not sure what digikam uses
> for image history.
>
>         http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf#G4.1133171
>
>         http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#xmpMM
>
> For your Face rectangles, the MWG Region properties are used by digikam.
>
>         http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf#page=51
>
>         http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html#Regions
>
> Hopefully this is a bit of help on getting the conversion further along.
>
> Cheers!
>
> On Mon, Aug 17, 2015 at 4:24 PM, Jens Benecke
> <[hidden email]> wrote:
>
>> - Import modified photos as a second version of the original
>>    ... how do I specify within the XMP sidecar that an image is a second
>> version of another image?
>> - Face rectangles within images
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Alan Pater]

I haven't seen a full sample XMP file, but if you are comfortable
reading code, the tags and properties that digikam imports are in

https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/dmetadata/dmetadata.cpp

I suspect you are covered for "caption, description/comment, ratings,
faces (rectangles, IDs and names), GPS location data + names. Not so
sure for image versions and "hidden" "flagged" attributes.


If you could share an exported XMP file from iPhoto?

On Sun, Aug 23, 2015 at 8:32 AM, Jens Benecke
<[hidden email]> wrote:

> Hello,
>
> thank you! I browsed through the documentation but the number of specific
> examples is very low (i.e. zero).
> Is there an example XMP sidecar file that
>
> - Digikam can fully import,
> - contains image versions, caption, description/comment, ratings, "hidden"
> and "flagged" attributes, faces (rectangles, IDs and names), GPS location
> data + names and so on?
>
> Is there any property that Digikam supports which can NOT be imported using
> sidecar files?
>
> Thanks!
>
> Jens Benecke
>
>
>
> Am 19.08.2015 um 16:53 schrieb Alan Pater:
>>
>> I can't claim a lot of expertise, but perhaps xmpMM DerivedFrom might
>> be what you are looking for. However, I am not sure what digikam uses
>> for image history.
>>
>>
>> http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf#G4.1133171
>>
>>
>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#xmpMM
>>
>> For your Face rectangles, the MWG Region properties are used by digikam.
>>
>>         http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf#page=51
>>
>>
>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html#Regions
>>
>> Hopefully this is a bit of help on getting the conversion further along.
>>
>> Cheers!
>>
>> On Mon, Aug 17, 2015 at 4:24 PM, Jens Benecke
>> <[hidden email]> wrote:
>>
>>> - Import modified photos as a second version of the original
>>>    ... how do I specify within the XMP sidecar that an image is a second
>>> version of another image?
>>> - Face rectangles within images
>>
>> _______________________________________________
>> Digikam-users mailing list
>> [hidden email]
>> https://mail.kde.org/mailman/listinfo/digikam-users
>
>
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jens Benecke-2]

Thanks! Unfortunately, it's not so easy: iPhoto does not export anything
by itself.

The iPhoto database consists of an "AlbumData.xml" file which contains a
lot, but not all, info about all images and which is being created
explicitly for apps that want to access the images read-only (eg.
viewers). This is documented.

The rest (keywords, location names, flagged and hidden attributes, and
lots of internal info like whether an image has been modified and what
has been done with it) is scattered over half a dozen ".apdb" files
which are SQLite3 databases with an undocumented and sometimes cryptic
schema. I have done my best to decrypt and document this schema. Now I
need to find out where to write the image metadata so that Digikam and
as many other apps as possible can benefit.

The Ruby script which I'm extending uses AlbumData.xml and two of the
SQLite3 databases to create XMP sidecar files which are then copied to a
directory structure in which the iPhoto images are hardlinked (to
minimize additional disk space). This new directory scheme can then be
imported by Digikam. If this works I can delete the iPhoto database and
the sidecar files again (or archive them, or just ignore them - don't
know yet). The actual image files (JPG, MP4, ...) should never be
modified themselves.

My current implementation creates a JSON structure like this

  [{"XMP:GPSLatitude":48.123,"XMP:GPSLongitude":8.789,"XMP:Headline":"Movie Example","XMP:Rating":2,"XMP:description":"Whatever","SourceFile":"/......./IMG_5451.xmp"}]

and a command like this

   exiftool -j=JSONFILE XMPFILE

to generate XMP from the JSON input.

Now, this JSON already contains GPS data, Title, description, rating,
and SourceFile. But it's missing the info below because I don't know in
which tags I should write it. I tried several combinations, but so far
I'm not successful. Any help - for example, an XMP file which contains
all metainformation that Digikam can import, would be great.


Regards
Jens


Am 23.08.2015 um 16:25 schrieb Alan Pater:

> I haven't seen a full sample XMP file, but if you are comfortable
> reading code, the tags and properties that digikam imports are in
>
> https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/dmetadata/dmetadata.cpp
>
> I suspect you are covered for "caption, description/comment, ratings,
> faces (rectangles, IDs and names), GPS location data + names. Not so
> sure for image versions and "hidden" "flagged" attributes.
>
>
> If you could share an exported XMP file from iPhoto?
>
> On Sun, Aug 23, 2015 at 8:32 AM, Jens Benecke
> <[hidden email]>  wrote:
>> Hello,
>>
>> thank you! I browsed through the documentation but the number of specific
>> examples is very low (i.e. zero).
>> Is there an example XMP sidecar file that
>>
>> - Digikam can fully import,
>> - contains image versions, caption, description/comment, ratings, "hidden"
>> and "flagged" attributes, faces (rectangles, IDs and names), GPS location
>> data + names and so on?
>>
>> Is there any property that Digikam supports which can NOT be imported using
>> sidecar files?
>>
>> Thanks!
>>
>> Jens Benecke
>>
>>
>>
>> Am 19.08.2015 um 16:53 schrieb Alan Pater:
>>> I can't claim a lot of expertise, but perhaps xmpMM DerivedFrom might
>>> be what you are looking for. However, I am not sure what digikam uses
>>> for image history.
>>>
>>>
>>> http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf#G4.1133171
>>>
>>>
>>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#xmpMM
>>>
>>> For your Face rectangles, the MWG Region properties are used by digikam.
>>>
>>>          http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf#page=51
>>>
>>>
>>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html#Regions
>>>
>>> Hopefully this is a bit of help on getting the conversion further along.
>>>
>>> Cheers!
>>>
>>> On Mon, Aug 17, 2015 at 4:24 PM, Jens Benecke
>>> <[hidden email]>  wrote:
>>>
>>>> - Import modified photos as a second version of the original
>>>>     ... how do I specify within the XMP sidecar that an image is a second
>>>> version of another image?
>>>> - Face rectangles within images
>>> _______________________________________________
>>> Digikam-users mailing list
>>> [hidden email]
>>> https://mail.kde.org/mailman/listinfo/digikam-users
>> _______________________________________________
>> Digikam-users mailing list
>> [hidden email]
>> https://mail.kde.org/mailman/listinfo/digikam-users
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users

[Jens Benecke-2]

Hello,

I have extended and reworked my iPhoto conversion script and published
it here:

     https://github.com/jensb/iphoto2xmp

I have a problem exporting face rectangles. I cannot get them from the
XMP sidecar file into Digikam. I am using the exact same XMP format that
Digikam produces when I let it *create* XMP sidecar files but Digikam
simply shows no face rectangles when the files are loaded.

  Here is what I did:

1. Create face tags in a photo in Digikam.
2. Select Image / Write Metadata to file    (in Settings I have
activated "read from sidecar" and "write to sidecar file")
3. Open sidecar file, extract XMP faces information
4. Replace the face location and name information with template variables
5. Use this template for my conversion script

But when I recreate the sidecar files (with different values) Digikam
does not recognize the face information. Title, caption, description,
rating, etc. are correctly imported however.

Q1: Here is an example face information. Does this (within the <rdf:RDF
...> tag) have the right format?

    <mwg-rs:Regions rdf:parseType="Resource">
     <mwg-rs:RegionList>
      <rdf:Bag>
              <rdf:li>
         <rdf:Description mwg-rs:Name="John Doe"
                          mwg-rs:Email="" mwg-rs:Type="Face">
          <mwg-rs:Area stArea:x="0.50781250"
                       stArea:y="0.28675933"
                       stArea:w="0.04687500"
                       stArea:h="0.07022677" stArea:unit="normalized" />
         </rdf:Description>
        </rdf:li>"
            </rdf:Bag>
     </mwg-rs:RegionList>
    </mwg-rs:Regions>

    <MP:RegionInfo rdf:parseType="Resource">
     <MPRI:Regions>
      <rdf:Bag>
             <rdf:li
        MPReg:Rectangle="0.50781250, 0.28675933, 0.04687500, 0.07022677"
        MPReg:PersonDisplayName="John Doe"/>
            </rdf:Bag>
     </MPRI:Regions>
    </MP:RegionInfo>


Q2: When Digikam exports a sidecar file, the values for top left x and y
are different for the mwg-rs and the MP structure above. Is this
intentional? If so, how are both values calculated?


Thank you!

Regards



Am 28.08.2015 um 22:50 schrieb Jens Benecke:

> Thanks! Unfortunately, it's not so easy: iPhoto does not export
> anything by itself.
>
> The iPhoto database consists of an "AlbumData.xml" file which contains
> a lot, but not all, info about all images and which is being created
> explicitly for apps that want to access the images read-only (eg.
> viewers). This is documented.
>
> The rest (keywords, location names, flagged and hidden attributes, and
> lots of internal info like whether an image has been modified and what
> has been done with it) is scattered over half a dozen ".apdb" files
> which are SQLite3 databases with an undocumented and sometimes cryptic
> schema. I have done my best to decrypt and document this schema. Now I
> need to find out where to write the image metadata so that Digikam and
> as many other apps as possible can benefit.
>
> The Ruby script which I'm extending uses AlbumData.xml and two of the
> SQLite3 databases to create XMP sidecar files which are then copied to
> a directory structure in which the iPhoto images are hardlinked (to
> minimize additional disk space). This new directory scheme can then be
> imported by Digikam. If this works I can delete the iPhoto database
> and the sidecar files again (or archive them, or just ignore them -
> don't know yet). The actual image files (JPG, MP4, ...) should never
> be modified themselves.
>
> My current implementation creates a JSON structure like this
>
>  [{"XMP:GPSLatitude":48.123,"XMP:GPSLongitude":8.789,"XMP:Headline":"Movie
> Example","XMP:Rating":2,"XMP:description":"Whatever","SourceFile":"/......./IMG_5451.xmp"}]
>
>
> and a command like this
>
>   exiftool -j=JSONFILE XMPFILE
>
> to generate XMP from the JSON input.
>
> Now, this JSON already contains GPS data, Title, description, rating,
> and SourceFile. But it's missing the info below because I don't know
> in which tags I should write it. I tried several combinations, but so
> far I'm not successful. Any help - for example, an XMP file which
> contains all metainformation that Digikam can import, would be great.
>
>
> Regards
> Jens
>
>
> Am 23.08.2015 um 16:25 schrieb Alan Pater:
>> I haven't seen a full sample XMP file, but if you are comfortable
>> reading code, the tags and properties that digikam imports are in
>>
>> https://projects.kde.org/projects/extragear/graphics/digikam/repository/revisions/master/entry/libs/dmetadata/dmetadata.cpp 
>>
>>
>> I suspect you are covered for "caption, description/comment, ratings,
>> faces (rectangles, IDs and names), GPS location data + names. Not so
>> sure for image versions and "hidden" "flagged" attributes.
>>
>>
>> If you could share an exported XMP file from iPhoto?
>>
>> On Sun, Aug 23, 2015 at 8:32 AM, Jens Benecke
>> <[hidden email]>  wrote:
>>> Hello,
>>>
>>> thank you! I browsed through the documentation but the number of
>>> specific
>>> examples is very low (i.e. zero).
>>> Is there an example XMP sidecar file that
>>>
>>> - Digikam can fully import,
>>> - contains image versions, caption, description/comment, ratings,
>>> "hidden"
>>> and "flagged" attributes, faces (rectangles, IDs and names), GPS
>>> location
>>> data + names and so on?
>>>
>>> Is there any property that Digikam supports which can NOT be
>>> imported using
>>> sidecar files?
>>>
>>> Thanks!
>>>
>>> Jens Benecke
>>>
>>>
>>>
>>> Am 19.08.2015 um 16:53 schrieb Alan Pater:
>>>> I can't claim a lot of expertise, but perhaps xmpMM DerivedFrom might
>>>> be what you are looking for. However, I am not sure what digikam uses
>>>> for image history.
>>>>
>>>>
>>>> http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf#G4.1133171 
>>>>
>>>>
>>>>
>>>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/XMP.html#xmpMM
>>>>
>>>> For your Face rectangles, the MWG Region properties are used by
>>>> digikam.
>>>>
>>>> http://www.metadataworkinggroup.org/pdf/mwg_guidance.pdf#page=51
>>>>
>>>>
>>>> http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/MWG.html#Regions
>>>>
>>>> Hopefully this is a bit of help on getting the conversion further
>>>> along.
>>>>
>>>> Cheers!
>>>>
>>>> On Mon, Aug 17, 2015 at 4:24 PM, Jens Benecke
>>>> <[hidden email]>  wrote:
>>>>
>>>>> - Import modified photos as a second version of the original
>>>>>     ... how do I specify within the XMP sidecar that an image is a
>>>>> second
>>>>> version of another image?
>>>>> - Face rectangles within images
>>>> _______________________________________________
>>>> Digikam-users mailing list
>>>> [hidden email]
>>>> https://mail.kde.org/mailman/listinfo/digikam-users
>>> _______________________________________________
>>> Digikam-users mailing list
>>> [hidden email]
>>> https://mail.kde.org/mailman/listinfo/digikam-users
>> _______________________________________________
>> Digikam-users mailing list
>> [hidden email]
>> https://mail.kde.org/mailman/listinfo/digikam-users
>
> _______________________________________________
> Digikam-users mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-users

_______________________________________________
Digikam-users mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-users