[Bug 293832] New: Digitaglinktree - Multi tag level combination - find any image though directory browsing [patch]

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #20 from  <cyril raphanel gmail com>  2012-02-16 18:39:54 ---
I need help

I call the function by
# Manage tags
createLinkTreeMultiBranch($refImages->{$i},${linktreeRootDir},$photoRootDir,0);


sub createLinkTreeMultiBranch{

    my(@tag_list,$linktreeRootDir,$photoRootDir,$counttag)=@_;
    $path=$tag_list->{"path"};
    $image=$tag_list->{"image"};
    $imgid=$tag_list->{"imgid"};
    $imgYear=$tag_list->{"imgYear"};
    $counttag++;
    $toto=$tag_list->{"tagpath"};
    print "IMAGEID - $tag_list - $path - $image - $imgid - $imgYear\n";
    for( $j=0;  $j<=$#{${tag_list}->{"tag"}}; $j++){
            $tag=$tag_list->{"tagpath"}->[$j];
            print "NB -- COUNTTAG -- $counttag -- TAG -- $tag\n";
            # Check if we have not tried the path before
            if(${linktreeRootDir} !~ /($tag)/) {
                # print "$item -- $image --$item/$tag $tagcount $j\n";
                # Create subdirectories needed
                if( ! -d "${linktreeRootDir}/$tag/_all" ){
                    $ret=mkpath("${linktreeRootDir}/$tag/_all", 0, 0755);

                    if( !$ret ){
                        die "Cannot mkdir \"${linktreeRootDir}/$tag/_all\"\n";
                    }
                }
                # Avoid conflicts for images with the same name in one tag
directory
                $linkName=$imgid."_".$image;

                # Get relative path from absolute one
                $relPath="";
                $relPath=File::Spec->abs2rel("${photoRootDir}$path",
"${linktreeRootDir}/$tag/_all" ) ;
                if( !length($relPath)){
                    die "Cannot create relative path from
\"${linktreeRootDir}/$tag/_all\" to \"${photoRootDir}$path\" . Abort.\n";
                }
                # Create link
                $sourceDir="$relPath/$image";
                $destDir="${linktreeRootDir}/$tag/_all/$linkName";

                if (!-e $destDir){
                    $ret=symlink($sourceDir, $destDir);
                    if( !$ret ){
                        $ret="$!";
                        warn "Warning: Failed symbolic linking \"$sourceDir\"
to \"$destDir\": ",
                        "\"$ret\"\n";
                    }
                }

                if ($counttag < $#{$refImages->{$i}->{"tag"}}){
                    # create subdir
                   
createLinkTreeMultiBranch(\%refImages,${linktreeRootDir}/$tag,$photoRootDir,$counttag);
                }

            }
    }
}

I get
IMAGEID - HASH(0xaa06d0c) -  -  -  -

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #21 from  <cyril raphanel gmail com>  2012-02-16 18:41:22 ---
Created an attachment (id=68853)
 --> (http://bugs.kde.org/attachment.cgi?id=68853)
HELP -- PERL SUPPORT NEEDED

I think I am lost with hash & table reference ....

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68853|0                           |1
        is obsolete|                            |




--- Comment #22 from  <cyril raphanel gmail com>  2012-02-16 19:14:00 ---
Created an attachment (id=68855)
 --> (http://bugs.kde.org/attachment.cgi?id=68855)
Recursion buggy but some hash & array issue fixed

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #23 from  <cyril raphanel gmail com>  2012-02-16 19:59:01 ---
My test case is
tag->tag1
tag->tag2
tag->tag3
tag->tag4
tag->tag5
tag->tag6
tag->tag7
tag->tag8
tag->tag9
tag->tag10

all images have all the 10 tags it create only the following directories
/tag/tag1/tag/tag2/tag/tag3/tag/tag4/tag/tag5/.../tag/tag10/

where it should also create all alternative combination such as
tag/tag2/tag/tag1/..../tag/tag10/
until
tag/tag10/tag/tag9/..../tag/tag1/

Any idea what I am doing in my recursion logic?

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #24 from  <cyril raphanel gmail com>  2012-02-17 05:49:04 ---
Created an attachment (id=68863)
 --> (http://bugs.kde.org/attachment.cgi?id=68863)
Good base for recursion version - Performance issue still

Bottleneck seems to be directory creation (and read?).

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68855|0                           |1
        is obsolete|                            |




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #25 from  <cyril raphanel gmail com>  2012-02-17 05:59:02 ---
(From update of attachment 68863)
Performance improvement possibility:
- use mkdir instead of mkpath
- limit the number of levels managed with directory logic
- created directories stored in memory to avoid disk read when checking if
directory exist or not

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #26 from  <cyril raphanel gmail com>  2012-02-17 06:50:21 ---
(From update of attachment 68863)
Performance:
- nb of link write operation also impacting

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[krienke]

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #27 from  <krienke uni-koblenz de>  2012-02-17 12:21:18 ---
Just to give some feedback. Today I tried the latest patch. This time I was
away for an hour but the script was still busy. Afterwards I wanted to remove
the base directory and this also took more than 20min (on a Core I5 8GB Ext4
fs). It seems that simply the number of directory entries created by your
algorithm is to big.

Perhaps it would be a good idea to go the other way round: To make it mandatory
that -M takes a list of tags to be included where all tags are excluded by
default. This at least would ensure a positive user experience in a way that
the script terminates after a short while and the user can inspect if the
result is what he intended to to. The other way round depending on the number
of photos and tags the script runs virtual for ever and the user might decide
not to use it any longer.


This

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #28 from  <cyril raphanel gmail com>  2012-02-17 14:59:54 ---
Thanks,

I agree the volum of link & directories it creates is not manageable. It is
actually not a processor or memory issue (during the whole process both are low
in term of resource) but a disk access challenge.
Noticed also the delete is then a pain ....

I will look in the possibility you mention, I am also testing a level option to
limit (only x levels ....) the volum.
Thinking also about another tradeoff option as for know the logic create a lot
of duplicate link and directory in order to follow all possible human logic:
tag1->tag2->tag3
tag1->tag3->tag2
tag2->tag1->tag3
tag2->tag3->tag1
tag3->tag2->tag1
tag3->tag1->tag2
contains the same set of link ... some potential here if you can limit the
number of "thinking" you can allow like People->Places->People->Places might be
an overkill human thinking path :-)

Combine then keyword filtering (positive or negative) and level setting should
give enought flexibility.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #29 from  <cyril raphanel gmail com>  2012-02-19 11:26:23 ---
Created an attachment (id=68923)
 --> (http://bugs.kde.org/attachment.cgi?id=68923)
Recursion version working with all flexibility to control performance.

 -M            Make multi-level directory combining all tag classification
               including in each image.
               Number specifies the maximum tree level number.
-i             Including tags having specific string in their tag path.
               Use all if all tags needed.
-e             Excluding tags having specific string in their tag path.
               Use all if all tags needs to be exluded. Default:all with -M and
none without -M
               example Digikam,People would exclude all tags having Digikam or
People in their path
-V             Verbose mode

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68923|0                           |1
        is obsolete|                            |




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68923|1                           |0
        is obsolete|                            |




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68863|0                           |1
        is obsolete|                            |




--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[krienke]

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #30 from  <krienke uni-koblenz de>  2012-02-21 13:57:36 ---
Hi, looks better now. I guess that -M should always be used with a number as
argument defining the number of levels? At the moment one can call
digitaglinktree with -M and nothing happens. In this case an error should be
printed that gives a hint.  

The same is true for -i . Without using -i nothing happened even when I called
the script with  -M 100. This is correct and safe for the user, but the script
should print an error message that -M was used without -i

By the way last time I called  the previous version of digitaglinktree using -M
where it did not terminate even after a hour, I noticed that at least 8.000.000
inodes  in my filesystem had been eaten up after I stopped the script. So the
defaults as they are now (do nothing without explicit option from the user) are
good but some error handling is missing as well as the manual pages.

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #31 from  <cyril raphanel gmail com>  2012-02-26 19:59:44 ---
Thanks for the feedback

(In reply to comment #30)
> Hi, looks better now. I guess that -M should always be used with a number as
> argument defining the number of levels? At the moment one can call
> digitaglinktree with -M and nothing happens. In this case an error should be
> printed that gives a hint.  

Good point!
This was what I had in mind but did not have time to find how to make the test
in perl ... I am fairly new in parameter management in this language.


> The same is true for -i . Without using -i nothing happened even when I called
> the script with  -M 100. This is correct and safe for the user, but the script
> should print an error message that -M was used without -i

I wanted to put a default value based but error message is better solution.

A bit overloaded right now but will try to man the change in the coming days
... no idea how to update the man package but willing to learn :-)

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[krienke]

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #32 from  <krienke uni-koblenz de>  2012-02-29 06:15:02 ---
The man page is simply a text file in roff format (it contains special commands
for indentation, boldface etc). You can edit it with any text editor. Just look
at the existing file to see how it works and add a description of the new
options and how your extension to digitaglinktree works and what it can do for
the user.

If you want to preview your writings on linux you can do this by:
nroff -man digitaglinktree.1|more

Rainer

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Gilles Caulier-4]

https://bugs.kde.org/show_bug.cgi?id=293832





--- Comment #33 from Gilles Caulier <caulier gilles gmail com>  2012-02-29 06:21:39 ---
Under KDE desktop, you can use this url into Dolphin or Konqueror :

man:/digitaglinktree/

Gilles Caulier

--
Configure bugmail: https://bugs.kde.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #68765|0                           |1
        is obsolete|                            |
  Attachment #68923|0                           |1
        is obsolete|                            |

--- Comment #34 from [hidden email] ---
Created attachment 69972
  --> https://bugs.kde.org/attachment.cgi?id=69972&action=edit
Version 1.8.2

# 1.8.0 -> 1.8.1 2012/03/28 Cyril Raphanel, [hidden email]
# Added (-i) option to include only some tags - default 'all'
# Added (-e) option to exclude some tags - default ''
# Added (-M) option to create directory hierarchy base on tags selection,
number of hierarchy level to be specified
# Added (-V) option to enable verbose mode

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

https://bugs.kde.org/show_bug.cgi?id=293832

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #69972|0                           |1
        is obsolete|                            |

--- Comment #35 from [hidden email] ---
Created attachment 69973
  --> https://bugs.kde.org/attachment.cgi?id=69973&action=edit
Version 1.8.2

# 1.8.1 -> 1.8.2 2012/03/28 Cyril Raphanel, [hidden email]
# Added (-i) option to include only some tags - default 'all'
# Added (-e) option to exclude some tags - default ''
# Added (-M) option to create directory hierarchy base on tags selection,
number of hierarchy level to be specified
# Added (-V) option to enable verbose mode

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel