[digiKam-users] removing orphaned xmp files under Linux

[Peter Mc Donough]

Hi,

I noticed that there are some orphaned xmp-files in my photo folders.

Does anybody know some command line magic for Linux for at least finding
them?

cu
Peter

[Chris Green]

Peter Mc Donough <[hidden email]> wrote:
> Hi,
>
> I noticed that there are some orphaned xmp-files in my photo folders.
>
> Does anybody know some command line magic for Linux for at least finding
> them?
>
Do they have a suffix or at least a recognisble pattern in their name?
If so it's dead easy using the 'find' command:-

    find <name of top level directory> -name '*.xmp'

If you want to remove the files as well:-

    find <name of top level directory> -name '*.xmp' -exec rm {} \;

--
Chris Green
·

[digikam-2]

On Wed, 7 Mar 2018 21:35:30 +0000
Chris Green <[hidden email]> wrote:

> Do they have a suffix or at least a recognisble pattern in their
> name? If so it's dead easy using the 'find' command:-
>
>     find <name of top level directory> -name '*.xmp'
>
> If you want to remove the files as well:-
>
>     find <name of top level directory> -name '*.xmp' -exec rm {} \;

I don't think so. It finds ALL the files that have an .xmp
extension, including all the files that have a raw counterpart.

Doing the rm will delete all the xmp files

> find ~/darktable/ -name '*.xmp'|less

Here is an xmp from the find:

/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp

> ls /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363*

/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2
/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp



--
sknahT

vyS

[woenx]

I'm not a bash expert whatsoever, but I think this could work, I based it on the script found here: https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extension Run it from your picture directory. It will scan and delete all orphaned xmp files. Right now it would only show which files would be deleted. Uncomment the second to last line for actually deleting the orphaned files, or the last one to rename to orphaned files with a ".old" extension (so you can find and delete them later). I tried it with a dir with fake contents, and it seems to work. But please, use it at your own risk.

#!/bin/bash
shopt -s nullglob extglob;

#dir=~/mypictures
#cd $dir

# explore dirs recursively, finding .xmp files
find . -type f -name "*.xmp" | while read file
do
    #echo "$file"
    candidates=("${file%.*}"@() "${file%.*}".{cr2}@());
    [[ ${#candidates[@]} -eq 0 ]] && echo "deleted orphaned file $file"
    #[[ ${#candidates[@]} -eq 0 ]] && rm "$file" 
    #[[ ${#candidates[@]} -eq 0 ]] && mv "${file}" "${file}.old"
done

---

Sent from the digikam-users mailing list archive at Nabble.com.

[Peter Mc Donough]

Am 08.03.2018 um 03:30 schrieb woenx:

> I'm not a bash expert whatsoever, but I think this could work, I based it on
> the script found here:
> https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
> it from your picture directory. It will scan and delete all orphaned xmp
> files. Right now it would only show which files would be deleted. Uncomment
> the second to last line for actually deleting the orphaned files, or the
> last one to rename to orphaned files with a ".old" extension (so you can
> find and delete them later).I tried it with a dir with fake contents, and it
> seems to work. But please, use it at your own risk.
>

Whow! Here is the copy from:

#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#
# --------------------

  I'll test it later with "mv" instead of "rm"

Thanks
Peter

[Peter Mc Donough]

Am 08.03.2018 um 10:00 schrieb Peter Mc Donough:
> Am 08.03.2018 um 03:30 schrieb woenx:
>> I'm not a bash expert whatsoever, but I think this could work, I based
>> it on
...

Tested, but my Kubuntu's Bash doesn't like "shopt", error message below.
I supose a shell option. It probably has to be activated somehow:

Any ideas
Peter

-----------------
Error Message:
/home/peter/foto_xmp_ex.sh: 5: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 12: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")

> shopt -s nullglob extglob;
>
> # Get all sidecar files
> for file in *.{xmp,pts,pp3}
> do
>    # Generate all permutations of filenames that it may belong to,
>    # and let globbing delete the ones that don't exist
>    candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());
>
>    # If none exist, the file can be deleted
>    [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
> done
> #

[Peter Mc Donough]

Am 08.03.2018 um 12:28 schrieb Peter Mc Donough:

I better add the complete shell script

cu
Peter


#!/bin/bash
# 20180308 - entfernen ueberfluessiger "xmp/pts/pp3" -Dateien
#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#

Error message:
peter@kubu2-lux:~/temp/oldies_mcd/2$ sh  ~/foto_xmp_ex.sh
/home/peter/foto_xmp_ex.sh: 5: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 12: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")
peter@kubu2-lux:~/temp/oldies_mcd/2$

[woenx]

Did you keep the "#!/bin/bash " at the beggining of the script? I think the shopt command only works in bash. What happens if you type it directly on the terminal? By the way, the difference between the script on stackoverflow and the one I pasted, is that mine scans directories recursively.

---

Sent from the digikam-users mailing list archive at Nabble.com.

[Chris Green]

[hidden email] wrote:

> On Wed, 7 Mar 2018 21:35:30 +0000
> Chris Green <[hidden email]> wrote:
>
> > Do they have a suffix or at least a recognisble pattern in their
> > name? If so it's dead easy using the 'find' command:-
> >
> >     find <name of top level directory> -name '*.xmp'
> >
> > If you want to remove the files as well:-
> >
> >     find <name of top level directory> -name '*.xmp' -exec rm {} \;
>
> I don't think so. It finds ALL the files that have an .xmp
> extension, including all the files that have a raw counterpart.
>
> Doing the rm will delete all the xmp files
>
> > find ~/darktable/ -name '*.xmp'|less
>
> Here is an xmp from the find:
>
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp
>
> > ls /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363*
>
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp
>

OK, that's why I asked "Do they have a suffix...", I meant the orphan
ones.

So you'll have to write a simple (bash?) script that uses find (as
above) to find all the xmp files and then removes only the orphans.

The logic would be something like:-

    find an .xmp file
    strip off the .xmp (use basename)
    find the 'parent' file (the one with the .xmp removed)
    if the above fails then remove the .xmp file

--
Chris Green
·

[woenx]

This is exactly the script I pasted above does. I recursively scans for .xmp
files and checks if there's a matching cr2. If there isn't, the xmp is
deleted.

I think the "shopt" commands requires a bash console. Did you put the
"#!/bin/bash" at the beginning of the script? What happens if you just run
the "shopt -s nullglob extglob" in a console?



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[digikam-2]

On Thu, 8 Mar 2018 10:00:05 +0100
Peter Mc Donough <[hidden email]> wrote:

> # shopt -s nullglob extglob;
>
> # Get all sidecar files
> for file in *.{xmp,pts,pp3}
> do
>    # Generate all permutations of filenames that it may belong to,
>    # and let globbing delete the ones that don't exist
>    candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());
>
>    # If none exist, the file can be deleted
>    [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
> done
> #
> # --------------------
>
>   I'll test it later with "mv" instead of "rm"

Watch for this script, there is a "major weakness".

Some softwares (like darktable) create multiple xmp files (xxx_01,
xxx_02, xxx-1, xxx-2...) for the same raw file as virtual copies and
the script will also delete them.

--
sknahT

vyS

[Peter Mc Donough]

Am 08.03.2018 um 17:32 schrieb woenx:
> This is exactly the script I pasted above does. I recursively scans for .xmp
> files and checks if there's a matching cr2. If there isn't, the xmp is
> deleted.
>
> I think the "shopt" commands requires a bash console. Did you put the
> "#!/bin/bash" at the beginning of the script? What happens if you just run
> the "shopt -s nullglob extglob" in a console?
>

just
shopt -s nullglob extglob

as user and as root

No output??

cu
Peter

[woenx]

Yep, that's right, no output means that the command was executed
successfully.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Peter Mc Donough]

Am 08.03.2018 um 21:43 schrieb woenx:
> Yep, that's right, no output means that the command was executed
> successfully.
>

Where is my mistake?

I create an test-folder im my home-directory,
copy three raw-files including their xmp-files,
remove one raw-file to get one ophaned xmp-file.

I added ls -l to the script and "rw2" to "candidates"

If I run the script in that folder, the output should show, at the end,
the orphaned xmp-file but I get only the list of all five files and
still the error message below.
The error points to  one "(" too many or too few in the line with the
candidates comparisons.


#!/bin/bash
# 20180308 - entfernen ueberfluessiger "xmp/pts/pp3" -Dateien
#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
ls -l
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf,rw2}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#

-------------
error message
-------------

peter@kubu2-lux:~/test$ sh ~/foto_xmp_ex.sh
insgesamt 38332
-rw-r--r-- 1 peter peter 19478572 Apr  9  2015
20150314_155955_AIDAluna_orgP.rw2
-rw-r--r-- 1 peter peter     6283 Jan 17 17:42
20150314_155955_AIDAluna_orgP.rw2.xmp
-rw-r--r-- 1 peter peter 19744364 Apr  9  2015
20150314_162153_AIDAluna_orgP.rw2
-rw-r--r-- 1 peter peter     5923 Jan 17 17:42
20150314_162153_AIDAluna_orgP.rw2.xmp
-rw-r--r-- 1 peter peter     5924 Jan 17 17:42
20150314_163131_AIDAluna_orgP.rw2.xmp
/home/peter/foto_xmp_ex.sh: 7: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 14: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")
peter@kubu2-lux:~/test$

[woenx]

This happens if the /shopt -s nullglob extglob;/ command is not recognised in
the script. I guess that for some reason, the script is not using bash.

Include this line at the beginning of the script, to identify which shell
you're using:



For instance, if your script uses /zsh /instead of /bash/, the command
should be:





--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[woenx]

Dammit, the message lost all its formatting.

Let me repeat it:

This happens if the shopt -s nullglob extglob; command is not recognised in the script. I guess that for some reason, the script is not using bash.

Include this line at the beginning of the script, to identify which shell you're using:

ps -p $$

For instance, if your script uses zsh instead of bash, the command should be:

setopt nullglob extglob;

---

Sent from the digikam-users mailing list archive at Nabble.com.

[Peter Mc Donough]

Am 09.03.2018 um 00:52 schrieb woenx:
> This happens if the /shopt -s nullglob extglob;/ command is not recognised in
> the script. I guess that for some reason, the script is not using bash.
>
> Include this line at the beginning of the script, to identify which shell
> you're using:

??

> For instance, if your script uses /zsh /instead of /bash/, the command
> should be:

Sorry, what line and what command ?

Peter

[woenx]

Argg, I'm sorry, the message lost its format and misses half the content. I
resent it again but now it says that it's "pending moderation", so I guess
it'll take a while to be published.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html

[Peter Mc Donough]

Am 09.03.2018 um 01:01 schrieb woenx:
> Argg, I'm sorry, the message lost its format and misses half the content. I
> resent it again but now it says that it's "pending moderation", so I guess
> it'll take a while to be published.
>

No problem, Google helped

the shell is bash.

peter@kubu2-lux:~$ ps -p $$
   PID TTY          TIME CMD
  9108 pts/1    00:00:00 bash
peter@kubu2-lux:~$


Peter

[Peter Mc Donough]

Am 09.03.2018 um 00:57 schrieb woenx:

> Dammit, the message lost all its formatting.
>
> Let me repeat it:
>
> This happens if the /shopt -s nullglob extglob;/ command is not recognised
> inthe script. I guess that for some reason, the script is not using bash.
>
> Include this line at the beginning of the script, to identify which
> shellyou're using:For instance, if your script uses /zsh /instead of /bash/,
> the commandshould be:
>

Still, lost in transmission!

but the shell is

peter@kubu2-lux:~$ echo $0
/bin/bash
peter@kubu2-lux:~$

and I included in the script

#!/bin/bash
# and
ls -l
# which listed the files.

cu
Peter