OSX Command line Find Duplicate Filenames with different extensions

Question

I've gone through all the similar questions + installed few command line utilities like duff but with no sucess.

In my Images directory, I have images with same name but with different extensions (jpg & png). I just want to print the filenames of duplicate filename with different extensions. (e.g. foo.jpg & foo.png) Or at least one file name (foo)

So far I tried these methods

find . -exec bash -c 'basename "0ドル" ".${0##*.}"' {} \; | sort | uniq
find . -type f \( -name "*.jpg" -o -name "*.png" \)

Most of these commands returns me the either Nothing OR All the files OR unique filenames but not the Duplicate ones.

Answer 1

I would suggest a modification of your second command:

find . -type f -name "*.jpg" | \
 while read -r f; do [ -e "${f%.jpg}.png" ] && echo "${f%.jpg}"; done

This finds all .jpg files and checks if the corresponding .png file exists, and displays the full path without the extensions.

Note that if there are a lot fewer .png files it will be more efficient to search for these and check for the corresponding .jpg files.

[Tested with bash on Ubuntu 18.04.1.]

• Naveed Abbas: find: illegal option -- t
• AFH: @ToughGuy - I didn't use any -t option, but on Linux find . is implicit if no directory is given, and maybe it isn't on OSX. I have added the . and quoted the file name in the echo in case you have some some odd file names which could appear as options.
• xenoid: -type is normally understood by OSX find according to this
• Kamil Maciorowski: Yes, find on Mac requires . or any path, POSIX requires it. Here on Super User I advise never omit . for the sake of portability.
• AFH: @KamilMaciorowski - Thanks: there are things I have just got used to doing. Without a Mac, I can't check every option of every command in case it's not compatible.

Answer 2

I accept and appreciate the answer. Meanwhile I got this python script that worked somewhat near to what I was looking for. I tried to find the source but couldn't find it in 100's of tabs I searched.

#!/usr/bin/env python
# Syntax: duplicates.py DIRECTORY
import os, sys
top = sys.argv[1]
d = {}
for root, dirs, files in os.walk(top, topdown=False):
 for name in files:
 fn = os.path.join(root, name)
 basename, extension = os.path.splitext(name)
 basename = basename.lower() # ignore case
 if basename in d:
 print(d[basename])
 print(fn)
 else:
 d[basename] = fn

Save this file as duplicates.py and give it rights and then execute it on the folder.

./duplicates.py Images

• Hogstrom: Matching on name and alternate suffixes is useful but it doesn't guarantee the files are actual duplicates. You would need to include comparison of file sizes and if they match actually compare the file contents.
• Naveed Abbas: @Hogstorm Good idea but my requirement was different. There are plenty of utilities apps for duplicate finder but none was promising as I needed the output in a text file.

Answer 3

You have to use uniq -c to get the counts, then reverse-sort at the end to list the duplicates first. At the end awk is used to filter for lines that start with 2 or higher.

$ find . -type f -exec sh -c 'basename ${0%.*}' {} \; | sort | uniq -c | sort -r | awk 'int(1ドル)>=2'
 2 foo

Here, %.* strips the extension, so foo.x.y becomes foo.x.

Instead of a simple find . -type f, which would find all files, you could also filter for *.jpg or *.png files like in your second command.

• Naveed Abbas: Certainly it gave me a long list of all the files with the count of duplicates. Can I get only the duplicates? (e.g. foo which is used twice or thrice)