Finding duplicate File names in a directory tree
Just a little one liner. Essentially I have a nested file system structure with a lot of subfolders and I wanted to know if there are duplicate file names.
This does the trick for me:
find . -iname "*.txt" | xargs -I {} basename {} | uniq -d -i
So what this does:
- Call find to find all files that match *.txt (case insensitive) in the current folder and all subfolders
- Call basename to turn /path/to/file.txt into file.txt. We have to use xargs since basename does not seem to be able to read from stdin
- Call uniq to check the list for duplicates. -d means that we only show duplicates (default is to show unique values) and -i performs a case insensitive comparison
Sidenote: As much as I love working with Windows, it just does not stand a chance against a long chain of simple, small, well-defined UNIX tools that are piped together. As much as PowerShell is a step in the right direction, it's not the same because the philosophy of having tons of very limited and specialized tools doesn't seem to exist in the Windows universe - most command line tools do too much. Also, UNIX has a 30 year head start against PowerShell.
January 26th, 2011 in
UNIX Shell | tags: mac os x
Just a quick correction:
the 'uniq' command only compares continuous duplicates (matches found directly next to each other). Therefore, you need to add a 'sort' command before calling 'uniq'. The end-product looks like this:
find . -iname "*.txt" | xargs -I {} basename {} | sort | uniq -d -i
ls -include *.txt -recurse | group name | ?{$_.count -gt 1}
The FIND code resulted in an error on error on OSX 10.7
The error message is "xargs: unterminated quote"
At a guess, the might be caused by spaces in file name or directory names. I tried to modify the code to allow for spaces but couldnt get it to work.
If you know how to fix this error, please post an update. Thank you
P.S I tried Mirko's code, but it did not work on OSX 10.7 for me