Find Duplicate Files (based on size first, then MD5 hash)

• Find Duplicate Files (based on size first, then MD5 hash)

  If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

  This is sample output - yours may be different.

  
  

  24
  fdupes -r .
  Vilemirth · 2011-02-19 17:02:30 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash)

  Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

  This is sample output - yours may be different.

  
  

  18
  find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
  infinull · 2009-08-04 07:05:12 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files, excluding .svn-directories (based on size first, then MD5 hash)

  Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored. Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv": find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate Show Sample Output

  This is sample output - yours may be different.

  [...]
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/darkblue_orange/img/b_relations.png
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/original/img/b_relations.png
  
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/darkblue_orange/img/b_deltbl.png
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/original/img/b_deltbl.png
  
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/darkblue_orange/img/s_lang.png
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/original/img/s_lang.png
  
  f63a5ad833147eeb94adb4496ddbec41  ./themes/darkblue_orange/img/s_theme.png
  f63a5ad833147eeb94adb4496ddbec41  ./themes/original/img/s_theme.png
  
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/darkblue_orange/img/bd_drop.png
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/original/img/bd_drop.png
  
  f95d66c11bfed9198d13a278269c32b2  ./themes/darkblue_orange/img/s_loggoff.png
  f95d66c11bfed9198d13a278269c32b2  ./themes/original/img/s_loggoff.png
  [...]
  

  
  

  2
  find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
  2chg · 2010-01-28 09:45:29 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash) -- For Mac OS X

  This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

  This is sample output - yours may be different.

  
  

  2
  find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"
  noahspurrier · 2012-01-14 08:54:12 4
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on size first, then MD5 hash)

  Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on: find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate The new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

  This is sample output - yours may be different.

  
  

  2
  find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate
  fobos3 · 2014-10-19 02:00:55 1
  • Submit alternative
  • Report as malicious
  • Report as a duplicate

awsome, much faster then fdupes.

dakunesu · 852 weeks and 4 days ago

Isn't the -D redundant?

dennisw · 851 weeks and 1 day ago

yes it is... thanks for noticing, I fixed it.

grokskookum · 851 weeks and 1 day ago

How can you mass delete these files once they're found? (I'd like to keep one of them)

matthewbauer · 847 weeks ago

you might want to look at fdupes or fslint in order to help with hardlinking / deleting, etc... my command is really just a quick hack to list them.

grokskookum · 847 weeks ago

Fantastic, man. this is truly great.

oernii3 · 775 weeks and 5 days ago

"find -type" doesn?t work on Mac OS X.

ELV1S · 714 weeks and 4 days ago

can filename comparison be added as a first step to the first solution given? find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate seems to me checking filename first could speed things up. if two files lack the same filename then in many cases i would not consider them a dupe. thanks

johnywhy · 706 weeks and 4 days ago

Hi, I would like if it is any way of find duplicates of a given file (not all duplicates on the fs) maybe searching directly by md5. It would be great for me.

tia · 667 weeks and 3 days ago

Does anyone know a way to find duplicate files between 2 volumes based only on size and name?

d0g · 624 weeks and 5 days ago

kos omkon 3a hal exemple

mahmoud · 607 weeks and 2 days ago

Here is the tool to find and delete duplicate files "DuplicateFilesDeleter"

ivanden · 601 weeks and 1 day ago

Should be noted that md5sum is not a collision free algorithm, so there's a probability (OK, a very small probability) that the above commands will reports files as dupes even when they're not. Should use a sha1sum if you're paranoid.

befyber · 577 weeks and 4 days ago

Just a thot: Sequence of detection 1. size 2. MAX=say 100Mb or 10 Mb (a) Full MD5 for files (b) MD5 only the first MAX bytes for files > MAX e.g. MD5 (dd if=file of=/tmp/file count=200000) 3. Full MD5 or SHA1 for files found in 2(b) --- Wud b nice for media collection. Any takers??

Atanu · 572 weeks and 3 days ago

Duplicatefilesdeleter is best removal for duplicate files

Ketan · 546 weeks and 2 days ago

I use Duplicate Files Deleter as it is very effective. It is 100% accurate and performs the scan quickly.

rewanya · 523 weeks and 2 days ago

Please use Duplicate files deleter, it is very simple to use. But make it sure to keep the important files in backup. Thanks everybody for giving me your valuable times.

TinaRodrigo · 491 weeks and 3 days ago

ffgg

chenlixiang · 442 weeks and 2 days ago

You must be signed in to comment.