Find Duplicate Files (based on size first, then MD5 hash)

• Find Duplicate Files (based on size first, then MD5 hash)

  If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

  This is sample output - yours may be different.

  
  

  24
  fdupes -r .
  Vilemirth · 2011-02-19 17:02:30 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash)

  Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

  This is sample output - yours may be different.

  
  

  18
  find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
  infinull · 2009-08-04 07:05:12 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files, excluding .svn-directories (based on size first, then MD5 hash)

  Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored. Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv": find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate Show Sample Output

  This is sample output - yours may be different.

  [...]
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/darkblue_orange/img/b_relations.png
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/original/img/b_relations.png
  
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/darkblue_orange/img/b_deltbl.png
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/original/img/b_deltbl.png
  
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/darkblue_orange/img/s_lang.png
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/original/img/s_lang.png
  
  f63a5ad833147eeb94adb4496ddbec41  ./themes/darkblue_orange/img/s_theme.png
  f63a5ad833147eeb94adb4496ddbec41  ./themes/original/img/s_theme.png
  
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/darkblue_orange/img/bd_drop.png
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/original/img/bd_drop.png
  
  f95d66c11bfed9198d13a278269c32b2  ./themes/darkblue_orange/img/s_loggoff.png
  f95d66c11bfed9198d13a278269c32b2  ./themes/original/img/s_loggoff.png
  [...]
  

  
  

  2
  find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
  2chg · 2010-01-28 09:45:29 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash) -- For Mac OS X

  This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

  This is sample output - yours may be different.

  
  

  2
  find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"
  noahspurrier · 2012-01-14 08:54:12 4
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on size first, then MD5 hash)

  Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on: find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate The new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

  This is sample output - yours may be different.

  
  

  2
  find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate
  fobos3 · 2014-10-19 02:00:55 1
  • Submit alternative
  • Report as malicious
  • Report as a duplicate

awsome, much faster then fdupes.

dakunesu · 882 weeks and 1 day ago

Isn't the -D redundant?

dennisw · 880 weeks and 5 days ago

yes it is... thanks for noticing, I fixed it.

grokskookum · 880 weeks and 5 days ago

How can you mass delete these files once they're found? (I'd like to keep one of them)

matthewbauer · 876 weeks and 4 days ago

you might want to look at fdupes or fslint in order to help with hardlinking / deleting, etc... my command is really just a quick hack to list them.

grokskookum · 876 weeks and 4 days ago

Fantastic, man. this is truly great.

oernii3 · 805 weeks and 3 days ago

"find -type" doesn?t work on Mac OS X.

ELV1S · 744 weeks and 1 day ago

can filename comparison be added as a first step to the first solution given? find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate seems to me checking filename first could speed things up. if two files lack the same filename then in many cases i would not consider them a dupe. thanks

johnywhy · 736 weeks and 2 days ago

Hi, I would like if it is any way of find duplicates of a given file (not all duplicates on the fs) maybe searching directly by md5. It would be great for me.

tia · 697 weeks and 1 day ago

Does anyone know a way to find duplicate files between 2 volumes based only on size and name?

d0g · 654 weeks and 3 days ago

kos omkon 3a hal exemple

mahmoud · 636 weeks and 6 days ago

Here is the tool to find and delete duplicate files "DuplicateFilesDeleter"

ivanden · 630 weeks and 6 days ago

Should be noted that md5sum is not a collision free algorithm, so there's a probability (OK, a very small probability) that the above commands will reports files as dupes even when they're not. Should use a sha1sum if you're paranoid.

befyber · 607 weeks and 1 day ago

Just a thot: Sequence of detection 1. size 2. MAX=say 100Mb or 10 Mb (a) Full MD5 for files (b) MD5 only the first MAX bytes for files > MAX e.g. MD5 (dd if=file of=/tmp/file count=200000) 3. Full MD5 or SHA1 for files found in 2(b) --- Wud b nice for media collection. Any takers??

Atanu · 602 weeks and 1 day ago

Duplicatefilesdeleter is best removal for duplicate files

Ketan · 576 weeks ago

I use Duplicate Files Deleter as it is very effective. It is 100% accurate and performs the scan quickly.

rewanya · 553 weeks ago

Please use Duplicate files deleter, it is very simple to use. But make it sure to keep the important files in backup. Thanks everybody for giving me your valuable times.

TinaRodrigo · 521 weeks and 1 day ago

ffgg

chenlixiang · 471 weeks and 6 days ago

You must be signed in to comment.