Find Duplicate Files (based on size first, then MD5 hash)

• Find Duplicate Files (based on size first, then MD5 hash)

  If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

  This is sample output - yours may be different.

  
  

  24
  fdupes -r .
  Vilemirth · 2011-02-19 17:02:30 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash)

  Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

  This is sample output - yours may be different.

  
  

  18
  find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
  infinull · 2009-08-04 07:05:12 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files, excluding .svn-directories (based on size first, then MD5 hash)

  Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored. Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv": find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate Show Sample Output

  This is sample output - yours may be different.

  [...]
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/darkblue_orange/img/b_relations.png
  f2e6bb247f110dcab63b4d38ff7b2dee  ./themes/original/img/b_relations.png
  
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/darkblue_orange/img/b_deltbl.png
  f5309bd2a2fc5e512a0cc38ac6f10c09  ./themes/original/img/b_deltbl.png
  
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/darkblue_orange/img/s_lang.png
  f60bfbb7ce218a55650c1abbbbee06ae  ./themes/original/img/s_lang.png
  
  f63a5ad833147eeb94adb4496ddbec41  ./themes/darkblue_orange/img/s_theme.png
  f63a5ad833147eeb94adb4496ddbec41  ./themes/original/img/s_theme.png
  
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/darkblue_orange/img/bd_drop.png
  f6ae61146ce3de8fa11b9e84e086bd04  ./themes/original/img/bd_drop.png
  
  f95d66c11bfed9198d13a278269c32b2  ./themes/darkblue_orange/img/s_loggoff.png
  f95d66c11bfed9198d13a278269c32b2  ./themes/original/img/s_loggoff.png
  [...]
  

  
  

  2
  find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
  2chg · 2010-01-28 09:45:29 0
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on MD5 hash) -- For Mac OS X

  This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

  This is sample output - yours may be different.

  
  

  2
  find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"
  noahspurrier · 2012-01-14 08:54:12 4
  • Submit alternative
  • Report as malicious
  • Report as a duplicate
• Find Duplicate Files (based on size first, then MD5 hash)

  Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on: find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate The new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

  This is sample output - yours may be different.

  
  

  2
  find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate
  fobos3 · 2014-10-19 02:00:55 1
  • Submit alternative
  • Report as malicious
  • Report as a duplicate

awsome, much faster then fdupes.

dakunesu · 867 weeks and 3 days ago

Isn't the -D redundant?

dennisw · 866 weeks ago

yes it is... thanks for noticing, I fixed it.

grokskookum · 866 weeks ago

How can you mass delete these files once they're found? (I'd like to keep one of them)

matthewbauer · 861 weeks and 6 days ago

you might want to look at fdupes or fslint in order to help with hardlinking / deleting, etc... my command is really just a quick hack to list them.

grokskookum · 861 weeks and 6 days ago

Fantastic, man. this is truly great.

oernii3 · 790 weeks and 4 days ago

"find -type" doesn?t work on Mac OS X.

ELV1S · 729 weeks and 3 days ago

can filename comparison be added as a first step to the first solution given? find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate seems to me checking filename first could speed things up. if two files lack the same filename then in many cases i would not consider them a dupe. thanks

johnywhy · 721 weeks and 3 days ago

Hi, I would like if it is any way of find duplicates of a given file (not all duplicates on the fs) maybe searching directly by md5. It would be great for me.

tia · 682 weeks and 3 days ago

Does anyone know a way to find duplicate files between 2 volumes based only on size and name?

d0g · 639 weeks and 5 days ago

kos omkon 3a hal exemple

mahmoud · 622 weeks and 1 day ago

Here is the tool to find and delete duplicate files "DuplicateFilesDeleter"

ivanden · 616 weeks and 1 day ago

Should be noted that md5sum is not a collision free algorithm, so there's a probability (OK, a very small probability) that the above commands will reports files as dupes even when they're not. Should use a sha1sum if you're paranoid.

befyber · 592 weeks and 3 days ago

Just a thot: Sequence of detection 1. size 2. MAX=say 100Mb or 10 Mb (a) Full MD5 for files (b) MD5 only the first MAX bytes for files > MAX e.g. MD5 (dd if=file of=/tmp/file count=200000) 3. Full MD5 or SHA1 for files found in 2(b) --- Wud b nice for media collection. Any takers??

Atanu · 587 weeks and 2 days ago

Duplicatefilesdeleter is best removal for duplicate files

Ketan · 561 weeks and 1 day ago

I use Duplicate Files Deleter as it is very effective. It is 100% accurate and performs the scan quickly.

rewanya · 538 weeks and 2 days ago

Please use Duplicate files deleter, it is very simple to use. But make it sure to keep the important files in backup. Thanks everybody for giving me your valuable times.

TinaRodrigo · 506 weeks and 3 days ago

ffgg

chenlixiang · 457 weeks and 1 day ago

You must be signed in to comment.