What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Universal configuration monitoring and system of record for IT.

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:



May 19, 2015 - A Look At The New Commandlinefu
I've put together a short writeup on what kind of newness you can expect from the next iteration of clfu. Check it out here.
March 2, 2015 - New Management
I'm Jon, I'll be maintaining and improving clfu. Thanks to David for building such a great resource!

Top Tags





Maintained by Jon H.

Site originally by David Winterbottom (user root).

Psst. Open beta.

Wow, didn't really expect you to read this far down. The latest iteration of the site is in open beta. It's a gentle open beta-- not in prime-time just yet. It's being hosted over at UpGuard (link) and you are more than welcome to give it a shot. Couple things:

  • » The open beta is running a copy of the database that will not carry over to the final version. Don't post anything you don't mind losing.
  • » If you wish to use your user account, you will probably need to reset your password.
Your feedback is appreciated via the form on the beta page. Thanks! -Jon & CLFU Team

Find Duplicate Files (based on MD5 hash)

Terminal - Find Duplicate Files (based on MD5 hash)
find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
2009-08-04 07:05:12
User: infinull
Functions: cut find md5sum sort uniq
Find Duplicate Files (based on MD5 hash)

Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.


There are 9 alternatives - vote for the best!

Terminal - Alternatives
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
2009-09-21 00:24:14
User: syssyphus
Functions: find md5sum sort uniq xargs

This dup finder saves time by comparing size first, then md5sum, it doesn't delete anything, just lists them.

fdupes -r .
2011-02-19 17:02:30
User: Vilemirth
Tags: xargs parallel

If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
2010-01-28 09:45:29
User: 2chg
Functions: find md5sum sort uniq xargs

Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv":

find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/"
2012-01-14 08:54:12
User: noahspurrier
Functions: find sed sort uniq

This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate
2014-10-19 02:00:55
User: fobos3
Functions: cut find md5sum sort uniq xargs

Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on:

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

The new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

Know a better way?

If you can do better, submit your command here.

What others think

http://www.pixelbeat.org/fslint/ fslint is a neat tool that does that and more, if you feel like a GUI.

Comment by bwoodacre 393 weeks and 4 days ago

[url=http://www.longchampoutletstore.org/][b]Longchamp Store[/b][/url]

[url=http://www.nfljerseys.us/][b]NFL Jerseys[/b][/url]

[url=http://www.yeezyboost350.us.com/][b]Yeezy Boost 350[/b][/url]

[url=http://www.jordan4.us/][b]Jordan 4[/b][/url]

[url=http://www.nikeoutletsstore.com/][b]Nike Outlets[/b][/url]

[url=http://www.longchampbags.us.com/][b]Longchamp Handbags[/b][/url]

[url=http://www.airmax2016.us.com/][b]Nike Air Max 2016[/b][/url]

[url=http://www.adidas-uk.org.uk/][b]Adidas UK[/b][/url]


[url=http://www.underarmouroutlet.us.com/][b]Under Armour Outlet[/b][/url]

[url=http://www.timberlanduk.org.uk/][b]Timberland UK[/b][/url]

[url=http://www.jordan12.us/][b]Jordan 12[/b][/url]

[url=http://www.poloralphlaurenoutlets.us.com/][b]Ralph Lauren Outlet[/b][/url]

[url=http://www.yeezy-shoes.us.com/][b]Yeezys Shoes[/b][/url]

[url=http://www.nikeoutlet.org.uk/][b]Nike Outlet[/b][/url]

[url=http://www.nikehuarache.us/][b]Nike Air Huarache[/b][/url]

[url=http://www.kedsshoesforwomen.com/][b]Keds Shoes[/b][/url]

[url=http://www.nikerosherun.us.com/][b]Nike Roshe Run[/b][/url]


[url=http://www.nikeairmax.us/][b]Air Max[/b][/url]

[url=http://www.oakleyoutlet-sale.us/][b]Oakley Outlet[/b][/url]

[url=http://www.adidasnmd.us.com/][b]Adidas NMD R1[/b][/url]

[url=http://www.cheap--jordanshoes.us.com/][b]Jordan Shoes[/b][/url]

[url=http://www.outlettoms.us/][b]Toms Shoes Outlet Online[/b][/url]

[url=http://www.ultraboostuncaged.us/][b]Adidas Ultra Boost Uncaged[/b][/url]

[url=http://www.raybansoutlet.us/][b]Ray Ban Outlet[/b][/url]

[url=http://www.mlb-jerseys.us/][b]MLB Jerseys[/b][/url]

[url=http://www.uggboots-outlets.com/][b]Ugg Boots[/b][/url]

[url=http://www.katespadeoutlets.us/][b]Kate Spade Outlet[/b][/url]

[url=http://www.uggbootsclearanceoutlet.us/][b]Ugg Clearance Sale Outlet[/b][/url]

[url=http://www.uggs-outletboots.us/][b]Ugg Outlet[/b][/url]

[url=http://www.timberland-outlet.us/][b]Timberland Outlet[/b][/url]

[url=http://www.ray-banoutlets.us/][b]Ray Ban Outlet[/b][/url]

[url=http://www.rayban-sunglasses.ca/][b]Ray Ban Sunglasses[/b][/url]

[url=http://www.adidas-nmd.org.uk/][b]Adidas NMD[/b][/url]

[url=http://www.katespadeoutlet.us.org/][b]Kate Spade Outlet[/b][/url]

[url=http://www.ralphlaurens.org.uk/][b]Ralph Laurens[/b][/url]

Comment by lovelin1987 13 weeks and 2 days ago

Your point of view

You must be signed in to comment.