Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

OpenID

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged duplicate

Hide

News

2011-03-12 - Confoo 2011 presentation: Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands: To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner: Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed: Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.

Hide

Functions

Hide

Credits

Site by David Winterbottom (user root).

Commands tagged duplicate

Terminal - Commands tagged duplicate - 8 results

sort namesd.txt | uniq ?cd

2012-06-26 19:23:58

User: ankush108

Functions: sort uniq

Tags: sort duplicate unique uni

Up
Down

Show duplicate lines in a file

The following displays only the entries that are duplicates.

awk -F":" '!list[$3]++{print $3}' /etc/passwd

2012-01-24 12:47:52

User: richard

Functions: awk

Tags: awk uniq duplicate uid /etc/passwd

Up
Down

Find duplicate UID in /etc/passwd

You can use only awk

Comments (2) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

awk -F: '{print $3}' /etc/passwd | sort |uniq -d

2012-01-17 11:16:35

User: ultips

Functions: awk sort uniq

Tags: uniq duplicate uid /etc/passwd

Up
Down

Find duplicate UID in /etc/passwd

Detect duplicate UID in you /etc/passwd (or GID in /etc/group file).

Duplicate UID is often forbidden for it can be a security breach.

find . -type f |sed "s#.*/##g" |sort |uniq -c -d

2010-02-17 11:59:54

User: shadycraig

Functions: find sed sort uniq

Tags: file duplicate files

Up
Down

Count and show duplicate file names

Useful for C projects where header file names must be unique (e.g. when using autoconf/automake), or when diagnosing if the wrong header file is being used (due to dupe file names)

find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

2010-01-28 09:45:29

User: 2chg

Functions: find md5sum sort uniq xargs

Tags: find duplicate files md5sum VCS subversion

Up
Down

Find Duplicate Files, excluding .svn-directories (based on size first, then MD5 hash)

Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv":

find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

(cd /source/dir ; tar cv .)|(cd /dest/dir ; tar xv)

2009-07-19 10:31:13

User: marssi

Functions: cd tar

Tags: sed awk du duplicate tar directory pv duplicate directories duplicate directory

-11

Up
Down

Duplicate a directory tree using tar and pipes

the f is for file and - stdout, This way little shorter.

I Like copy-directory function It does the job but looks like SH**, and this doesn't understand folders with whitespaces and can only handle full path, but otherwise fine,

function copy-directory () { ; FrDir="$(echo $1 | sed 's:/: :g' | awk '/ / {print $NF}')" ; SiZe="$(du -sb $1 | awk '{print $1}')" ; (cd $1 ; cd .. ; tar c $FrDir/ )|pv -s $SiZe|(cd $2 ; tar x ) ; }

(cd /source/dir ; tar cvf - .)|(cd /dest/dir ; tar xvpf -)

2009-07-14 20:03:23

User: tkunz

Functions: cd tar

Tags: duplicate directory duplicate directories duplicate directory

-3

Up
Down

Duplicate a directory tree using tar and pipes

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

cat file.txt | sort | uniq -dc

2009-03-21 18:15:14

User: Vadi

Functions: cat sort uniq

Tags: sort awk uniq duplicate occur

Up
Down

Display duplicated lines in a file

Displays the duplicated lines in a file and their occuring frequency.

Comments (3) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate