Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

OpenID
or
Sign in Register

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

Hide

News

2011-03-12 - Confoo 2011 presentation
Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands
To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner
Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed
Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.
Hide

Tags

Hide

Functions

Hide

Credits

Site by David Winterbottom (user root).

Commands using uniq

Commands using uniq from sorted by
Terminal - Commands using uniq - 172 results
netstat -lantp | grep -i establ | awk -F/ '{print $2}' | uniq | sort
2009-09-19 13:54:36
User: ktoso
Functions: awk grep netstat uniq
Tags: bash grep netstat internet
-1
Up
Down
Show apps that use internet connection at the moment.

Can be used to discover what programms create internet traffic. Skip the part after awk to get more details.

Has anyone an idea why the uniq doesn't work propperly here (see sample output)?

Show sample output | Comments (2) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
awk '{print $1}' /var/log/httpd/access_log | sort | uniq -c | sort -rnk1 | head -n 10
2009-08-30 15:32:26
User: flip387
Functions: awk head sort uniq
3
Up
Down
Top 10 requestors by IP address from Apache/NCSA Logs
Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
2009-08-04 07:05:12
User: infinull
Functions: cut find md5sum sort uniq
17
Up
Down
Find Duplicate Files (based on MD5 hash)

Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
netstat -ant | grep :80 | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n
2009-07-31 18:28:16
User: garchacesaug
Functions: awk grep netstat sort uniq
3
Up
Down
Show in a web server, running in the port 80, how many ESTABLISHED connections by ip it has.

The command could show you all conecctions if you skip "grep ESTABLISHED"

Show sample output | Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
svn ls -R | egrep -v -e "\/$" | xargs svn blame | awk '{print $2}' | sort | uniq -c | sort -r
2009-07-29 02:10:45
User: askedrelic
Functions: awk egrep ls sort uniq xargs
Tags: svn count
16
Up
Down
Prints total line count contribution per user for an SVN repository

I'm working in a group project currently and annoyed at the lack of output by my teammates. Wanting hard metrics of how awesome I am and how awesome they aren't, I wrote this command up.

It will print a full repository listing of all files, remove the directories which confuse blame, run svn blame on each individual file, and tally the resulting line counts. It seems quite slow, depending on your repository location, because blame must hit the server for each individual file. You can remove the -R on the first part to print out the tallies for just the current directory.

Show sample output | Comments (6) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
DD=`cat /etc/my.cnf | sed "s/#.*//g;" | grep datadir | tr '=' ' ' | gawk '{print $2;}'` && ( cd $DD ; find . -mindepth 2 | grep -v db\.opt | sed 's/\.\///g; s/\....$//g; s/\//./;' | sort | uniq | tr '/' '.' | gawk '{print "CHECK TABLE","`"$1"`",";";}' )
2009-07-25 03:42:31
User: atcroft
Functions: cd find gawk grep sed sort tr uniq
-1
Up
Down
Generate CHECK TABLE statements for all MySQL database tables on a server

This command will generate "CHECK TABLE `db_name.table_name` ;" statements for all tables present in databases on a MySQL server, which can be piped into the mysql command. (Can also be altered to perform OPTIMIZE and REPAIR functions.)

Tested on MySQL 4.x and 5.x systems in a Linux environment under bash.

Show sample output | Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
cat /var/log/secure.log | awk '{print substr($0,0,12)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'
2009-07-24 07:20:06
User: knassery
Functions: awk cat sort uniq
15
Up
Down
Show me a histogram of the busiest minutes in a log file:

Busiest seconds:

cat /var/log/secure.log | awk '{print substr($0,0,15)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'
Show sample output | Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
lsof |awk ' {if ( $0 ~ /home/) print substr($0, index($0,"/home") ) }'|cut -d / -f 1-4|sort|uniq -c|sort -bgr
2009-07-12 02:25:57
User: lucainvernizzi
Functions: awk cut sort uniq
2
Up
Down
find out which directories in /home have the most files currently open
Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
egrep -r '(render_message|multipart).*('`find app/views -name '*.erb' | grep mailer | sed -e 's/\..*//' -e 's/.*\///' | uniq | xargs | sed 's/ /|/g'`')' app/models
2009-06-12 18:53:29
User: foobarfighter
Functions: egrep grep sed uniq xargs
0
Up
Down
Finds all of the mailers being used in your rails app
Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
awk -F\" '{print $4}' *.log | grep -v "eviljaymz\|\-" | sort | uniq -c | awk -F\ '{ if($1>500) print $1,$2;}' | sort -n
2009-05-05 22:21:04
User: jaymzcd
Functions: awk grep sort uniq
1
Up
Down
Print summary of referers with X amount of occurances

This prints a summary of your referers from your logs as long as they occurred a certain number of times (in this case 500). The grep command excludes the terms, I add this in to remove results Im not interested in.

Show sample output | Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}'
2009-05-04 16:00:39
User: alperyilmaz
Functions: awk grep perl sed sort tr uniq wget
Tags: sort perl sed awk uniq wget tr
-4
Up
Down
count occurences of each word in novel David Copperfield

This command might not be useful for most of us, I just wanted to share it to show power of command line.

Download simple text version of novel David Copperfield from Poject Gutenberg and then generate a single column of words after which occurences of each word is counted by sort | uniq -c combination.

This command removes numbers and single characters from count. I'm sure you can write a shorter version.

Show sample output | Comments (8) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'
2009-04-27 22:02:19
User: knassery
Functions: awk grep netstat sort uniq
41
Up
Down
Graph # of connections for each hosts.

Written for linux, the real example is how to produce ascii text graphs based on a numeric value (anything where uniq -c is useful is a good candidate).

Show sample output | Comments (7) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
emerge -epv world | grep USE | cut -d '"' -f 2 | sed 's/ /\n/g' | sed '/[(,)]/d' | sed s/'*'//g | sort | uniq > use && grep ^- use | sed s/^-// | sed ':a;N;$!ba;s/\n/ /g' > notuse && sed -i /^-/d use && sed -i ':a;N;$!ba;s/\n/ /g' use
2009-04-19 07:54:08
User: Keruspe
Functions: cut grep sed sort uniq
Tags: Gentoo emerge USE
1
Up
Down
For Gentoo users : helping with USE / emerge

This command puts all the flags of the USE variable actually used by the packages you emerged to the file "use", and those which are unused but available to the file "notuse"

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
grep -r "mystring" . |uniq | cut -d: -f1 | xargs sed -i "s/mystring//"
2009-04-09 12:49:01
User: mohamednasr2000
Functions: cut grep sed uniq xargs
-3
Up
Down
remove exact phrase from multiple files

Linux : these script enable you to edit multiple files and remove exact phrase from multiple files

Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r
2009-04-05 13:40:56
User: kayowas
Functions: awk sort uniq
6
Up
Down
See most used commands

It will return a ranked list of your most commonly-entered commands using your command history

Show sample output | Comments (5) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
cat /var/log/secure | grep smtp | awk '{print $9}' | cut -f2 -d= | sort | uniq -c | sort -n | tail
2009-03-30 15:49:54
User: empulse
Functions: awk cat cut grep sort uniq
-2
Up
Down
List SMTP connections by host

Searches /var/log/secure for smtp connections then lists these by number of connections made and hosts.

Comments (2) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
cat /var/log/secure | grep sshd | grep Failed | sed 's/invalid//' | sed 's/user//' | awk '{print $11}' | sort | uniq -c | sort -n
2009-03-30 15:48:24
User: empulse
Functions: awk cat grep sed sort sshd uniq
8
Up
Down
Find brute force attempts on SSHd

Searches the /var/log/secure log file for Failed and/or invalid user log in attempts.

Show sample output | Comments (7) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
2009-03-28 21:02:26
User: tiagofischer
Functions: awk cut netstat sort uniq
Tags: sort awk uniq netstat cut
14
Up
Down
Number of open connections per ip.

Here is a command line to run on your server if you think your server is under attack. It prints our a list of open connections to your server and sorts them by amount.

BSD Version:

netstat -na |awk '{print $5}' |cut -d "." -f1,2,3,4 |sort |uniq -c |sort -nr
Show sample output | Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
cat file.txt | sort | uniq -dc
2009-03-21 18:15:14
User: Vadi
Functions: cat sort uniq
Tags: sort awk uniq duplicate occur
1
Up
Down
Display duplicated lines in a file

Displays the duplicated lines in a file and their occuring frequency.

Comments (3) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
sort file1.txt | uniq > file2.txt
2009-03-14 20:24:06
User: rain
Functions: sort uniq
Tags: bash Linux
3
Up
Down
To have only unique lines in a file
Comments (6) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
grep Mar/2009 /var/log/apache2/access.log | awk '{ print $1 }' | sort -n | uniq -c | sort -rn | head
2009-03-10 16:46:19
User: allinurl
Functions: awk grep sort uniq
4
Up
Down
grep apache access.log and list IP's by hits and date - sorted
Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less
2009-03-03 13:45:56
User: dbart
Functions: awk sort uniq zgrep
Tags: ssh sort zgrep awk uniq
7
Up
Down
Check for login failures and summarize

This command checks for the number of times when someone has tried to login to your server and failed. If there are a lot, then that user is being targeted on your system and you might want to make sure that user either has remote logins disabled, or has a strong password, or both. If your output has an "invalid" line, it is a summary of all logins from users that don't exist on your system.

Show sample output | Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
gunzip -c /var/log/auth.log.*.gz | cat - /var/log/auth.log /var/log/auth.log.0 | grep "Invalid user" | awk '{print $8;}' | sort | uniq -c | less
2009-03-03 04:26:57
User: eanx
Functions: awk cat grep gunzip sort uniq
8
Up
Down
Quickly find a count of how many times invalid users have attempted to access your system
Comments (6) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
cat file1.txt | uniq > file2.txt
2009-03-01 05:10:10
User: olafrv
Functions: cat uniq
-3
Up
Down
Eliminate duplicate lines on a file
Comments (4) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate
find . -type f -name "*.java" -print0 | xargs -0 -n 1 svn blame | sed -n 's/^[^a-z]*\([a-z]*\).*$/\1/p' | sort | uniq -c | sort -n
2009-02-23 22:24:41
User: Parker
Functions: find sed sort uniq xargs
2
Up
Down
sort through source to find most common authors
Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate