Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

OpenID

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged uniq

Hide

News

2011-03-12 - Confoo 2011 presentation: Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands: To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner: Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed: Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.

Hide

Functions

Hide

Credits

Site by David Winterbottom (user root).

Commands tagged uniq

Terminal - Commands tagged uniq - 44 results

tr A-Z a-z | tr -cs a-z '\n' | sort | uniq -c

2010-10-19 22:49:13

User: putnamhill

Functions: sort tr uniq

Tags: sort uniq tr

Up
Down

Generate list of words and their frequencies in a text file.

Comments (3) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

ls | perl -lne '++$x{lc $1} if /[.](.+)$/ }{ print for keys %x'

2010-08-13 20:05:15

User: recursiverse

Functions: ls perl

Tags: sort uniq find extensions

-3

Up
Down

list all file extensions in a directory

All with only one pipe. Should be much faster as well (sort is slow). Use find instead of ls for recursion or reliability.

Edit: case insensitive

ls -Xp /path/to/dir | grep -Eo "\.[^/]+$" | uniq

2010-08-12 16:32:54

User: karpoke

Functions: grep ls

Tags: uniq ls grep

Up
Down

list all file extensions in a directory

If we want files with more than one extension, like .tar.gz, only appear the latest, .gz:

ls -Xp /path/to/dir | grep -Eo "\.[^./]+$" | uniq

find /path/to/dir -type f -name '*.*' | sed 's@.*/.*\.@.@' | sort | uniq

2010-08-12 15:48:54

User: putnamhill

Functions: find sed sort

Tags: sort uniq find extensions

Up
Down

list all file extensions in a directory

If your grep doesn't have an -o option, you can use sed instead.

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

find /path/to/dir -type f | grep -o '\.[^./]*$' | sort | uniq

2010-08-12 14:32:48

User: dooblem

Functions: find grep sort

Tags: sort uniq find grep extensions

Up
Down

list all file extensions in a directory

Just a little simplification.

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

alias sorth='sort --help|sed -n "/^ *-[^-]/s/^ *$-[^ ]* -[^ ]*$ *$.*$/\1:\2/p"|column -ts":"'

2010-06-10 21:30:31

User: AskApache

Functions: alias column sed

Tags: bash sort sed uniq alias column sorting reverse help askapache sh fields unique buffer

Up
Down

View advanced Sort options, Quick Reference Help Alias

Once you get into advanced/optimized scripts, functions, or cli usage, you will use the sort command alot. The options are difficult to master/memorize however, and when you use sort commands as much as I do (some examples below), it's useful to have the help available with a simple alias. I love this alias as I never seem to remember all the options for sort, and I use sort like crazy (much better than uniq for example).

# Sorts by file permissions

find . -maxdepth 1 -printf '%.5m %10M %p\n' | sort -k1 -r -g -bS 20%

00761 drwxrw---x ./tmp

00755 drwxr-xr-x .

00701 drwx-----x ./askapache-m

00644 -rw-r--r-- ./.htaccess

# Shows uniq history fast

history 1000 | sed 's/^[0-9 ]*//' | sort -fubdS 50%

exec bash -lxv

export TERM=putty-256color

Taken from my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html

pgrep -cu ioggstream

2010-05-21 10:53:57

User: ioggstream

Tags: sort uniq ps processes

Up
Down

processes per user counter

use Linux ;)

tail -n2000 /var/www/domains/*/*/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -n | awk '{ if ($1 > 20)print $1,$2}'

2010-05-10 19:08:37

User: allrightname

Functions: awk sort tail uniq

Tags: bash sort awk uniq tail IP apache

Up
Down

Count number of hits per IP address in last 2000 lines of apache logs and print the IP and hits if hits > 20

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

netstat -an | awk '/tcp/ {print $6}' | sort | uniq -c

2010-05-06 17:04:37

User: Kered557

Functions: awk netstat sort uniq

Tags: sort awk uniq netstat

Up
Down

Count TCP States From Netstat

Counts TCP states from Netstat and displays in an ordered list.

grep <something> logfile | cut -c2-18 | uniq -c

2010-04-29 11:26:09

User: buzzy

Functions: cut grep uniq

Tags: uniq grep cut

Up
Down

Count occurrences per minute in a log file

The cut should match the relevant timestamp part of the logfile, the uniq will count the number of occurrences during this time interval.

tail -f access_log | cut -c2-21 | uniq -c

2010-04-29 11:16:54

User: buzzy

Functions: cut tail uniq

Tags: uniq tail cut

Up
Down

Realtime apache hits per second

Change the cut range for hits per 10 sec, minute and so on... Grep can be used to filter on url or source IP.

ps hax -o user | sort | uniq -c

2010-04-29 10:43:03

User: buzzy

Functions: ps sort uniq

Tags: sort uniq ps processes

Up
Down

processes per user counter

awk is evil!

grep current_state= /var/log/nagios/status.dat|sort|uniq -c|sed -e "s/[\t ]*$[0-9]*$.*current_state=$[0-9]*$/\2:\1/"|tr "\n" " "

2010-03-11 06:04:14

User: c3w

Functions: grep sed tr uniq

Tags: netcat sort sed uniq grep tr nagios

Up
Down

display a one-liner of current nagios exit statuses. great with netcat/irccat

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

curl -s http://tinyurl.com/create.php?url=http://<website.url>/ | sed -n 's/.*$http:\/\/tinyurl.com\/[a-z0-9][a-z0-9]*$.*/\1/p' | uniq

2009-08-30 07:55:48

User: amaymon

Functions: sed

Tags: sed uniq curl

Up
Down

Creating shortened URLs from the command line

wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}'

2009-05-04 16:00:39

User: alperyilmaz

Functions: awk grep perl sed sort tr uniq wget

Tags: sort perl sed awk uniq wget tr

-4

Up
Down

count occurences of each word in novel David Copperfield

This command might not be useful for most of us, I just wanted to share it to show power of command line.

Download simple text version of novel David Copperfield from Poject Gutenberg and then generate a single column of words after which occurences of each word is counted by sort | uniq -c combination.

This command removes numbers and single characters from count. I'm sure you can write a shorter version.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

2009-03-28 21:02:26

User: tiagofischer

Functions: awk cut netstat sort uniq

Tags: sort awk uniq netstat cut

Up
Down

Number of open connections per ip.

Here is a command line to run on your server if you think your server is under attack. It prints our a list of open connections to your server and sorts them by amount.

BSD Version:

netstat -na |awk '{print $5}' |cut -d "." -f1,2,3,4 |sort |uniq -c |sort -nr

cat file.txt | sort | uniq -dc

2009-03-21 18:15:14

User: Vadi

Functions: cat sort uniq

Tags: sort awk uniq duplicate occur

Up
Down

Display duplicated lines in a file

Displays the duplicated lines in a file and their occuring frequency.

Comments (3) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less

2009-03-03 13:45:56

User: dbart

Functions: awk sort uniq zgrep

Tags: ssh sort zgrep awk uniq

Up
Down

Check for login failures and summarize

This command checks for the number of times when someone has tried to login to your server and failed. If there are a lot, then that user is being targeted on your system and you might want to make sure that user either has remote logins disabled, or has a strong password, or both. If your output has an "invalid" line, it is a summary of all logins from users that don't exist on your system.

netstat -alpn | grep :80 | awk '{print $4}' |awk -F: '{print $(NF-1)}' |sort | uniq -c | sort -n

2009-02-19 04:59:32

User: nitins

Functions: awk grep netstat sort uniq

Tags: sort awk uniq netstat ddos

-2

Up
Down

Checking total connections to each Ip inserver

Useful to check DDoS attacks on servers.

< 1 2