Commands using uniq (255)
-
1find /path/to/dir -type f |sed 's/^.*\.//' |sort -f |uniq -i
-
We can get useful statistics from tcpdump with this simple command. Thanks "Babak Farrokhi" to teaching me this ;)
1tcpdump -nr capture.file | awk '{print }' | grep -oE '[0-9]{1,}.[0-9]{1,}.[0-9]{1,}.[0-9]{1,}' | sort | uniq -c | sort -n -
Useful for checking the number and state of TCP connections. Show Sample Output
1netstat -nt | awk '{print $6}' | sort | uniq -c | sort -n -k 1 -r -
Grabs the complete module list from CPAN, pulls the first column, ditches html lines, counts, ditches small namespaces.
1curl http://www.cpan.org/modules/01modules.index.html |awk '{print $1}'|grep -v "<"|sort|uniq -c|grep -v " +[0-9] " -
1grep -R Subject /var/spool/exim/input/ | sed s/^.*Subject:\ // | sort | uniq -c | sort -n > ~/email_sort.$(date +%m.%d.%y).txt
-
1sed -e 's/[;|][[:space:]]*/\n/g' .bash_history | cut --delimiter=' ' --fields=1 | sort | uniq --count | sort --numeric-sort --reverse | head --lines=20
-
See the summary. Show Sample Output
1lsof +c 15 | awk '{print $1}' | sort | uniq -c | sort -rn | head -
shows number of mysql bin log events (which are mysql server events) per minute, useful to check stress times postmortem Show Sample Output
1mysqlbinlog <logfiles> | grep exec | grep end_log_pos | cut -d' ' -f2- | cut -d: -f-2 | uniq -c -
1tcpdump -ntr NAME_OF_CAPTURED_FILE.pcap 'tcp[13] = 0x02 and dst port 80' | awk '{print $4}' | tr . ' ' | awk '{print $1"."$2"."$3"."$4}' | sort | uniq -c | awk ' {print $2 "\t" $1 }'
-
1git log | grep Date | awk '{print " : "$4" "$3" "$6}' | uniq -c
-
make usable on OSX with filenames containing spaces. note: will still break if filenames contain newlines... possible, but who does that?!
1svn ls -R | egrep -v -e "\/$" | tr '\n' '\0' | xargs -0 svn blame | awk '{print $2}' | sort | uniq -c | sort -nr -
parallel can be installed on your central node and can be used to run a command multiple times. In this example, multiple ssh connections are used to run commands. (-j is the number of jobs to run at the same time). The result can then be piped to commands to perform the "reduce" stage. (sort then uniq in this example). This example assumes "keyless ssh login" has been set up between the central node and all machines in the cluster. bashreduce may also do what you want. Show Sample Output
1parallel -j 50 ssh {} "ls" ::: host1 host2 hostn | sort | uniq -c -
1awk '{print $1}' ~/.bash_history | sort | uniq -c | sort -rn | head -n 10
-
Show's per IP of how many requests they did to the Apache webserver
1cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n -
The first sort is necessary for ips in a list to be actually unique.
1cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n -
Same as the rest, but handle IPv6 short IPs. Also, sort in the order that you're probably looking for. Show Sample Output
1netstat -ntu | awk ' $5 ~ /^(::ffff:|[0-9|])/ { gsub("::ffff:","",$5); print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr -
1grep "10/Sep/2013" access.log| cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'
-
displays a list of all file extensions in current directory and how many files there are of each type of extension in ascending order (case insensitive) Show Sample Output
1ls | tr [:upper:] [:lower:] | grep -oP '\.[^\.]+$' | sort | uniq -c | sort -
on some distro's you have to replace "BogoMIPS" with "bogomips". Show Sample Output
1cat /proc/cpuinfo | grep BogoMIPS | uniq | sed 's/^.*://g' | awk '{print($1 / 4) }' -
1git log origin/master..master --name-only --pretty="format:" | sort | uniq | xargs tar -rf mytarfile.tar
-
Sort netflow packet capture by unique connections excluding source port. Show Sample Output
1grep -o -P '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\:[0-9]{1,5}\s->\s{5}[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\:[0-9]{1,5}' <capture file> | tr -d ' ' | sed 's/:.....//g' | sort -n | uniq -c | sort -nr -
1/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | awk 'BEGIN { FS = " " } ; { print $12 }' | sed '/^$/d' | sort | uniq -c | sort -n
-
1netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
-
credit shall fall to this for non-gzipped version: https://gist.github.com/marcanuy/a08d5f2d9c19ba621399 Show Sample Output
1zcat error.log.gz | sed 's^\[.*\]^^g' | sed 's^\, referer: [^\n]*^^g' | sort | uniq -c | sort -n -
capture 2000 packets and print the top 10 talkers
1tcpdump -tnn -c 2000 -i eth0 | awk -F "." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr | awk ' $1 > 10 ' - ‹ First < 3 4 5 6 7 > Last ›
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands using uniq