Commands using uniq (255)
-
This dup finder saves time by comparing size first, then md5sum, it doesn't delete anything, just lists them.
82find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate -
Written for linux, the real example is how to produce ascii text graphs based on a numeric value (anything where uniq -c is useful is a good candidate). Show Sample Output
54netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }' -
show only the name of the apps that are using internet Show Sample Output
36lsof -P -i -n | cut -f 1 -d " "| uniq | tail -n +2 -
26netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -c
-
23history | awk '{print $2}' | sort | uniq -c | sort -rn | head
-
This uses awk to grab the IP address from each request and then sorts and summarises the top 10.
22tail -10000 access_log | awk '{print $1}' | sort | uniq -c | sort -n | tail -
awk is evil! Show Sample Output
22ps hax -o user | sort | uniq -c -
Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.
18find -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35- -
Busiest seconds:
cat /var/log/secure.log | awk '{print substr($0,0,15)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'Show Sample Output17cat /var/log/secure.log | awk '{print substr($0,0,12)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}' -
I'm working in a group project currently and annoyed at the lack of output by my teammates. Wanting hard metrics of how awesome I am and how awesome they aren't, I wrote this command up. It will print a full repository listing of all files, remove the directories which confuse blame, run svn blame on each individual file, and tally the resulting line counts. It seems quite slow, depending on your repository location, because blame must hit the server for each individual file. You can remove the -R on the first part to print out the tallies for just the current directory. Show Sample Output
16svn ls -R | egrep -v -e "\/$" | xargs svn blame | awk '{print $2}' | sort | uniq -c | sort -r -
This command takes the output of the 'last' command, removes empty lines, gets just the first field ($USERNAME), sort the $USERNAMES in reverse order and then gives a summary count of unique matches. Show Sample Output
15last | grep -v "^$" | awk '{ print $1 }' | sort -nr | uniq -c -
Here is a command line to run on your server if you think your server is under attack. It prints our a list of open connections to your server and sorts them by amount. BSD Version:
netstat -na |awk '{print $5}' |cut -d "." -f1,2,3,4 |sort |uniq -c |sort -nrShow Sample Output14netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n -
usefull in case of abuser/DoS attacks. Show Sample Output
12netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -n -
... plus do a sort according frequency Show Sample Output
11find . -type f | awk -F'.' '{print $NF}' | sort| uniq -c | sort -g -
11netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r
-
9sed 's/[ \t]*$//' < emails.txt | tr 'A-Z' 'a-z' | sort | uniq > emails_sorted.txt
-
This command checks for the number of times when someone has tried to login to your server and failed. If there are a lot, then that user is being targeted on your system and you might want to make sure that user either has remote logins disabled, or has a strong password, or both. If your output has an "invalid" line, it is a summary of all logins from users that don't exist on your system. Show Sample Output
9zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less -
Adapted using your usefull comments !
9grep -e `date +%Y-%m-%d` /var/log/dpkg.log | awk '/install / {print $4}' | uniq | xargs apt-get -y remove -
8gunzip -c /var/log/auth.log.*.gz | cat - /var/log/auth.log /var/log/auth.log.0 | grep "Invalid user" | awk '{print $8;}' | sort | uniq -c | less
-
Searches the /var/log/secure log file for Failed and/or invalid user log in attempts. Show Sample Output
8cat /var/log/secure | grep sshd | grep Failed | sed 's/invalid//' | sed 's/user//' | awk '{print $11}' | sort | uniq -c | sort -n -
Finds the top ten pages returning an http response code of 404 in an apache log.
8awk '$9 == 404 {print $7}' access_log | uniq -c | sort -rn | head -
credit to tumblr engineering blog @ http://engineering.tumblr.com/ Show Sample Output
8netstat -tn | awk 'NR>2 {print $6}' | sort | uniq -c | sort -rn -
8history | awk '{print $2}' | sort | uniq -c | sort -rn | head
-
7sed -e 's/ *$//' ~/.bash_history | sort | uniq -cd | sort -nr | head
-
It will return a ranked list of your most commonly-entered commands using your command history Show Sample Output
7history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r - 1 2 3 > Last ›
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands using uniq