Commands using uniq (255)
-
3sort file1.txt | uniq > file2.txt
-
The command could show you all conecctions if you skip "grep ESTABLISHED" Show Sample Output
3netstat -ant | grep :80 | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n -
3awk '{print $1}' /var/log/httpd/access_log | sort | uniq -c | sort -rnk1 | head -n 10
-
Searches all log files (including archived bzip2 files) for invalid user and PAM authentication errors, both of which are indicative of brute force attempts at logging into computer. A list of all unique IP addresses and domain names is appended to hosts.deny. The command (and grep error messages) will work on Mac OS X 10.6, small adjustments may be needed for other OSs.
3(bzcat BZIP2_FILES && cat TEXT_FILES) | grep -E "Invalid user|PAM" | grep -o -E "from .+" | awk '{print $2}' | sort | uniq >> /etc/hosts.deny -
Revised approach to and3k's version, using pipes and read rather than command substitution. This does not require fiddling with IFS when paths have whitespace, and does not risk hitting command-line size limits. It's less verbose on the missing files, but it stops iterating at the first file that's missing, so it should be definitely faster. I expanded all the qlist options to be more self-describing.
3emerge -av1 `qlist --installed --nocolor | uniq | while read cp; do qlist --exact $cp | while read file; do test -e $file || { echo $cp; echo "$cp: missing $file (and maybe more)" 1>&2; break; }; done; done` -
perl-less way
3(netstat -atn | awk '{printf "%s\n%s\n", $4, $4}' | grep -oE '[0-9]*$'; seq 32768 61000) | sort -n | uniq -u | head -n 1 -
3history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr
-
I'm sure there's a more elegant sed version for the tr + grep section.
3ls | tr '[[:punct:][:space:]]' '\n' | grep -v "^\s*$" | sort | uniq -c | sort -bn -
Updated according to flatcap's suggestion, thanks!
3groups user1 user2|cut -d: -f2|xargs -n1|sort|uniq -d -
2sudo lsof | awk '{printf("%s %s %s\n", $1, $3, $NF)}' | grep -v "(" | sort -k 4 | gawk '$NF==prv{ct++;next} {printf("%d %s\n",ct,$0);ct=1;prv=$NF}' | uniq | sort -nr
-
2perl -pe 's/.+;//' ~/.zsh_history | sort | uniq -c | sort -r|head -10
-
2# cut -d ' ' -f1 /var/log/nginx/nginx-access.log | sort | uniq -c | sort -nr | head -10 | nl
-
This command does a tally of concurrent active connections from single IPs and prints out those IPs that have the most active concurrent connections. VERY useful in determining the source of a DoS or DDoS attack. Show Sample Output
2netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -
2find . -type f -name "*.java" -print0 | xargs -0 -n 1 svn blame | sed -n 's/^[^a-z]*\([a-z]*\).*$/\1/p' | sort | uniq -c | sort -n
-
2lsof |awk ' {if ( $0 ~ /home/) print substr($0, index($0,"/home") ) }'|cut -d / -f 1-4|sort|uniq -c|sort -bgr
-
Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored. Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv":
find -type d -name ".csv" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".csv" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separateShow Sample Output2find -type d -name ".svn" -prune -o -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type d -name ".svn" -prune -o -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate -
2grep current_state= /var/log/nagios/status.dat|sort|uniq -c|sed -e "s/[\t ]*\([0-9]*\).*current_state=\([0-9]*\)/\2:\1/"|tr "\n" " "
-
count the times a domain appears on a file which lines are URLs in the form http://domain/resource. Show Sample Output
2cut -d'/' -f3 file | sort | uniq -c -
2svn log 2>&1 | egrep '^r[0-9]+' | cut -d "|" -f2 | sort | uniq -c
-
list top committers (and number of their commits) of svn repository. in this example it counts revisions of current directory. Show Sample Output
2svn log -q | grep '^r[0-9]' | cut -f2 -d "|" | sort | uniq -c | sort -nr -
This command will allow to search for duplicate processes and sort them by their run count. Note that if there are same processes run by different users you'll see only one user in the result line, so you'll need to do:
ps aux | grep <process>to see all users that run this command. Show Sample Output2ps aux | sort --key=11 | uniq -c -d --skip-fields=10 | sort -nr --key=1,1 -
2grep "HTTP/1.1\" 404" access_log | awk '{print $7 } ' | sort | uniq -c | sort -n
-
This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.
2find . -type f -exec md5 '{}' ';' | sort | uniq -f 3 -d | sed -e "s/.*(\(.*\)).*/\1/" -
Uses line-porcelain in git blame, which makes it easier to parse the output.
2git ls-files | xargs -n1 git blame --line-porcelain | sed -n 's/^author //p' | sort -f | uniq -ic | sort -nr -
# find assumes email files start with a number 1-9 # sed joins the lines starting with " " to the previous line # gawk print the received and from lines # sort according to the second field (received+from) # uniq print the duplicated filename # a message is viewed as duplicate if it is received at the same time as another message, and from the same person. The command was intended to be run under cron. If run in a terminal, mutt can be used: mutt -e "push otD~=xq" -f $folder Show Sample Output
2find $folder -name "[1-9]*" -type f -print|while read file; do echo $file $(sed -e '/^$/Q;:a;$!N;s/\n //;ta;s/ /_/g;P;D' $file|awk '/^Received:/&&!r{r=$0}/^From:/&&!f{f=$0}r&&f{printf "%s%s",r,f;exit(0)}');done|sort -k 2|uniq -d -f 1 - < 1 2 3 4 5 > Last ›
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands using uniq