Commands using uniq (255)
-
grabbed from Andrew Aylett post: http://superuser.com/questions/133313/can-i-speed-up-cygwins-fork Show Sample Output
2while (true); do date --utc; done | uniq -c -
Get the longest match of file extension (Ex. For 'foo.tar.gz', you get '.tar.gz' instead of '.gz') Show Sample Output
2find /some/path -type f -printf '%f\n' | grep -o '\..\+$' | sort | uniq -c | sort -rn -
2tail -F some.log | perl -ne 'print time(), "\n";' | uniq -c
-
Analyze an Apache access log for the time period with most activity and display the hit count, requesting IP and the timestamp. May help detect a brute force dos attack.
2cut -d " " -f1,4 access_log | sort | uniq -c | sort -rn | head -
IP addresses and number of connections connected to port 80. Show Sample Output
2netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' |sed -e 's/::ffff://' | cut -f1 -d: | sort | uniq -c | sort -rn | head -
2export PORT=11211; ss -an4 | grep -E "ESTAB.*$PORT" | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -nr
-
2awk -F: '{print $2}' access_log | sort | uniq -c
-
Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on:
find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separateThe new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.2find -not -empty -type f -printf "%-30s'\t\"%h/%f\"\n" | sort -rn -t$'\t' | uniq -w30 -D | cut -f 2 -d $'\t' | xargs md5sum | sort | uniq -w32 --all-repeated=separate -
Access a random news web page on the internet. The Links browser can of course be replaced by Firefox or any modern graphical web browser.
2links $( a=( $( lynx -dump -listonly "http://news.google.com" | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | grep -v "google.com" | sort | uniq ) ) ; amax=${#a[@]} ; n=$(( `date '+%s'` % $amax )) ; echo ${a[n]} ) -
In this case searches for where .desktop files are stored. The resulted is a sorted list of the top directories containing such files. Show Sample Output
2find . -name '*.desktop' | sed s/[^/]*\.desktop$// | uniq -c | sort -g -
You can append these commands to the bottom of the history file to access them easier with the Up key:
sort ~/.bash_history|uniq -c|sort -n|tail -n 10|tr -s " "|cut -d' ' -f3- >> ~/.bash_history2sort ~/.bash_history|uniq -c|sort -n|tail -n 10 -
1last | awk '{ print $1 }' | sort | uniq -c | grep -v wtmp
-
Displays the duplicated lines in a file and their occuring frequency.
1cat file.txt | sort | uniq -dc -
This command puts all the flags of the USE variable actually used by the packages you emerged to the file "use", and those which are unused but available to the file "notuse"
1emerge -epv world | grep USE | cut -d '"' -f 2 | sed 's/ /\n/g' | sed '/[(,)]/d' | sed s/'*'//g | sort | uniq > use && grep ^- use | sed s/^-// | sed ':a;N;$!ba;s/\n/ /g' > notuse && sed -i /^-/d use && sed -i ':a;N;$!ba;s/\n/ /g' use -
This prints a summary of your referers from your logs as long as they occurred a certain number of times (in this case 500). The grep command excludes the terms, I add this in to remove results Im not interested in. Show Sample Output
1awk -F\" '{print $4}' *.log | grep -v "eviljaymz\|\-" | sort | uniq -c | awk -F\ '{ if($1>500) print $1,$2;}' | sort -n -
1nmap -sP <subnet>.* | egrep -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' > results.txt ; for IP in {1..254} ; do echo "<subnet>.${IP}" ; done >> results.txt ; cat results.txt | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq -u
-
1cut -d\ -f 1 ~/.bash_history | sort | uniq -c | sort -rn | head -n 10 | sed 's/.*/ &/g'
-
This command will return a full list of Error 404 pages in the given access log. The following variables have been given to awk Hostname ($2), ERROR Code ($9), Missing Item ($7), Referrer ($11) You can then send this into a file (>> /path/to/file), which you can open with OpenOffice as a CSV
1sudo awk '($9 ~ /404/)' /var/log/httpd/www.domain-access_log | awk '{print $2,$9,$7,$11}' | sort | uniq -c -
Counts TCP states from Netstat and displays in an ordered list. Show Sample Output
1netstat -an | awk '/tcp/ {print $6}' | sort | uniq -c -
1grep -R usepackage * | cut -d']' -f2 | cut -s -d'{' -f 2 | sed s/"}"/.sty"}"/g | cut -d'}' -f1 | sort | uniq | xargs dpkg -S | cut -d':' -f1 | sort | uniq
-
This command is primarily going to work on linux boxes. and needs to be changed, for example IP=10\.194\.194\.2 PORT=389
1netstat -n | grep '^tcp.*<IP>:<PORT>' | tr " " | awk 'BEGIN{FS="( |:)"}{print $6}' | sort | uniq -c | sort -n -k1 | awk '{if ($1 >= 10){print $2}}' -
1tr A-Z a-z | tr -d "[[:punct:]][[:digit:]]" | tr ' /_' '\n' | sort | uniq -c
-
To learn more about Google Ngram Viewer: http://ngrams.googlelabs.com/info
1wget -qO - http://ngrams.googlelabs.com/datasets | grep -E href='(.+\.zip)' | sed -r "s/.*href='(.+\.zip)'.*/\1/" | uniq | while read line; do `wget $line`; done -
Counts the frequency of words in a file Show Sample Output
1cat YOUR_FILE|tr -d '[:punct:]'|tr '[:upper:]' '[:lower:]'|tr -s ' ' '\n'|sort|uniq -c|sort -rn -
1cat /backup/hd7/rdiff-log.txt |grep Processing | awk '{ print $4 }' | sed -e 's/\// /g' | awk '{ print $1 }' |uniq -c |sort -n
- ‹ First < 2 3 4 5 6 > Last ›
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands using uniq