Commands using uniq (255)
-
0zgrep 'pattern' /var/logs/apache2/access.log* | awk '{print $7}' | sort -n | uniq -c | sort -rn
-
To allow recursivity :
find -type f -exec md5sum '{}' ';' | sort | uniq -c -w 33 | sort -gr | head -n 5 | cut -c1-7,41-Display only filenames :find -maxdepth 1 -type f -exec md5sum '{}' ';' | sort | uniq -c -w 33 | sort -gr | head -n 5 | cut -c43-Show Sample Output0find -maxdepth 1 -type f -exec md5sum '{}' ';' | sort | uniq -c -w 33 | sort -gr | head -n 5 | cut -c1-7,41- -
Useful in scripts to compare known malicious IP addresses with what you are actually blocking. Show Sample Output
0ufw status | sort | uniq | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" -
Check the connection of the maximum number of IP Show Sample Output
0netstat -na | grep ESTABLISHED | awk '{print$5}' | awk -F : '{print$1}' | sort | uniq -c | sort -r -
It works extremely fast, because it calculates md5sum only on the files that have the same size and name. But there is nothing for free - it won't find duplicates with the different names. Show Sample Output
0find -type f -printf '%20s\t%100f\t%p\n' | sort -n | uniq -Dw121 | awk -F'\t' '{print $3}' | xargs -d '\n' md5sum | uniq -Dw32 | cut -b 35- | xargs -d '\n' ls -lU -
0netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head
-
if you have less free inode space alert in your monitoring app you can use this command to find the any directory with too many files and subdirectory . Show Sample Output
0find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n -
0cat *pound* | grep someURL.com | awk '{print $7}'| uniq -c | sort -bgr >> uniqueIPwC.txt
-
If you have two sets of files that may share hard-linked files, it can be useful to identify which ones are hard links to same underlying inode (file). This command shows you all of those, sorted by inode#. In my example the two directory trees to compare share a common parent, so I run the command in that parent and just use
find .to start from the current directory. If yours are in different locations, you can pass both paths to find:find /directory1 /directory2 -type f -printf '%10i %p\n' | sort | uniq -w 11 -d -D | lessShow Sample Output0find . -type f -printf '%10i %p\n' | sort | uniq -w 11 -d -D | less -
This command will generate "CHECK TABLE `db_name.table_name` ;" statements for all tables present in databases on a MySQL server, which can be piped into the mysql command. (Can also be altered to perform OPTIMIZE and REPAIR functions.) Tested on MySQL 4.x and 5.x systems in a Linux environment under bash. Show Sample Output
-1DD=`cat /etc/my.cnf | sed "s/#.*//g;" | grep datadir | tr '=' ' ' | gawk '{print $2;}'` && ( cd $DD ; find . -mindepth 2 | grep -v db\.opt | sed 's/\.\///g; s/\....$//g; s/\//./;' | sort | uniq | tr '/' '.' | gawk '{print "CHECK TABLE","`"$1"`",";";}' ) -
Can be used to discover what programms create internet traffic. Skip the part after awk to get more details. Has anyone an idea why the uniq doesn't work propperly here (see sample output)? Show Sample Output
-1netstat -lantp | grep -i establ | awk -F/ '{print $2}' | uniq | sort -
You'll run into trouble if you have files w/ missing newlines at the end. I tried to use
PAGER='sed \$q' git blameand evenPAGER='sed \$q' git -p blameto force a newline at the end, but as soon as the output is redirected, git seems to ignore the pager.-1git ls-files | while read i; do git blame $i | sed -e 's/^[^(]*(//' -e 's/^\([^[:digit:]]*\)[[:space:]]\+[[:digit:]].*/\1/'; done | sort | uniq -ic | sort -nr -
print members both in file1 and file2
-1sort file1 file2 | uniq -d -
First column is number of photos, second column is the focal length. Show Sample Output
-1exiv2 *JPG | grep Focal | awk '{print $5}' | sort -n | uniq -c -
Per country GET report, based on access log. Easy to transform to unique IP Show Sample Output
-1cat /var/log/nginx/access.log | grep -oe '^[0-9.]\+' | perl -ne 'system("geoiplookup $_")' | grep -v found | grep -oe ', [A-Za-z ]\+$' | sort | uniq -c | sort -n -
#_connects src_IP dst_IP When_It_Happened_Secs Show Sample Output
-1tshark -qr [cap] -z conv,tcp | awk '{printf("%s:%s:%s\n",$1,$3,$10)}' | awk -F: '{printf("%s %s %s\n",$1,$3,substr($5,1,length($5)-10))}' | sort | uniq -c | sort -nr -
Output contains also garbage (text parts from netstat's output) but it's good enough for quick check who's overloading your server.
-1netstat -antu | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n -
This commands queries the delicious api then runs the xml through xml2, grabs the urls cuts out the first two columns, passes through uniq to remove duplicates if any, and then goes into linkchecker who checks the links. the links go the blacklist in ~/.linkchecker/blacklist. please see the manual pages for further info peeps. I took me a few days to figure this one out. I how you enjoy it. Also don't run these api more then once a few seconds you can get banned by delicious see their site for info. ~updated for no recursive Show Sample Output
-1curl -k https://Username:Password@api.del.icio.us/v1/posts/all?red=api | xml2| \grep '@href' | cut -d\= -f 2- | sort | uniq | linkchecker -r0 --stdin --complete -v -t 50 -F blacklist -
Btrfs reports the inode numbers of files with failed checksums. Use `find` to lookup the file names of those inodes.
-1dmesg | grep -Po 'csum failed ino\S* \d+' | sort | uniq | xargs -n 3 find / -inum 2> /dev/null -
-1cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100
-
display IP's that unsuccessfully attempted to login 5 or more times today may want to filter any trusted IP's and the localhost useful for obtaining a list IP addresses to block on the firewall Show Sample Output
-1lastb -i | grep "$(date '+%a %b %d')" | awk '{ print $3 }' | sort | uniq -c | awk '{ if ($1 >= 5) print $2; }' -
Useful to check DDoS attacks on servers. Show Sample Output
-2netstat -alpn | grep :80 | awk '{print $4}' |awk -F: '{print $(NF-1)}' |sort | uniq -c | sort -n -
Searches /var/log/secure for smtp connections then lists these by number of connections made and hosts.
-2cat /var/log/secure | grep smtp | awk '{print $9}' | cut -f2 -d= | sort | uniq -c | sort -n | tail -
I make an extensive use of sudo, so I had to exclude the sudo part of the command history
-2history | sed -e 's/^sudo //' | awk '{print $2}' | sort | uniq -c | sort -rn | head -
This grabs all lines that make an instantation or static call, then filters out the cruft and displays a summary of each class called and the frequency. Show Sample Output
-3grep -o "\(new \(\w\+\)\|\w\+::\)" file.php | sed 's/new \|:://' | sort | uniq -c | sort - ‹ First < 8 9 10 11 >
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands using uniq