Commands using awk (1,244)

  • This prints a summary of your referers from your logs as long as they occurred a certain number of times (in this case 500). The grep command excludes the terms, I add this in to remove results Im not interested in. Show Sample Output


    1
    awk -F\" '{print $4}' *.log | grep -v "eviljaymz\|\-" | sort | uniq -c | awk -F\ '{ if($1>500) print $1,$2;}' | sort -n
    jaymzcd · 2009-05-05 22:21:04 0
  • I use this (well I normally just drop the F=*.log bit and put that straight into the awk command) to count how many times I get referred from another site. I know its rough, its to give me an idea where any posts I make are ending up. The reason I do the Q="query" bit is because I often want to check another domain quickly and its quick to use CTRL+A to jump to the start and then CTRL+F to move forward the 3 steps to change the grep query. (I find this easier than moving backwards because if you group a lot of domains with the pipe your command line can get quite messy so its normally easier to have it all at the front so you just have to edit it & hit enter). For people new to the shell it does the following. The Q and F equals bits just make names we can refer to. The awk -F\" '{print $4}' $F reads the file specified by $F and splits it up using double-quotes. It prints out the fourth column for egrep to work on. The 4th column in the log is the referer domain. egrep then matches our query against this list from awk. Finally wc -l gives us the total number of lines (i.e. matches). Show Sample Output


    0
    Q="reddit|digg"; F=*.log; awk -F\" '{print $4}' $F | egrep $Q | wc -l
    jaymzcd · 2009-05-05 21:51:16 0
  • It's like `prstat -t` under Solaris Show Sample Output


    1
    ps -eo user,pcpu,pmem | tail -n +2 | awk '{num[$1]++; cpu[$1] += $2; mem[$1] += $3} END{printf("NPROC\tUSER\tCPU\tMEM\n"); for (user in cpu) printf("%d\t%s\t%.2f%\t%.2f%\n",num[user], user, cpu[user], mem[user]) }'
    feraudet · 2009-05-05 16:54:25 0
  • This command might not be useful for most of us, I just wanted to share it to show power of command line. Download simple text version of novel David Copperfield from Poject Gutenberg and then generate a single column of words after which occurences of each word is counted by sort | uniq -c combination. This command removes numbers and single characters from count. I'm sure you can write a shorter version. Show Sample Output


    -4
    wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}'
    alperyilmaz · 2009-05-04 16:00:39 8
  • Ok so it's rellay useless line and I sorry for that, furthermore that's nothing optimized at all... At the beginning I didn't managed by using netstat -p to print out which process was handling that open port 4444, I realize at the end I was not root and security restrictions applied ;p It's nevertheless a (good ?) way to see how ps(tree) works, as it acts exactly the same way by reading in /proc So for a specific port, this line returns the calling command line of every thread that handle the associated socket


    -5
    p=$(netstat -nate 2>/dev/null | awk '/LISTEN/ {gsub (/.*:/, "", $4); if ($4 == "4444") {print $8}}'); for i in $(ls /proc/|grep "^[1-9]"); do [[ $(ls -l /proc/$i/fd/|grep socket|sed -e 's|.*\[\(.*\)\]|\1|'|grep $p) ]] && cat /proc/$i/cmdline && echo; done
    j0rn · 2009-04-30 12:39:48 1
  • This works just like write or wall ... cept one thing the sender is anonymous ... if you really want to drive everyone insane replace echo \"The Matrix has you...\" with cat /dev/urandom nice one to do on April fool's day Show Sample Output


    -12
    w | egrep -v '(load|FROM)' | awk '{print $2}' | sed 's/^/tty/' | awk '{print "echo \"The Matrix has you...\" >> /dev/" $1}' | bash
    copremesis · 2009-04-29 22:04:56 6

  • 4
    ps auxf | grep httpd | grep -v grep | grep -v defunct | awk '{sum=sum+$6}; END {print sum/1024}'
    empulse · 2009-04-29 17:43:52 2
  • This will, for an application that has already been removed but had its configuration left behind, purge that configuration from the system. To test it out first, you can remove the last -y, and it will show you what it will purge without actually doing it. I mean it never hurts to check first, "just in case." ;)


    -3
    dpkg-query -l| grep -v "ii " | grep "rc " | awk '{print $2" "}' | tr -d "\n" | xargs aptitude purge -y
    thepicard · 2009-04-28 19:25:53 3
  • Purge all configuration files of removed packages Show Sample Output


    13
    sudo aptitude purge `dpkg --get-selections | grep deinstall | awk '{print $1}'`
    kelevra · 2009-04-28 11:44:04 5
  • Useful for removes a package and its depends, for example to remove the gnome desktop environment, also configuration files will be removed, you should be carefully and sure that you want to do this. Show Sample Output


    -4
    sudo apt-get remove --purge `dpkg -l | awk '{print $2}' | grep gnome` && apt-get autoremove
    kelevra · 2009-04-28 10:34:42 7
  • This is useful if you have need to do port forwarding and your router doesn't assign static IPs, you can add it to a script in a cron job that checks if you IP as recently changed or with a trigger script. This was tested on Mac OSX.


    9
    ifconfig en1 | awk '/inet / {print $2}' | mail -s "hello world" email@email.com
    rez0r · 2009-04-28 06:01:52 4
  • Written for linux, the real example is how to produce ascii text graphs based on a numeric value (anything where uniq -c is useful is a good candidate). Show Sample Output


    49
    netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'
    knassery · 2009-04-27 22:02:19 7
  • Replace FILE with a filename (or - for stdin).


    -3
    awk 'BEGIN {srand()} {print int(rand()*1000000) "\t" $0}' FILE | sort -n | cut -f 2-
    udim · 2009-04-19 20:04:58 2
  • Use the find command to match certain files and summarise their total size in KBytes. Show Sample Output


    2
    find /path/to/my/files/ -type f -name "*txt*" | xargs du -k | awk 'BEGIN{x=0}{x=x+$1}END{print x}'
    unixmonkey3280 · 2009-04-16 14:17:04 3
  • A variation of a script I found on this site and then slimmed down to just use awk. It displays all users who have attempted to login to the box and failed using SSH. Pipe it to the sort command to see which usernames have the most failed logins. Show Sample Output


    10
    awk '/sshd/ && /Failed/ {gsub(/invalid user/,""); printf "%-12s %-16s %s-%s-%s\n", $9, $11, $1, $2, $3}' /var/log/auth.log
    frailotis · 2009-04-16 00:56:23 0

  • 3
    /sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1
    d4n3sh · 2009-04-15 06:44:18 8

  • 5
    find | sed -e "s/^.*\///" | awk ' BEGIN { FS=""} { print NF " " $0 } ' | sort -nrf | head -10
    nitehawk · 2009-04-13 21:03:07 0
  • When you fill a formular with Firefox, you see things you entered in previous formulars with same field names. This command list everything Firefox has registered. Using a "delete from", you can remove anoying Google queries, for example ;-)


    19
    cd ~/.mozilla/firefox/ && sqlite3 `cat profiles.ini | grep Path | awk -F= '{print $2}'`/formhistory.sqlite "select * from moz_formhistory" && cd - > /dev/null
    klipz · 2009-04-13 20:23:37 3

  • 2
    svn status | grep "^\?" | awk '{print $2}' | xargs svn add
    miguelbaldi · 2009-04-13 18:37:58 1
  • Lists the local files that are not present in the remote repository (lines beginning with ?) and add them. Show Sample Output


    1
    svn status | grep '^?' | awk '{ print $2; }' | xargs svn add
    unixmonkey4200 · 2009-04-10 21:55:37 2

  • 4
    grep "install " /var/log/dpkg.log | awk '{print $4}' | xargs apt-get -y remove --purge
    skygreg · 2009-04-06 13:06:04 3
  • It will return a ranked list of your most commonly-entered commands using your command history Show Sample Output


    7
    history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r
    kayowas · 2009-04-05 13:40:56 5
  • List files above a given size threshold. Show Sample Output


    5
    find . -type f -size +25000k -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'
    benajnim · 2009-04-02 15:01:17 3
  • This command will display only the hosts that are active in the network. Show Sample Output


    0
    nmap -sP your network/submask | awk "/^Host/"'{ print $2 }'
    rauhmaru · 2009-04-02 12:41:54 3

  • 1
    find ./ -not -type d | xargs wc -l | cut -c 1-8 | awk '{total += $1} END {print total}'
    benschw · 2009-04-01 15:14:15 5
  • ‹ First  < 43 44 45 46 47 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: