Commands using awk (1,418)

  • 121
    history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
    unixmonkey611 · 2009-02-11 13:12:29 28
  • I find this terribly useful for grepping through a file, looking for just a block of text. There's "grep -A # pattern file.txt" to see a specific number of lines following your pattern, but what if you want to see the whole block? Say, the output of "dmidecode" (as root): dmidecode | awk '/Battery/,/^$/' Will show me everything following the battery block up to the next block of text. Again, I find this extremely useful when I want to see whole blocks of text based on a pattern, and I don't care to see the rest of the data in output. This could be used against the '/etc/securetty/user' file on Unix to find the block of a specific user. It could be used against VirtualHosts or Directories on Apache to find specific definitions. The scenarios go on for any text formatted in a block fashion. Very handy.

    awk '/start_pattern/,/stop_pattern/' file.txt
    atoponce · 2009-03-28 14:28:59 25
  • Using awk, find duplicates in a file without sorting, which reorders the contents. awk will not reorder them, and still find and remove duplicates which you can then redirect into another file.

    awk '!x[$0]++' <file>
    din7 · 2009-12-20 02:33:21 23
  • Written for linux, the real example is how to produce ascii text graphs based on a numeric value (anything where uniq -c is useful is a good candidate). Show Sample Output

    netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'
    knassery · 2009-04-27 22:02:19 24
  • Checks the Gmail ATOM feed for your account, parses it and outputs a list of unread messages. For some reason sed gets stuck on OS X, so here's a Perl version for the Mac: curl -u username:password --silent "" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | perl -pe 's/^<title>(.*)<\/title>.*<name>(.*)<\/name>.*$/$2 - $1/' If you want to see the name of the last person, who added a message to the conversation, change the greediness of the operators like this: curl -u username:password --silent "" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | perl -pe 's/^<title>(.*)<\/title>.*?<name>(.*?)<\/name>.*$/$2 - $1/' Show Sample Output

    curl -u username:password --silent "" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"
    postrational · 2009-09-07 21:56:40 42
  • Print all columns except the 1st and 3rd.

    awk '{$1=$3=""}1' file
    zlemini · 2011-10-25 22:15:06 15
  • You have an external USB drive or key. Apply this command (using the file path of anything on your device) and it will simulate the unplug of this device. If you just want the port, just type : echo $(sudo lshw -businfo | grep -B 1 -m 1 $(df "/path/to/file" | tail -1 | awk '{print $1}' | cut -c 6-8) | head -n 1 | awk '{print $1}' | cut -c 5- | tr ":" "-") Show Sample Output

    echo $(sudo lshw -businfo | grep -B 1 -m 1 $(df "/path/to/file" | tail -1 | awk '{print $1}' | cut -c 6-8) | head -n 1 | awk '{print $1}' | cut -c 5- | tr ":" "-") | sudo tee /sys/bus/usb/drivers/usb/unbind
    tweet78 · 2014-04-06 12:06:29 19
  • What happens here is we tell tar to create "-c" an archive of all files in current dir "." (recursively) and output the data to stdout "-f -". Next we specify the size "-s" to pv of all files in current dir. The "du -sb . | awk ?{print $1}?" returns number of bytes in current dir, and it gets fed as "-s" parameter to pv. Next we gzip the whole content and output the result to out.tgz file. This way "pv" knows how much data is still left to be processed and shows us that it will take yet another 4 mins 49 secs to finish. Credit: Peteris Krumins Show Sample Output

    tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz
    opertinicy · 2009-12-18 17:09:08 13

  • 26
    netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -c
    himynameisthor · 2009-02-05 18:02:59 34
  • Show the number of failed tries of login per account. If the user does not exist it is marked with *. Show Sample Output

    sudo zcat /var/log/auth.log.*.gz | awk '/Failed password/&&!/for invalid user/{a[$9]++}/Failed password for invalid user/{a["*" $11]++}END{for (i in a) printf "%6s\t%s\n", a[i], i|"sort -n"}'
    point_to_null · 2009-03-21 06:41:59 14
  • This makes an alias for a command named 'busy'. The 'busy' command opens a random file in /usr/include to a random line with vim. Drop this in your .bash_aliases and make sure that file is initialized in your .bashrc.

    alias busy='my_file=$(find /usr/include -type f | sort -R | head -n 1); my_len=$(wc -l $my_file | awk "{print $1}"); let "r = $RANDOM % $my_len" 2>/dev/null; vim +$r $my_file'
    busybee · 2010-03-09 21:48:41 18
  • This uses awk to grab the IP address from each request and then sorts and summarises the top 10.

    tail -10000 access_log | awk '{print $1}' | sort | uniq -c | sort -n | tail
    root · 2009-01-25 21:01:52 47

  • 22
    history | awk '{print $2}' | sort | uniq -c | sort -rn | head
    mikeda · 2009-02-17 14:25:49 17

  • 21
    sudo tcpdump -i wlan0 -n ip | awk '{ print gensub(/(.*)\..*/,"\\1","g",$3), $4, gensub(/(.*)\..*/,"\\1","g",$5) }' | awk -F " > " '{print $1"\n"$2}'
    tweet78 · 2014-04-11 22:41:32 9
  • Takes a input file (count.txt) that looks like: 1 2 3 4 5 It will add/sum the first column of numbers.

    cat count.txt | awk '{ sum+=$1} END {print sum}'
    duxklr · 2009-03-16 00:22:13 28
  • When you fill a formular with Firefox, you see things you entered in previous formulars with same field names. This command list everything Firefox has registered. Using a "delete from", you can remove anoying Google queries, for example ;-)

    cd ~/.mozilla/firefox/ && sqlite3 `cat profiles.ini | grep Path | awk -F= '{print $2}'`/formhistory.sqlite "select * from moz_formhistory" && cd - > /dev/null
    klipz · 2009-04-13 20:23:37 15
  • Parses /etc/group to "dot" format and pases it to "display" (imagemagick) to show a usefull diagram of users and groups (don't show empty groups).

    awk 'BEGIN{FS=":"; print "digraph{"}{split($4, a, ","); for (i in a) printf "\"%s\" [shape=box]\n\"%s\" -> \"%s\"\n", $1, a[i], $1}END{print "}"}' /etc/group|display
    point_to_null · 2011-12-04 01:56:44 14
  • This loops through all tables and changes their collations to UTF8. You should backup beforehand though in case some data is lost in the process.

    mysql --database=dbname -B -N -e "SHOW TABLES" | awk '{print "ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;"}' | mysql --database=dbname &
    root · 2009-03-21 18:45:15 15
  • Busiest seconds: cat /var/log/secure.log | awk '{print substr($0,0,15)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}' Show Sample Output

    cat /var/log/secure.log | awk '{print substr($0,0,12)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'
    knassery · 2009-07-24 07:20:06 12

  • 17
    man -t awk | ps2pdf - awk.pdf
    kev · 2011-11-23 01:40:23 8
  • I use this on debian testing, works like the other sorted du variants, but i like small numbers and suffixes :) Show Sample Output

    du --max-depth=1 | sort -r -n | awk '{split("k m g",v); s=1; while($1>1024){$1/=1024; s++} print int($1)" "v[s]"\t"$2}'
    hans · 2009-02-24 11:03:08 15
  • Breaks down and numbers each line and it's fields. This is really useful when you are going to parse something with awk but aren't sure exactly where to start. Show Sample Output

    awk '{print NR": "$0; for(i=1;i<=NF;++i)print "\t"i": "$i}'
    recursiverse · 2009-07-23 06:25:31 56
  • I'm working in a group project currently and annoyed at the lack of output by my teammates. Wanting hard metrics of how awesome I am and how awesome they aren't, I wrote this command up. It will print a full repository listing of all files, remove the directories which confuse blame, run svn blame on each individual file, and tally the resulting line counts. It seems quite slow, depending on your repository location, because blame must hit the server for each individual file. You can remove the -R on the first part to print out the tallies for just the current directory. Show Sample Output

    svn ls -R | egrep -v -e "\/$" | xargs svn blame | awk '{print $2}' | sort | uniq -c | sort -r
    askedrelic · 2009-07-29 02:10:45 15
  • This command displays a list of lines that are longer than 72 characters. I use this command to identify those lines in my scripts and cut them short the way I like it.

    awk 'length>72' file
    haivu · 2009-09-10 05:54:41 12
  • This is a very simple and lightweight way to play DI.FM stations For a more complete version of the command with proper strings in the menu, try: (couldnt fit in the command field above) zenity --list --width 500 --height 500 --title 'DI.FM' --text 'Pick a Radio' --column 'radio' --column 'url' --print-column 2 $(curl -s | awk -F '"' '/href="http:.*\.pls.*96k/ {print $2}' | sort | awk -F '/|\.' '{print $(NF-1) " " $0}') | xargs mplayer This command line parses the html returned from and display all radio stations in a nice graphical menu. After the radio is chosen, the url is passed to mplayer so the music can start dependencies: - x11 with gtk environment - zenity: simple app for displaying gtk menus (sudo apt-get install zenity on ubuntu) - mplayer: simple audio player (sudo apt-get install mplayer on ubuntu) Show Sample Output

    zenity --list --width 500 --height 500 --column 'radio' --column 'url' --print-column 2 $(curl -s | awk -F '"' '/href="http:.*\.pls.*96k/ {print $2}' | sort | awk -F '/|\.' '{print $(NF-1) " " $0}') | xargs mplayer
    polaco · 2010-04-28 23:45:35 14
  •  1 2 3 >  Last ›

What's this? is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Get a diff of two json arrays
jq is amazing for manipulating json on the commandline, but the developers have some weird ideas about how to handle shell redirections. This command works around them. Further reading:

SMS reminder
Send an e-mail to SMS reminder in 15 minutes from now, to call my wife. See list of carriers bellow Carrier Email to SMS Gateway Alltel [10-digit phone number] AT&T (formerly Cingular) [10-digit phone number] [10-digit phone number] (MMS) [10-digit phone number] Boost Mobile [10-digit phone number] Nextel (now Sprint Nextel) [10-digit telephone number] Sprint PCS (now Sprint Nextel) [10-digit phone number] [10-digit phone number] (MMS) T-Mobile [10-digit phone number] US Cellular [10-digit phone number] (SMS) [10-digit phone number] (MMS) Verizon [10-digit phone number] [10-digit phone number] (MMS) Virgin Mobile USA [10-digit phone number]

show git logging
shows some logging for the git repo.

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Listing directory content of a directory with a lot of entries
Ever wanted to get the directory content with 'ls' or 'find' and had to wait minutes until something was printed? Perl to the rescue. The one-liner above(redirected to a file) took less than five seconds to run in a directory with more man 2 million files. One can adapt it to e.g. delete files that match a certain pattern.

Rename files in batch

Find duplicate UID in /etc/passwd
You can use only awk

Project your desktop using xrandr
HDMI-1 is the interface in the example, which can be obtained just by typing xrandr and surfing through the output. There are a hell lot of configurations that can be done but I prefer auto because it works in most cases. $ Lifesaver

Vectorize xkcd strips
Uses ImageMagick and potrace to vectorize the input image, with parameters optimized for xkcd-like pictures.

Find the processes that are on the runqueue. Processes with a status of
Want to know why your load average is so high? Run this command to see what processes are on the run queue. Runnable processes have a status of "R", and commands waiting on I/O have a status of "D". On some older versions of Linux may require -emo instead of -eo. On Solaris: ps -aefL -o s -o user -o comm | egrep "^O|^R|COMMAND"

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.


Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: