Commands using awk sorted by votes

Commands using awk (1,418)

sorted by

Outputs unique error messages from the apache log and count, sorted by frequency

Outputs unique error messages from the apache log, sorted by frequency. Tested on log format: [Wed Aug 26 00:00:00 2015] [error] [client xx.xx.xx.xx] Error message here Show Sample Output
This is sample output - yours may be different.
```
63 PHP PHP Warning: json_encode(): Invalid UTF-8 sequence in argument
10 Warning: Warning: preg_match() expects parameter 2 to be string, array given
7 PHP PHP Warning: Invalid argument supplied for foreach() in ...
5 PHP PHP Warning: implode(): Invalid arguments passed in ...
1 PHP PHP Parse error: syntax error, unexpected ',' in ...
```
0

cat /var/log/apache2/error.log | awk '{out=$9;for(i=9;i<=NF;i++){out=out" "$i}; print out}' | sed s/,\ referer.*// | sort | uniq -c | sort -nr

cmattoon · 2015-08-26 14:14:12 10
generate a core dump of a process

tested on solaris 10
This is sample output - yours may be different.
0

ps -e | awk '$4~/<process name>/{print $1}' |xargs gcore -o ~/troubleshooting

cognosco41 · 2015-08-28 22:44:59 10
Sum columns on one line in a csv file.

sample.csv: 79.36,94.93,10.92,27.33,95.90 3.57, 20.80,67.06,2.16, 79.23 48.45,27.95,7.66, 56.71,59.97 69.02,89.59,33.88,42.73,22.60 10.15,44.86,70.86,98.45,22.23 Show Sample Output
This is sample output - yours may be different.
```
308.44
172.82
200.74
257.82
246.55
```
0

awk -F , '{for(i=1;i<=NF;i++)t[NR]+=$i;$0=t[NR]}1' sample.csv

YardApe · 2015-09-05 13:29:18 13
Deletes cpu time consuming plugin-containe triggered by firefox

Some times firefox run a plugin named "plugin-containe". It sometimes take so much cpu that it is impossible to keep browsing. When this happens, one can simply kill the process. If the process does not exist, the command returns an innocuous error message.
This is sample output - yours may be different.
0

kill -9 `top -b -n 1 | grep plugin-containe | awk '{print $1}'`

otaviolb · 2015-09-08 18:14:53 11
Find non common lines between two files according to a given column

Two files are compared according their (third, $3) column. The command shows the lines ($0) that are in file 2 and not in file1
This is sample output - yours may be different.
0

awk '(ARGIND==1) {tab[$3]=1} (ARGIND==2 && ! ($3 in tab) ) {print $0}' file1 file2

herve · 2015-09-11 07:16:14 10
Total space used by open but deleted files

A potential source of a full filesystem are large files left open but have been deleted. On Linux, a file may be deleted (removed/unlinked) while a process has it open. When this happens, the file is essentially invisible to other processes, but it still takes on physical space on the drive. Tools like du will not see it.
This is sample output - yours may be different.
0

sudo lsof -nP | awk '/deleted/ { sum+=$8 } END { print sum }'

jeffskinnerbox · 2015-09-19 00:45:23 16
Epoch from time protocol port 37

Time protocol serves up seconds since 1900 in a 32 bit binary number. Show Sample Output
This is sample output - yours may be different.
```
1443004481
```
0

telnet time.nist.gov 37 2>/dev/null |tail -1 | xxd | awk '{print $2 $3}'|tr -d '\n' | perl -nwe "print scalar(hex($1)-2208988800).\"\n\""

quesne · 2015-09-23 10:42:36 9
Monitor memory fine-grained usage (e.g. firefox)

I have modified the original one-liner to allow it to manage multiple PIDs. However, I haven't found the way it can work properly with the watch command. NOTE: brace expand must be activated via "set -B" in order to this script can work. Show Sample Output
This is sample output - yours may be different.
```
961936 KB
```
0

cat $(echo "ls /proc/{`pidof firefox | sed -e 's/$/,/' -e 's/ /,/g'`}/smaps 2>/dev/null" | bash) | awk '/Rss/{sum += $2; } END{print sum, "KB"}'

pchalim · 2015-09-30 22:22:53 9
display 10 biggest open files
This is sample output - yours may be different.
0

lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB "$9 }' | sort -n -u | tail

lolssl · 2015-10-02 19:43:04 9
display swap space used by a process
This is sample output - yours may be different.
0

awk '/VmSwap/{print $2 " " $3}' /proc/$PID/status

lolssl · 2015-10-02 19:43:46 10
ps to show child thread PIDs
This is sample output - yours may be different.
0

ls -1 /proc/$(ps ax | grep <Process Name> | head -n 1 | awk '{print $1;}')/task | tail -n +2

happymarmoset · 2015-10-06 07:44:48 10
Current directory disk usage by day for last 90 days in GB
This is sample output - yours may be different.
0

for x in {0..90}; do { echo $x $(find . -type f -mtime $x -exec du -cs {} \; |awk -F " " '{sum+=$1} END {sum/=1048576}END{print sum,"G"}'); }; done

g0dsp33d · 2015-10-08 18:19:00 9
To see which software currently open in directory /var/cache
This is sample output - yours may be different.
0

lsof -Pn +D /var/cache/ | awk '{print $1}' | sort | uniq

olto · 2015-11-12 05:30:44 14
get a random regex match from a file

seed the random number generator, find all matches in a file put all matches from the capture group into an array return a random element from the array Show Sample Output
This is sample output - yours may be different.
```
25
```
0

awk 'BEGIN{srand()} match($0, /DELTA=([0-9]+);/, a) {w[i++]=a[1]} END {print w[int(rand()*i)]}' file.name

jkirchartz · 2015-11-13 17:56:34 11
Filter ltab file based on minor allele count
This is sample output - yours may be different.
0

awk '{counts = 137 * $3 *$6; line = $0;}$1 == "CHROM" ||$4 < max_het && $3 > min_cov && $7 == "0" && counts > 1.1 {print line}' file.ltab |cut -f 1,2,9- > file.filtered.tab

gregory_owens · 2015-12-02 22:40:32 10
s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

get the size of S3 bucket Show Sample Output
This is sample output - yours may be different.
```
12.598 GB
```
0

s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

erez83 · 2015-12-16 08:48:28 9
Git reset added new files
This is sample output - yours may be different.
0

for f in `git status | grep new | awk '{print $3}'`; do git reset HEAD $f ; done

Wafelijzer · 2015-12-16 22:24:34 16
Get current Xorg resolution via xrandr

if there are multiple monitors, this command uses multiple lines Show Sample Output
This is sample output - yours may be different.
```
1280x800
```
0

xrandr | awk '/*/ {print $1}'

misterhat · 2015-12-21 15:29:39 11
quick livestream screencast

to view on another box: nc <server address> <port> | ffplay - use -r to adjust FPS and -q to adjust compression. use on trusted network only as nc is unencrypted.
This is sample output - yours may be different.
0

ffmpeg -f x11grab -s $(xrandr | awk '/*/ {print $1}') -r 10 -i :0 -an -q 10 -f mjpeg - | nc -lp <port>

misterhat · 2015-12-21 17:15:30 11
Get list of all apache virtual hosts

list of all virtual hosts which are defined in all apache configuration files
This is sample output - yours may be different.
0

sudo httpd -S 2>&1 | grep 'port' | awk '{print $4,$2,$5}' | sort | column -t

gangs · 2016-01-06 12:04:47 11
grab all m4a file from bbc radio 4 extra for some easy audio ebbok listening

use get_iplay to download all listed content from http://www.bbc.co.uk/radio4extra run every night to make sure no episodes are missed
This is sample output - yours may be different.
0

get_iplayer --type=radio --channel "Radio 4 Extra" | grep : | awk '{ if ( NR > 1 ) { print } }'|sed 's/:.*//' |sed '$ d' > pidlist && while read p; do get_iplayer --get --fields=pid $p; done <pidlist && rm pidlist

dunryc · 2016-01-16 17:20:54 11
Show UDID of iPhone

this is what i needed on ubuntu 12.04 for ios 9
This is sample output - yours may be different.
0

lsusb -s `lsusb | grep "Apple" | cut -d ' ' -f 2`:`lsusb | grep "Apple" | cut -d ' ' -f 4 | sed 's/://'` -v | grep iSerial | awk '{print $3}'

dormi · 2016-01-19 09:08:27 10
Get OSX Battery percentage
This is sample output - yours may be different.
0

pmset -g batt | awk '/-InternalBattery-/ {sub(/;/, "", $2); print $2}'

schlueter · 2016-01-26 21:55:33 11
list lines changed per all authors of a git repository
This is sample output - yours may be different.
0

for i in $(git log --format='%ae' | sort -u); do echo $i; git log --author="$i" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' - done

flavioab · 2016-01-27 20:46:51 11
enable all bash completions in gentoo

Pros: * it's much faster then for loop * removes first line of 'eselect bashcomp list' - which contains text "Available completions:"
This is sample output - yours may be different.
0

eselect bashcomp enable --global $(eselect bashcomp list | tail -n +2 | awk '{print $2}' |xargs)

eNcacz · 2016-02-12 16:53:22 12
‹ First < 42 43 44 45 46 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

find duplicate messages in a Maildir

# find assumes email files start with a number 1-9 # sed joins the lines starting with " " to the previous line # gawk print the received and from lines # sort according to the second field (received+from) # uniq print the duplicated filename # a message is viewed as duplicate if it is received at the same time as another message, and from the same person. The command was intended to be run under cron. If run in a terminal, mutt can be used: mutt -e "push otD~=xq" -f $folder

Format ps command output

ps command gives the possibility to display information with custom formatting with the -o options followed by the format specifier list.

Print only the odd lines of a file (GNU sed)

The tilde address operator is an extension of GNU sed. It won't work with POSIX sed.

Test disk I/O

Especially useful to gauge the performance of a VPS.

find co-ordinates of a location

Just add this to your .bashrc file. Use quotes when query has multiple word length.

Date shows dates at other times/dates

Use date to find the date at other days and times.

Create a mirror of a local folder, on a remote server

Create a exact mirror of the local folder "/root/files", on remote server 'remote_server' using SSH command (listening on port 22) (all files & folders on destination server/folder will be deleted)

Creates a 'path' command that always prints the full path to any file

The command creates an alias called 'path', so it's useful to add it to your .profile or .bash_profile. The path command then prints the full path of any file, directory, or list of files given. Soft links will be resolved to their true location. This is especially useful if you use scp often to copy files across systems. Now rather then using pwd to get a directory, and then doing a separate cut and paste to get a file's name, you can just type 'path file' and get the full path in one operation.

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Function that counts recursively number of lines of all files in specified folders

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using awk

Commands using awk (1,418) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using awk (1,418)

sorted by