Commands using awk sorted by votes

Commands using awk (1,418)

sorted by

Outputs unique error messages from the apache log and count, sorted by frequency

Outputs unique error messages from the apache log, sorted by frequency. Tested on log format: [Wed Aug 26 00:00:00 2015] [error] [client xx.xx.xx.xx] Error message here Show Sample Output
This is sample output - yours may be different.
```
63 PHP PHP Warning: json_encode(): Invalid UTF-8 sequence in argument
10 Warning: Warning: preg_match() expects parameter 2 to be string, array given
7 PHP PHP Warning: Invalid argument supplied for foreach() in ...
5 PHP PHP Warning: implode(): Invalid arguments passed in ...
1 PHP PHP Parse error: syntax error, unexpected ',' in ...
```
0

cat /var/log/apache2/error.log | awk '{out=$9;for(i=9;i<=NF;i++){out=out" "$i}; print out}' | sed s/,\ referer.*// | sort | uniq -c | sort -nr

cmattoon · 2015-08-26 14:14:12 10
generate a core dump of a process

tested on solaris 10
This is sample output - yours may be different.
0

ps -e | awk '$4~/<process name>/{print $1}' |xargs gcore -o ~/troubleshooting

cognosco41 · 2015-08-28 22:44:59 10
Sum columns on one line in a csv file.

sample.csv: 79.36,94.93,10.92,27.33,95.90 3.57, 20.80,67.06,2.16, 79.23 48.45,27.95,7.66, 56.71,59.97 69.02,89.59,33.88,42.73,22.60 10.15,44.86,70.86,98.45,22.23 Show Sample Output
This is sample output - yours may be different.
```
308.44
172.82
200.74
257.82
246.55
```
0

awk -F , '{for(i=1;i<=NF;i++)t[NR]+=$i;$0=t[NR]}1' sample.csv

YardApe · 2015-09-05 13:29:18 13
Deletes cpu time consuming plugin-containe triggered by firefox

Some times firefox run a plugin named "plugin-containe". It sometimes take so much cpu that it is impossible to keep browsing. When this happens, one can simply kill the process. If the process does not exist, the command returns an innocuous error message.
This is sample output - yours may be different.
0

kill -9 `top -b -n 1 | grep plugin-containe | awk '{print $1}'`

otaviolb · 2015-09-08 18:14:53 11
Find non common lines between two files according to a given column

Two files are compared according their (third, $3) column. The command shows the lines ($0) that are in file 2 and not in file1
This is sample output - yours may be different.
0

awk '(ARGIND==1) {tab[$3]=1} (ARGIND==2 && ! ($3 in tab) ) {print $0}' file1 file2

herve · 2015-09-11 07:16:14 10
Total space used by open but deleted files

A potential source of a full filesystem are large files left open but have been deleted. On Linux, a file may be deleted (removed/unlinked) while a process has it open. When this happens, the file is essentially invisible to other processes, but it still takes on physical space on the drive. Tools like du will not see it.
This is sample output - yours may be different.
0

sudo lsof -nP | awk '/deleted/ { sum+=$8 } END { print sum }'

jeffskinnerbox · 2015-09-19 00:45:23 16
Epoch from time protocol port 37

Time protocol serves up seconds since 1900 in a 32 bit binary number. Show Sample Output
This is sample output - yours may be different.
```
1443004481
```
0

telnet time.nist.gov 37 2>/dev/null |tail -1 | xxd | awk '{print $2 $3}'|tr -d '\n' | perl -nwe "print scalar(hex($1)-2208988800).\"\n\""

quesne · 2015-09-23 10:42:36 9
Monitor memory fine-grained usage (e.g. firefox)

I have modified the original one-liner to allow it to manage multiple PIDs. However, I haven't found the way it can work properly with the watch command. NOTE: brace expand must be activated via "set -B" in order to this script can work. Show Sample Output
This is sample output - yours may be different.
```
961936 KB
```
0

cat $(echo "ls /proc/{`pidof firefox | sed -e 's/$/,/' -e 's/ /,/g'`}/smaps 2>/dev/null" | bash) | awk '/Rss/{sum += $2; } END{print sum, "KB"}'

pchalim · 2015-09-30 22:22:53 9
display 10 biggest open files
This is sample output - yours may be different.
0

lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB "$9 }' | sort -n -u | tail

lolssl · 2015-10-02 19:43:04 9
display swap space used by a process
This is sample output - yours may be different.
0

awk '/VmSwap/{print $2 " " $3}' /proc/$PID/status

lolssl · 2015-10-02 19:43:46 10
ps to show child thread PIDs
This is sample output - yours may be different.
0

ls -1 /proc/$(ps ax | grep <Process Name> | head -n 1 | awk '{print $1;}')/task | tail -n +2

happymarmoset · 2015-10-06 07:44:48 10
Current directory disk usage by day for last 90 days in GB
This is sample output - yours may be different.
0

for x in {0..90}; do { echo $x $(find . -type f -mtime $x -exec du -cs {} \; |awk -F " " '{sum+=$1} END {sum/=1048576}END{print sum,"G"}'); }; done

g0dsp33d · 2015-10-08 18:19:00 9
To see which software currently open in directory /var/cache
This is sample output - yours may be different.
0

lsof -Pn +D /var/cache/ | awk '{print $1}' | sort | uniq

olto · 2015-11-12 05:30:44 14
get a random regex match from a file

seed the random number generator, find all matches in a file put all matches from the capture group into an array return a random element from the array Show Sample Output
This is sample output - yours may be different.
```
25
```
0

awk 'BEGIN{srand()} match($0, /DELTA=([0-9]+);/, a) {w[i++]=a[1]} END {print w[int(rand()*i)]}' file.name

jkirchartz · 2015-11-13 17:56:34 11
Filter ltab file based on minor allele count
This is sample output - yours may be different.
0

awk '{counts = 137 * $3 *$6; line = $0;}$1 == "CHROM" ||$4 < max_het && $3 > min_cov && $7 == "0" && counts > 1.1 {print line}' file.ltab |cut -f 1,2,9- > file.filtered.tab

gregory_owens · 2015-12-02 22:40:32 10
s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

get the size of S3 bucket Show Sample Output
This is sample output - yours may be different.
```
12.598 GB
```
0

s3cmd du s3://bucket-name | awk '{print $0/1024/1024/1024" GB"}'

erez83 · 2015-12-16 08:48:28 9
Git reset added new files
This is sample output - yours may be different.
0

for f in `git status | grep new | awk '{print $3}'`; do git reset HEAD $f ; done

Wafelijzer · 2015-12-16 22:24:34 16
Get current Xorg resolution via xrandr

if there are multiple monitors, this command uses multiple lines Show Sample Output
This is sample output - yours may be different.
```
1280x800
```
0

xrandr | awk '/*/ {print $1}'

misterhat · 2015-12-21 15:29:39 11
quick livestream screencast

to view on another box: nc <server address> <port> | ffplay - use -r to adjust FPS and -q to adjust compression. use on trusted network only as nc is unencrypted.
This is sample output - yours may be different.
0

ffmpeg -f x11grab -s $(xrandr | awk '/*/ {print $1}') -r 10 -i :0 -an -q 10 -f mjpeg - | nc -lp <port>

misterhat · 2015-12-21 17:15:30 11
Get list of all apache virtual hosts

list of all virtual hosts which are defined in all apache configuration files
This is sample output - yours may be different.
0

sudo httpd -S 2>&1 | grep 'port' | awk '{print $4,$2,$5}' | sort | column -t

gangs · 2016-01-06 12:04:47 11
grab all m4a file from bbc radio 4 extra for some easy audio ebbok listening

use get_iplay to download all listed content from http://www.bbc.co.uk/radio4extra run every night to make sure no episodes are missed
This is sample output - yours may be different.
0

get_iplayer --type=radio --channel "Radio 4 Extra" | grep : | awk '{ if ( NR > 1 ) { print } }'|sed 's/:.*//' |sed '$ d' > pidlist && while read p; do get_iplayer --get --fields=pid $p; done <pidlist && rm pidlist

dunryc · 2016-01-16 17:20:54 11
Show UDID of iPhone

this is what i needed on ubuntu 12.04 for ios 9
This is sample output - yours may be different.
0

lsusb -s `lsusb | grep "Apple" | cut -d ' ' -f 2`:`lsusb | grep "Apple" | cut -d ' ' -f 4 | sed 's/://'` -v | grep iSerial | awk '{print $3}'

dormi · 2016-01-19 09:08:27 10
Get OSX Battery percentage
This is sample output - yours may be different.
0

pmset -g batt | awk '/-InternalBattery-/ {sub(/;/, "", $2); print $2}'

schlueter · 2016-01-26 21:55:33 11
list lines changed per all authors of a git repository
This is sample output - yours may be different.
0

for i in $(git log --format='%ae' | sort -u); do echo $i; git log --author="$i" --pretty=tformat: --numstat | awk '{ add += $1; subs += $2; loc += $1 - $2 } END { printf "added lines: %s, removed lines: %s, total lines: %s\n", add, subs, loc }' - done

flavioab · 2016-01-27 20:46:51 11
enable all bash completions in gentoo

Pros: * it's much faster then for loop * removes first line of 'eselect bashcomp list' - which contains text "Available completions:"
This is sample output - yours may be different.
0

eselect bashcomp enable --global $(eselect bashcomp list | tail -n +2 | awk '{print $2}' |xargs)

eNcacz · 2016-02-12 16:53:22 12
‹ First < 42 43 44 45 46 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Fastest Sort. Sort Faster, Max Speed

sort is way slow by default. This tells sort to use a buffer equal to half of the available free memory. It also will use multiple process for the sort equal to the number of cpus on your machine (if greater than 1). For me, it is magnitudes faster. If you put this in your bash_profile or startup file, it will be set correctly when bash is started. $ sort -S1 --parallel=2 /dev/null && alias sortfast='sort -S$(($(sed '\''/MemF/!d;s/[^0-9]*//g'\'' /proc/meminfo)/2048)) $([ `nproc` -gt 1 ]&&echo -n --parallel=`nproc`)' Alternative $ echo|sort -S10M --parallel=2 &>/dev/null && alias sortfast="command sort -S$(($(sed '/MemT/!d;s/[^0-9]*//g' /proc/meminfo)/1024-200)) --parallel=$(($(command grep -c ^proc /proc/cpuinfo)*2))"

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Virtualbox: setup hardware

where - memory 256 assign 256 Mb RAM - acpi on enable ACPI (mandatory if you use Winfog 2000 - ioapic off disable the IO APIC. Not useful if you use one CPU (on virtual machine or a 32 bit operative system). As ACPI, this switch is mandatory for Winbug 2000 - pae on enable the Phisical Address Extension how to use more than 4Gb of RAM on x86 CPU - hwvirtex on enables hardware virtualization extensions for microprocessors that have this feature (which should be also enabled in the BIOS of the motherboard) - nestedpaging on allows part of the processes of memory management hardware are made directly

Testing php configuration

Command to logout all the users in one command

Remove executable bit from all files in the current directory recursively, excluding other directories

Useful if you have copied files from an OS without a permission structure (for example, DOS) and you need to disable all executable files but want to be able to descend into your directories.

Set gnome wallpaper to a random jpg from the specified directory

Every time this is run it will change your background picture. For added fun Add some DBUS magic: . $HOME/.dbus/session-bus/`cat /var/lib/dbus/machine-id`-0 export DBUS_SESSION_BUS_ADDRESS and a crontab entry: */5 * * * * above_command_in_script.sh >/dev/null 2>/dev/null now wallpaper changes every 5 mins

Show the total number of changes that every user committed to a Subversion repository

This saves Subversion's log output as XML and then runs an XQuery over it. This is standard XQuery 1.0 and should therefore also work with other XQuery processors. I have tested it with Zorba (http://www.zorba-xquery.com). XQilla (http://xqilla.sourceforge.net) also does it, but you'd have to save the query to a file and then execute "xqilla filename.xq". The query first finds all distinct authors and then, for each author, sums up the number of paths they have changed in each commit. This accounts for commits of multiple changes at once. The indenting space in all lines from the second one seems to be due to a bug in Zorba.

Console clock

Shows a simple clock in the console -t param removes the watch header Ctrl-c to exit

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using awk

Commands using awk (1,418) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using awk (1,418)

sorted by