Display the standard deviation of a column of numbers with awk

awk '{delta = \$1 - avg; avg += delta / NR; mean2 += delta * (\$1 - avg); } END { print sqrt(mean2 / NR); }'
This will calculate a running standard deviation in one pass and should never have the possibility for overflow that can happen with other implementations. I suppose there is a potential for underflow in the corner case where the deltas are small or the values themselves are small.

4
2009-09-11 04:46:01

2 Alternatives + Submit Alt

• 6
awk '{sum+=\$1; sumsq+=\$1*\$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' file.dat
· 2009-03-24 21:56:40
• I was concerned that the other scripts didn't use the correct order of operations for the calculation. Variance needs to be squared before being summed, then divided by NR and then square rooted. Not sure if this method is correct but this is what I came up with. I think it works correctly but I'm not totally sure. If anyone could critique it that would be great.

0
awk '{delta=\$1; avg+=\$1/NR;} END {print "stdev = " sqrt(((delta-avg)^2)/NR);}'
· 2016-04-19 08:49:49

What Others Think

I was the one who commented on the alternate/older system which was failing. Not 100% on how your scheme works, but it appears to get the same result at the stuff I posted. Good job! -- grep -e '^#BEST' 00111260_Control.ASC | grep 'L1_INT' | cut -d ',' -f 14 | less | awk '{sum+=\$1; array[NR]=\$1} END {for(x=1;x<=NR;x++){sumsq+=((array[x]-(sum/NR))**2);}print sqrt(sumsq/NR)}' 0.0111318 grep -e '^#BEST' 00111260_Control.ASC | grep 'L1_INT' | cut -d ',' -f 14 | less | awk '{sum+=\$1; sumsq+=\$1*\$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' 0.0111299 grep -e '^#BEST' 00111260_Control.ASC | grep 'L1_INT' | cut -d ',' -f 14 | less | awk '{delta = \$1 - avg; avg += delta / NR; mean2 += delta * (\$1 - avg); } END { print sqrt(mean2 / NR); }' 0.0111318 -- mungewell · 526 weeks and 2 days ago
Thanks. And thank you for testing it on your large data set. ashawley · 526 weeks and 2 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Stay in the loop…  