# Display the standard deviation of a column of numbers with awk

awk '{sum+=\$1; sumsq+=\$1*\$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' file.dat

6
2009-03-24 21:56:40

## 2 Alternatives + Submit Alt

• This will calculate a running standard deviation in one pass and should never have the possibility for overflow that can happen with other implementations. I suppose there is a potential for underflow in the corner case where the deltas are small or the values themselves are small.

4
awk '{delta = \$1 - avg; avg += delta / NR; mean2 += delta * (\$1 - avg); } END { print sqrt(mean2 / NR); }'
· 2009-09-11 04:46:01
• I was concerned that the other scripts didn't use the correct order of operations for the calculation. Variance needs to be squared before being summed, then divided by NR and then square rooted. Not sure if this method is correct but this is what I came up with. I think it works correctly but I'm not totally sure. If anyone could critique it that would be great.

0
awk '{delta=\$1; avg+=\$1/NR;} END {print "stdev = " sqrt(((delta-avg)^2)/NR);}'
· 2016-04-19 08:49:49

### What Others Think

I was finding that this was 'blowing up' when processing a large number (1000's) of lat/long positions with an error message about attempting to SQRT a negative number. Apparently the correct way is to do two passes, compute the mean first, sum the difference between value & mean. You can do this with: `awk '{sum+=\$1; array[NR]=\$1} END {for(x=1;x<=NR;x++){sumsq+=((array[x]-(sum/NR))**2);}print sqrt(sumsq/NR)}' file.dat` mungewell · 550 weeks and 2 days ago
can you help me how to edit this command, if I have the values in 2nd column? siti · 510 weeks and 3 days ago
siti: Just replace all occurrences of \$1 with \$2 (in either version). Thanks to mungewell for pointing out the overflow problem, and to ashawley for the elegant code. kaan · 510 weeks and 2 days ago
Can someone help me to modify this code to be able to normalize each value of the column. I guess, after having the mean and the standard deviation, this should be pretty easy. thanks in advance. anurag_bagaria · 475 weeks and 2 days ago

### What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

### What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

### Stay in the loop…  