Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.


Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Universal configuration monitoring and system of record for IT.
Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

Hide

News

May 19, 2015 - A Look At The New Commandlinefu
I've put together a short writeup on what kind of newness you can expect from the next iteration of clfu. Check it out here.
March 2, 2015 - New Management
I'm Jon, I'll be maintaining and improving clfu. Thanks to David for building such a great resource!
Hide

Top Tags

Hide

Functions

Psst. Open beta.

Wow, didn't really expect you to read this far down. The latest iteration of the site is in open beta. It's a gentle open beta-- not in prime-time just yet. It's being hosted over at UpGuard (link) and you are more than welcome to give it a shot. Couple things:

  • » The open beta is running a copy of the database that will not carry over to the final version. Don't post anything you don't mind losing.
  • » If you wish to use your user account, you will probably need to reset your password.
Your feedback is appreciated via the form on the beta page. Thanks! -Jon & CLFU Team

Display the standard deviation of a column of numbers with awk

Terminal - Display the standard deviation of a column of numbers with awk
awk '{sum+=$1; sumsq+=$1*$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' file.dat
2009-03-24 21:56:40
User: kaan
Functions: awk
6
Display the standard deviation of a column of numbers with awk

Alternatives

There is 1 alternative - vote for the best!

Terminal - Alternatives
awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { print sqrt(mean2 / NR); }'
2009-09-11 04:46:01
User: ashawley
Functions: awk delta
Tags: awk
4

This will calculate a running standard deviation in one pass and should never have the possibility for overflow that can happen with other implementations. I suppose there is a potential for underflow in the corner case where the deltas are small or the values themselves are small.

Know a better way?

If you can do better, submit your command here.

What others think

I was finding that this was 'blowing up' when processing a large number (1000's) of lat/long positions with an error message about attempting to SQRT a negative number.

Apparently the correct way is to do two passes, compute the mean first, sum the difference between value & mean.

You can do this with:

awk '{sum+=$1; array[NR]=$1} END {for(x=1;x<=NR;x++){sumsq+=((array[x]-(sum/NR))**2);}print sqrt(sumsq/NR)}' file.dat
Comment by mungewell 399 weeks ago

can you help me how to edit this command, if I have the values in 2nd column?

Comment by siti 359 weeks ago

siti: Just replace all occurrences of $1 with $2 (in either version).

Thanks to mungewell for pointing out the overflow problem, and to ashawley for the elegant code.

Comment by kaan 358 weeks and 6 days ago

Can someone help me to modify this code to be able to normalize each value of the column. I guess, after having the mean and the standard deviation, this should be pretty easy.

thanks in advance.

Comment by anurag_bagaria 323 weeks and 6 days ago

It is sometimes challenging to anticipate fashion trends, pertaining to companies realizing which often product or service fashion can certainly make the idea to the movements indicates large earnings. The direction they market many might decide the achievement and having a new pattern for that year. Handbags work just like exception to this rule, there are always styles that will remove very easily next you have in addition models that merely fizzle away. The entire year Next year might find this beginning of a few classic types that happen to be nonetheless planning solid in the end most of these a number of associated with some new variations women will appear for in their number of some sort of handbag. Here i will discuss many of the handbag variations which can be growing to be very well liked.1. Handbags that have fine equipment. These can possibly be handbags having straps which are woven as well as weaved as well as people that are manufactured by combining a variety of supplies. Many possess grips that happen to be woven as well as weaved via organizations. It may also present pleated styles.2. Gemstone-colored handbags may also be whatever you have to consider in 2011. From the palest Emerald green to your best Ruby crimson plus the deepest Opal azure, 2011's handbags are in the market to hook your eye area of any girl.3. Handbags which may have variable purses will always be all around. They often hold storage compartments outside and inside offering a great deal of efficiency. The particular purses placed external are meant to enhance the handbag's overall appearance in the moderate types of embellishment.4. Handbags should have two methods you are able to take all of them. Handbags with both manages along with bands include the styles with regard to This year. Although this has existed for quite a while today, it can make handbags more well-designed and also enjoyment to work with. The straps allow you to carry the actual bag over your own glenohumeral joint as the addresses enable you to take it in your hands.5. A different hot development you will certainly appreciate tend to be handbags that are big. http://www.hermesbeltsbirkinkellybag.co.uk/ You can have everything that you need inside. Be sure that you take it nicely though because if you have a tiny framework, it might overwhelm you in case you carry it through your adjustable rate mortgage. You must bring a big handbag near to your current body so that you know a large frameworked female.6. People amazing forms that you simply refer to more aged women have returned in vogue and therefore are growing to be milder every day. Be sure to function as the first to demonstrate the little rectangle flap bag and that is definitely warm.Giraffe designs are all around village previously and it has zero inclinations to get out of style yet. Expect you'll discover much more these types of prints that year This year.

Comment by stephanieg915 3 days ago

Your point of view

You must be signed in to comment.