computes the most frequent used words of a text file

cat WAR_AND_PEACE_By_LeoTolstoi.txt | tr -cs "[:alnum:]" "\n"| tr "[:lower:]" "[:upper:]" | awk '{h[$1]++}END{for (i in h){print h[i]" "i}}'|sort -nr | cat -n | head -n 30

using

cat WAR_AND_PEACE_By_LeoTolstoi.txt | tr -cs "[:alnum:]" "\n"| tr "[:lower:]" "[:upper:]" | sort -S16M | uniq -c |sort -nr | cat -n | head -n 30

("sort -S1G" - Linux/GNU sort only) will also do the job but as some drawbacks (caused by space/time complexity of sorting) for bigger files...

Sample Output

# get some input http://www.gutenberg.org
$ cat WAR_AND_PEACE_By_LeoTolstoi.txt | tr -cs "[:alnum:]" "\n"| tr "[:lower:]" "[:upper:]" | awk '{h[$1]++}END{for (i in h){print h[i]" "i}}'|sort -nr | cat -n | head -n 30 
     1  34720 THE
     2  22300 AND
     3  16753 TO
     4  15007 OF
     5  10608 A
     6  10004 HE
     7  9036 IN
     8  8204 THAT
     9  7984 HIS
    10  7359 WAS
    11  5710 WITH
    12  5617 IT
    13  5365 HAD
    14  4725 HER
    15  4697 NOT
    16  4637 HIM
    17  4547 AT
    18  4524 I
    19  4414 S
    20  4054 BUT
    21  4035 AS
    22  4014 ON
    23  3871 YOU
    24  3555 FOR
    25  3488 SHE
    26  3347 IS
    27  2842 SAID
    28  2813 ALL
    29  2709 FROM
    30  2458 BY

By: cp

2010-07-05 06:39:20

awk cat head sort tr

Submit An Alternative

What Others Think

i think there's database of most common used words in English, so we can remove those words from this list and see the frequently used words specifically by Tolstoi

alperyilmaz · 744 weeks and 6 days ago

right; do that it is fun! C.

cp · 744 weeks and 5 days ago

Very similar to my own script: http://l0b0.wordpress.com/2010/05/31/tag-cloud-shell-script/

l0b0 · 744 weeks and 5 days ago

The awk part is needlessly complicated. You can use "uniq -c" for that.

inof · 744 weeks and 4 days ago

uniq -c can not do this! have a look in my description.

cp · 744 weeks and 4 days ago

https://seofox.ir/seo/ https://seofox.ir/gray-hat-seo/ https://seofox.ir/search-engine/ https://seofox.ir/page-rank/ https://seofox.ir/backlink-definition/ https://seofox.ir/seo-techniques/ https://seofox.ir/technical-seo/ https://seofox.ir/on-page-seo/ https://seofox.ir/seo-tools/ https://seofox.ir/off-page-seo/ https://seofox.ir/seo-consultation/ https://seofox.ir/on-page-seo/ https://seofox.ir/google-seo-book/ https://seofox.ir/add-site-in-google/ https://seofox.ir/category/seo/ https://seofox.ir/seo-training/ https://seofox.ir/order/ https://seofox.ir/page-speed/ https://seofox.ir/seo-company https://seofox.ir/white-hat-seo/ https://seofox.ir/seo-principles/ https://seofox.ir/category/seo-content/ https://seofox.ir/seo-checklist/ https://seofox.ir/category/seo-content/ https://seofox.ir/google-algorithms/ https://seofox.ir/google-penalty-removal/ https://seofox.ir/rules-of-link-building/ https://seofox.ir/analyze-seo/ https://seofox.ir/backlink/ https://seofox.ir/seo-services-prices/ ??? ? ????? ????

seofox · 140 weeks and 4 days ago

seofox · 139 weeks and 5 days ago

https://absolute-iran.com https://mymakeupstory-iran.com https://londonhousedoctor.com https://ayams.ir https://foroosh-page.ir https://digitalboost.ir https://like7.ir https://karetehran.ir https://seo7x.ir https://qoou.ir https://ayams.com https://tosifyan.ir https://karekaraj.ir https://sefidii.ir https://seo7x.com https://drinstagram.com https://tabliigh.ir https://ghoyesiah.ir https://denizpet.ir https://rezabahrami.net https://damunsabt.com https://damunsabt.ir

instakade1 · 128 weeks and 1 day ago

EUSAPPLY.IR

saranikola · 117 weeks and 6 days ago

STUDY IN ITALY STUDY IN HUNGARY STUDY IN SPANIA STUDY IN RUSSIA STUDY IN TURKEY IMAT EXAM MEDICINA TOLC DUOLINGO IELTS EXAM TOEFL IBT HOME EDITION TOLC TEST CILS CELI EXAM EUSAPPLY.IR CKGS APPOINTMENT EUSAPPLY.IR ?

saranikola · 117 weeks and 6 days ago

rahimhh21 · 110 weeks and 5 days ago

Perfecthomepugs · 101 weeks and 6 days ago

themeoff.ir

themeoff · 98 weeks and 2 days ago

pugpuppies95 · 64 weeks ago

Thank you for sharing valuable information natural pearl wholesaler in India

onenaturalpearl · 39 weeks and 6 days ago

Enter the YouTube activation code displayed on your device by login into yt.be/activate. By this you can activate YouTube on Smart TV by login to the webpage yt.be and enter the activation code.