Commands using uniq sorted by votes

Commands using uniq (255)

sorted by

Show duplicate lines in a file

The following displays only the entries that are duplicates. Show Sample Output
This is sample output - yours may be different.
```
2 Alex Jason:200:Sales
2 Emma Thomas:100:Marketing
```
0

sort namesd.txt | uniq ?cd

ankush108 · 2012-06-26 19:23:58 3
List all the unique extensions in a folder and print their count
This is sample output - yours may be different.
0

find . -type f -print | awk -F'.' '{print $NF}' | sort | uniq -c

vishnu81 · 2012-10-31 12:03:10 5
count & sort one field of the log files

count & sort one field of the log files , such as nginx/apache access log files .
This is sample output - yours may be different.
0

tail -1000 `ls -ltr /var/log/CF* |tail -1|awk '{print $9}'`|cut -d "," -f 17|sort|uniq -c |sort -k2

raindylong · 2012-11-30 16:30:41 5
Find Duplicate Files (based on size first, then MD5 hash)

This is a modified version of the OP, wrapped into a bash function. This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names. It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.
This is sample output - yours may be different.
0

find-duplicates () { find "$@" -not -empty -type f -printf "%s\0" | sort -rnz | uniq -dz | xargs -0 -I{} -n1 find "$@" -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate; }

mpeschke · 2013-01-23 23:20:26 5
Count unique lines in file sorted by instance count (descending) and alphabetically (ascending)

I used to do this sorting with: sort file.txt | uniq -c | sort -nr But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order. Show Sample Output
This is sample output - yours may be different.
```
  24 Poland
  11 Portugal
   8 Australia
   8 USA
   3 Indonesia
   3 Romania
   2 Canada
   2 Slovakia
```
0

sort file.txt | uniq -c | sort -k1nr -k2d

westonruter · 2013-01-28 22:21:05 8
Find Duplicate Files (based on size first, then MD5 hash)

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.
This is sample output - yours may be different.
0

find . -type f -size +0 -printf "%-25s%p\n" | sort -n | uniq -D -w 25 | sed 's/^\w* *$.*$/md5sum "\1"/' | sh | sort | uniq -w32 --all-repeated=separate

jimetc · 2013-02-23 20:44:20 5
Display count of log entries via the previous minute for graphing purposes. Example given is for DHCPREQUESTS on an ISC dhcp service log.
This is sample output - yours may be different.
0

DATE=`date +"%H:%M" --date '-1 min'`; egrep "\ $DATE\:..\ " /var/log/dhcpd.log |awk '/DHCPREQUEST/ {split($3,t,":"); printf("%02d:%02d\n",t[1],t[2]);}' |uniq -c;

jobewan · 2013-03-17 02:23:37 5
list unique file extensions recursively for a path, include extension frequency stats

If you have GNU findutils, you can get only the file name with find /some/path -type f -printf '%f\n' instead of find /some/path -type f | gawk -F/ '{print $NF}' Show Sample Output
This is sample output - yours may be different.
```
100 txt
 25 gz
 10 vim
  3 gitignore
```
0

find /some/path -type f | gawk -F/ '{print $NF}' | gawk -F. '/\./{print $NF}' | sort | uniq -c | sort -rn

skkzsh · 2013-03-18 14:40:26 4
Convert uniq output to json
This is sample output - yours may be different.
0

uniq -c | sed -r 's/([0-9]+)\s(.*)/"\2": \1,/;$s/,/\n}/;1i{'

qistoph · 2013-04-13 16:48:59 7
find top 20 results in apache statistics for a specific month
This is sample output - yours may be different.
0

awk '/Dec\/2012/ {print $1,$8}' logfile | grep -ivE '(.gif|.jpg|.png|favicon|.css|.js|robots.txt|wp-l|wp-term)' | sort | uniq -c | sort -rn | head -n 20

vaaclav · 2013-04-30 06:51:47 6
top 10 of access log

top 10 of access log
This is sample output - yours may be different.
0

awk '{ print $9 }' access.log | sort | uniq -c | sort -nr | head -n 10

cuizhaohua · 2013-05-02 09:23:26 6
how much time restart the wls service?
This is sample output - yours may be different.
0

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

cuizhaohua · 2013-05-16 08:40:57 6
Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded. Show Sample Output
This is sample output - yours may be different.
```
      1 foo@example.com
      3 bar@example.org
    100 baz@example.net
```
0

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

blueskin · 2013-06-04 10:49:14 8
Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits Empty lines are deleted, before the output in format "prefix,ocurrences" Show Sample Output
This is sample output - yours may be different.
```
$ cat SubsxPrefix.csv
52551,392
52842,93
52843,193
```
0

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

neomefistox · 2013-07-17 07:58:56 6
Filter IP's in apache access logs based on use

avoiding UUOC! cut can handle files as well. No neet for a cat.
This is sample output - yours may be different.
0

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

BorneBjoern · 2013-09-17 20:05:03 9
Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts? Show Sample Output
This is sample output - yours may be different.
```
      1 ori
      1 swf
      1 ttf
      1 zip
      2 txt
      4 gz
      8 css
     10 ico
     27 tpl
     28 html
    169 js
    469 php
    590 gif
```
0

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

t3o · 2013-09-26 07:45:19 7
Check where mail was sent from
This is sample output - yours may be different.
0

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

franB · 2013-10-09 20:59:14 7
Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.
This is sample output - yours may be different.
0

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

alafrosty · 2013-10-22 13:34:19 7
Remove duplicate line in a text file.

Remove duplicate line in a text file.
This is sample output - yours may be different.
0

sort in-file.txt | uniq -u > out-file.txt

smed79 · 2014-02-16 07:26:37 11
List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running). Show Sample Output
This is sample output - yours may be different.
```
# /usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head
     17 website899.com
     11 website2.com
      7 website4.com
      7 website8.com
      7 website3.com
      7 website5.com
      6 website7.com
      6 website6.com
      5 website9.com
      5 website79.com
```
0

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 12:31:34 7

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12). Show Sample Output

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 13:24:40 8

最常使用的10个命令

最常使用的10个命令 Show Sample Output
This is sample output - yours may be different.
```
142 cd
    100 ll
     98 tail
     76 vi
     70 service
     53 ps
     52 dstat
     44 cat
     41 stat_peek.sh
     30 ifconfig
```
0

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

cypggs · 2014-04-20 06:40:56 7
Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.
This is sample output - yours may be different.
0

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

metaverse · 2014-04-24 17:18:33 6
Parse logs for IP addresses and how many hits from each IP

Original command: cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100 This is a waste of multiple cats and greps, esp when awk is being used
This is sample output - yours may be different.
0

awk '/text to grep/{print $1}' "log" | sort -n | uniq -c | sort -rn | head -n 100

kln0thing · 2014-07-09 08:48:06 9
Append html-extension to all files in a directory structure that contains html-code.
This is sample output - yours may be different.
0

find . |xargs grep '<html\|<body\|<table' |sed '/~/d;s/:.*//' |sed 's/.*/mv & &.html/' |uniq >run.sh; sh run.sh

mobluse · 2014-07-09 19:20:40 13
‹ First < 6 7 8 9 10 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Get the weather forecast for the next 24 to 48 for your location.

This shell function grabs the weather forecast for the next 24 to 48 hours from weatherunderground.com. Replace <YOURZIPORLOCATION> with your zip code or your "city, state" or "city, country", then calling the function without any arguments returns the weather for that location. Calling the function with a zip code or place name as an argument returns the weather for that location instead of your default. To add a bit of color formatting to the output, use the following instead: $weather(){ curl -s "http://api.wunderground.com/auto/wui/geo/ForecastXML/index.xml?query=${@:-}"|perl -ne '/([^

Clean up the garbage an accidental unzipping makes

List out all the names from the zip file and pass it to xargs utility to delete each one of them

Remove multiple spaces

The command removes all the spaces whithin a file and leaves only one space.

Bruteforce dm-crypt using shell expansion

Lost your luks passphrase? You can always bruteforce from the command line. See the sample output, a simple command for the "pass" word, using combinations of upper/lowercase or number replacement. The generated combinations are: for a in {p,P}{a,A,4}{s,S,5}{s,S,5}; do echo $a; done pass pasS pas5 paSs paSS paS5 ...

Command line calculator

This function defines a command line calculator that handles everything pythons math module can handle, e.g. trigonometric functions, sqrt, log, erf, ... (see http://docs.python.org/library/math.html). It even knows about the constants pi and e.

Recover remote tar backup with ssh

Here how to recover the remote backup over ssh

Be notified about overheating of your CPU and/or motherboard

You'll be notified if your core 1 temperature exceeds 50 degrees, you can change the monitored device by editing the "Core 1" or change the critical temperature by editing the "-gt 50" part. Note: you must have lm-sensors installed and configured in order to get this command working.

Grep only files matching certain pattern (Advanced)

Sort files in folders alphabetically

Creates one letter folders in the current directory and moves files with corresponding initial in the folder.

Quickly generate an MD5 hash for a text string using OpenSSL

Here Strings / A variant of here documents, the format is:

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Commands using uniq (255) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using uniq (255)

sorted by