Commands using uniq sorted by votes

Commands using uniq (255)

sorted by

Show duplicate lines in a file

The following displays only the entries that are duplicates. Show Sample Output
This is sample output - yours may be different.
```
2 Alex Jason:200:Sales
2 Emma Thomas:100:Marketing
```
0

sort namesd.txt | uniq ?cd

ankush108 · 2012-06-26 19:23:58 3
List all the unique extensions in a folder and print their count
This is sample output - yours may be different.
0

find . -type f -print | awk -F'.' '{print $NF}' | sort | uniq -c

vishnu81 · 2012-10-31 12:03:10 5
count & sort one field of the log files

count & sort one field of the log files , such as nginx/apache access log files .
This is sample output - yours may be different.
0

tail -1000 `ls -ltr /var/log/CF* |tail -1|awk '{print $9}'`|cut -d "," -f 17|sort|uniq -c |sort -k2

raindylong · 2012-11-30 16:30:41 5
Find Duplicate Files (based on size first, then MD5 hash)

This is a modified version of the OP, wrapped into a bash function. This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names. It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.
This is sample output - yours may be different.
0

find-duplicates () { find "$@" -not -empty -type f -printf "%s\0" | sort -rnz | uniq -dz | xargs -0 -I{} -n1 find "$@" -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate; }

mpeschke · 2013-01-23 23:20:26 5
Count unique lines in file sorted by instance count (descending) and alphabetically (ascending)

I used to do this sorting with: sort file.txt | uniq -c | sort -nr But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order. Show Sample Output
This is sample output - yours may be different.
```
  24 Poland
  11 Portugal
   8 Australia
   8 USA
   3 Indonesia
   3 Romania
   2 Canada
   2 Slovakia
```
0

sort file.txt | uniq -c | sort -k1nr -k2d

westonruter · 2013-01-28 22:21:05 8
Find Duplicate Files (based on size first, then MD5 hash)

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.
This is sample output - yours may be different.
0

find . -type f -size +0 -printf "%-25s%p\n" | sort -n | uniq -D -w 25 | sed 's/^\w* *$.*$/md5sum "\1"/' | sh | sort | uniq -w32 --all-repeated=separate

jimetc · 2013-02-23 20:44:20 5
Display count of log entries via the previous minute for graphing purposes. Example given is for DHCPREQUESTS on an ISC dhcp service log.
This is sample output - yours may be different.
0

DATE=`date +"%H:%M" --date '-1 min'`; egrep "\ $DATE\:..\ " /var/log/dhcpd.log |awk '/DHCPREQUEST/ {split($3,t,":"); printf("%02d:%02d\n",t[1],t[2]);}' |uniq -c;

jobewan · 2013-03-17 02:23:37 5
list unique file extensions recursively for a path, include extension frequency stats

If you have GNU findutils, you can get only the file name with find /some/path -type f -printf '%f\n' instead of find /some/path -type f | gawk -F/ '{print $NF}' Show Sample Output
This is sample output - yours may be different.
```
100 txt
 25 gz
 10 vim
  3 gitignore
```
0

find /some/path -type f | gawk -F/ '{print $NF}' | gawk -F. '/\./{print $NF}' | sort | uniq -c | sort -rn

skkzsh · 2013-03-18 14:40:26 4
Convert uniq output to json
This is sample output - yours may be different.
0

uniq -c | sed -r 's/([0-9]+)\s(.*)/"\2": \1,/;$s/,/\n}/;1i{'

qistoph · 2013-04-13 16:48:59 7
find top 20 results in apache statistics for a specific month
This is sample output - yours may be different.
0

awk '/Dec\/2012/ {print $1,$8}' logfile | grep -ivE '(.gif|.jpg|.png|favicon|.css|.js|robots.txt|wp-l|wp-term)' | sort | uniq -c | sort -rn | head -n 20

vaaclav · 2013-04-30 06:51:47 6
top 10 of access log

top 10 of access log
This is sample output - yours may be different.
0

awk '{ print $9 }' access.log | sort | uniq -c | sort -nr | head -n 10

cuizhaohua · 2013-05-02 09:23:26 6
how much time restart the wls service?
This is sample output - yours may be different.
0

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

cuizhaohua · 2013-05-16 08:40:57 6
Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded. Show Sample Output
This is sample output - yours may be different.
```
      1 foo@example.com
      3 bar@example.org
    100 baz@example.net
```
0

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

blueskin · 2013-06-04 10:49:14 8
Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits Empty lines are deleted, before the output in format "prefix,ocurrences" Show Sample Output
This is sample output - yours may be different.
```
$ cat SubsxPrefix.csv
52551,392
52842,93
52843,193
```
0

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

neomefistox · 2013-07-17 07:58:56 6
Filter IP's in apache access logs based on use

avoiding UUOC! cut can handle files as well. No neet for a cat.
This is sample output - yours may be different.
0

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

BorneBjoern · 2013-09-17 20:05:03 9
Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts? Show Sample Output
This is sample output - yours may be different.
```
      1 ori
      1 swf
      1 ttf
      1 zip
      2 txt
      4 gz
      8 css
     10 ico
     27 tpl
     28 html
    169 js
    469 php
    590 gif
```
0

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

t3o · 2013-09-26 07:45:19 7
Check where mail was sent from
This is sample output - yours may be different.
0

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

franB · 2013-10-09 20:59:14 7
Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.
This is sample output - yours may be different.
0

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

alafrosty · 2013-10-22 13:34:19 7
Remove duplicate line in a text file.

Remove duplicate line in a text file.
This is sample output - yours may be different.
0

sort in-file.txt | uniq -u > out-file.txt

smed79 · 2014-02-16 07:26:37 11
List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running). Show Sample Output
This is sample output - yours may be different.
```
# /usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head
     17 website899.com
     11 website2.com
      7 website4.com
      7 website8.com
      7 website3.com
      7 website5.com
      6 website7.com
      6 website6.com
      5 website9.com
      5 website79.com
```
0

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 12:31:34 7

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12). Show Sample Output

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 13:24:40 8

最常使用的10个命令

最常使用的10个命令 Show Sample Output
This is sample output - yours may be different.
```
142 cd
    100 ll
     98 tail
     76 vi
     70 service
     53 ps
     52 dstat
     44 cat
     41 stat_peek.sh
     30 ifconfig
```
0

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

cypggs · 2014-04-20 06:40:56 7
Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.
This is sample output - yours may be different.
0

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

metaverse · 2014-04-24 17:18:33 6
Parse logs for IP addresses and how many hits from each IP

Original command: cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100 This is a waste of multiple cats and greps, esp when awk is being used
This is sample output - yours may be different.
0

awk '/text to grep/{print $1}' "log" | sort -n | uniq -c | sort -rn | head -n 100

kln0thing · 2014-07-09 08:48:06 9
Append html-extension to all files in a directory structure that contains html-code.
This is sample output - yours may be different.
0

find . |xargs grep '<html\|<body\|<table' |sed '/~/d;s/:.*//' |sed 's/.*/mv & &.html/' |uniq >run.sh; sh run.sh

mobluse · 2014-07-09 19:20:40 13
‹ First < 6 7 8 9 10 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

ZSH prompt. ':)' after program execution with no error, ':(' after failure.

sorry for my english

Convert seconds to [DD:][HH:]MM:SS

Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Get a list of the erroring cifs entries in fstab

It disturbs me when my logwatch report tells me a share or machine has disappeared, esp as mount isn't telling me what's gone. This command outputs to stderr the erroring cifs entries from fstab.

Shows space used by each directory of the root filesystem excluding mountpoints/external filesystems (and sort the output)

Excludes other mountpoints with acavagni's "mountpoint" idea, but with -exec instead of piping to an xargs subshell. Then, calling "du" only once with -exec's "+" option. The first "\! -exec" acts as a test so only those who match are passed to the second "-exec" for du.

Sort output by length of line

This provides a way to sort output based on the length of the line, so that shorter lines appear before longer lines. It's an addon to the sort that I've wanted for years, sometimes it's very useful. Taken from my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html

Copy data using gtar

It copies the entire current working directory to the destination directory with compression enabled.

New files from parts of current buffer

On command mode in Vim you can save parts of the current buffer in another file. * The 'n' value represents the first line of the new file. * The 'm' value represents the last line of the new file. * newfile.txt is the newfile. The results are similar to this command in perl: $ perl -ne 'print if n..m' in.sql > out.sql

Count Files in a Directory with Wildcards.

Remove the '-maxdepth 1' option if you want to count in directories as well

Show your current network interface in use

Find usb device in realtime

Using this command you can track a moment when usb device was attached.

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Commands using uniq (255) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using uniq (255)

sorted by