Commands using uniq sorted by votes

Commands using uniq (255)

sorted by

Show duplicate lines in a file

The following displays only the entries that are duplicates. Show Sample Output
This is sample output - yours may be different.
```
2 Alex Jason:200:Sales
2 Emma Thomas:100:Marketing
```
0

sort namesd.txt | uniq ?cd

ankush108 · 2012-06-26 19:23:58 3
List all the unique extensions in a folder and print their count
This is sample output - yours may be different.
0

find . -type f -print | awk -F'.' '{print $NF}' | sort | uniq -c

vishnu81 · 2012-10-31 12:03:10 5
count & sort one field of the log files

count & sort one field of the log files , such as nginx/apache access log files .
This is sample output - yours may be different.
0

tail -1000 `ls -ltr /var/log/CF* |tail -1|awk '{print $9}'`|cut -d "," -f 17|sort|uniq -c |sort -k2

raindylong · 2012-11-30 16:30:41 5
Find Duplicate Files (based on size first, then MD5 hash)

This is a modified version of the OP, wrapped into a bash function. This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names. It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.
This is sample output - yours may be different.
0

find-duplicates () { find "$@" -not -empty -type f -printf "%s\0" | sort -rnz | uniq -dz | xargs -0 -I{} -n1 find "$@" -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate; }

mpeschke · 2013-01-23 23:20:26 5
Count unique lines in file sorted by instance count (descending) and alphabetically (ascending)

I used to do this sorting with: sort file.txt | uniq -c | sort -nr But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order. Show Sample Output
This is sample output - yours may be different.
```
  24 Poland
  11 Portugal
   8 Australia
   8 USA
   3 Indonesia
   3 Romania
   2 Canada
   2 Slovakia
```
0

sort file.txt | uniq -c | sort -k1nr -k2d

westonruter · 2013-01-28 22:21:05 8
Find Duplicate Files (based on size first, then MD5 hash)

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.
This is sample output - yours may be different.
0

find . -type f -size +0 -printf "%-25s%p\n" | sort -n | uniq -D -w 25 | sed 's/^\w* *$.*$/md5sum "\1"/' | sh | sort | uniq -w32 --all-repeated=separate

jimetc · 2013-02-23 20:44:20 5
Display count of log entries via the previous minute for graphing purposes. Example given is for DHCPREQUESTS on an ISC dhcp service log.
This is sample output - yours may be different.
0

DATE=`date +"%H:%M" --date '-1 min'`; egrep "\ $DATE\:..\ " /var/log/dhcpd.log |awk '/DHCPREQUEST/ {split($3,t,":"); printf("%02d:%02d\n",t[1],t[2]);}' |uniq -c;

jobewan · 2013-03-17 02:23:37 5
list unique file extensions recursively for a path, include extension frequency stats

If you have GNU findutils, you can get only the file name with find /some/path -type f -printf '%f\n' instead of find /some/path -type f | gawk -F/ '{print $NF}' Show Sample Output
This is sample output - yours may be different.
```
100 txt
 25 gz
 10 vim
  3 gitignore
```
0

find /some/path -type f | gawk -F/ '{print $NF}' | gawk -F. '/\./{print $NF}' | sort | uniq -c | sort -rn

skkzsh · 2013-03-18 14:40:26 4
Convert uniq output to json
This is sample output - yours may be different.
0

uniq -c | sed -r 's/([0-9]+)\s(.*)/"\2": \1,/;$s/,/\n}/;1i{'

qistoph · 2013-04-13 16:48:59 7
find top 20 results in apache statistics for a specific month
This is sample output - yours may be different.
0

awk '/Dec\/2012/ {print $1,$8}' logfile | grep -ivE '(.gif|.jpg|.png|favicon|.css|.js|robots.txt|wp-l|wp-term)' | sort | uniq -c | sort -rn | head -n 20

vaaclav · 2013-04-30 06:51:47 6
top 10 of access log

top 10 of access log
This is sample output - yours may be different.
0

awk '{ print $9 }' access.log | sort | uniq -c | sort -nr | head -n 10

cuizhaohua · 2013-05-02 09:23:26 6
how much time restart the wls service?
This is sample output - yours may be different.
0

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

cuizhaohua · 2013-05-16 08:40:57 6
Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded. Show Sample Output
This is sample output - yours may be different.
```
      1 foo@example.com
      3 bar@example.org
    100 baz@example.net
```
0

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

blueskin · 2013-06-04 10:49:14 8
Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits Empty lines are deleted, before the output in format "prefix,ocurrences" Show Sample Output
This is sample output - yours may be different.
```
$ cat SubsxPrefix.csv
52551,392
52842,93
52843,193
```
0

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

neomefistox · 2013-07-17 07:58:56 6
Filter IP's in apache access logs based on use

avoiding UUOC! cut can handle files as well. No neet for a cat.
This is sample output - yours may be different.
0

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

BorneBjoern · 2013-09-17 20:05:03 9
Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts? Show Sample Output
This is sample output - yours may be different.
```
      1 ori
      1 swf
      1 ttf
      1 zip
      2 txt
      4 gz
      8 css
     10 ico
     27 tpl
     28 html
    169 js
    469 php
    590 gif
```
0

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

t3o · 2013-09-26 07:45:19 7
Check where mail was sent from
This is sample output - yours may be different.
0

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

franB · 2013-10-09 20:59:14 7
Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.
This is sample output - yours may be different.
0

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

alafrosty · 2013-10-22 13:34:19 7
Remove duplicate line in a text file.

Remove duplicate line in a text file.
This is sample output - yours may be different.
0

sort in-file.txt | uniq -u > out-file.txt

smed79 · 2014-02-16 07:26:37 11
List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running). Show Sample Output
This is sample output - yours may be different.
```
# /usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head
     17 website899.com
     11 website2.com
      7 website4.com
      7 website8.com
      7 website3.com
      7 website5.com
      6 website7.com
      6 website6.com
      5 website9.com
      5 website79.com
```
0

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 12:31:34 7

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12). Show Sample Output

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 13:24:40 8

最常使用的10个命令

最常使用的10个命令 Show Sample Output
This is sample output - yours may be different.
```
142 cd
    100 ll
     98 tail
     76 vi
     70 service
     53 ps
     52 dstat
     44 cat
     41 stat_peek.sh
     30 ifconfig
```
0

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

cypggs · 2014-04-20 06:40:56 7
Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.
This is sample output - yours may be different.
0

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

metaverse · 2014-04-24 17:18:33 6
Parse logs for IP addresses and how many hits from each IP

Original command: cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100 This is a waste of multiple cats and greps, esp when awk is being used
This is sample output - yours may be different.
0

awk '/text to grep/{print $1}' "log" | sort -n | uniq -c | sort -rn | head -n 100

kln0thing · 2014-07-09 08:48:06 9
Append html-extension to all files in a directory structure that contains html-code.
This is sample output - yours may be different.
0

find . |xargs grep '<html\|<body\|<table' |sed '/~/d;s/:.*//' |sed 's/.*/mv & &.html/' |uniq >run.sh; sh run.sh

mobluse · 2014-07-09 19:20:40 13
‹ First < 6 7 8 9 10 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Convert seconds to [DD:][HH:]MM:SS

Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

clear screen, keep prompt at eye-level (faster than clear(1), tput cl, etc.)

this leaves the cursor at the bottom of the terminal screen, where your eyes are. ctrl-l moves it to the top, forcing you to look up.

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

True Random Dice Roll

/dev/urandom is cryptographically secure, and indistinguishable from true random, as it gathers data from external sources, influenced by human timing interactions with computers, to fill the entropy pool, and hashes the input with SHA-1. As such, this is a quick way to do a "true random" fair-6 dice roll. Using this method, you could easily create passphrases with Diceware http://diceware.com. Change the head(1) count to something other than 5 for more or less numbers.

Batch edition of all OpenOffice.org Writer files in the current directory (body text)

This function does a batch edition of all OOO3 Writer files in current directory. It uses sed to search a FOO pattern into body text of each file, then replace it to foo pattern (only the first match) . I did it because I've some hundreds of OOO3 Writer files where I did need to edit one word in each ones and open up each file in OOO3 gui wasn't an option. Usage: bsro3 FOO foo

Find usb device in realtime

Using this command you can track a moment when usb device was attached.

analyze traffic remotely over ssh w/ wireshark

This captures traffic on a remote machine with tshark, sends the raw pcap data over the ssh link, and displays it in wireshark. Hitting ctrl+C will stop the capture and unfortunately close your wireshark window. This can be worked-around by passing -c # to tshark to only capture a certain # of packets, or redirecting the data through a named pipe rather than piping directly from ssh to wireshark. I recommend filtering as much as you can in the tshark command to conserve bandwidth. tshark can be replaced with tcpdump thusly: $ ssh root@example.com tcpdump -w - 'port !22' | wireshark -k -i -

Find the cover image for an album

This bash function uses albumart.org to find the cover for an album. It returns an amazon.com url to the image. Usage: albumart [artist] [album] These arguments can be reversed and if the album name is distinct enough, it may be possible to omit the artist. The command can be extended with wget to automatically download the matching image like this: $ albumart(){ local x y="$@";x=$(awk '/View larger image/{gsub(/^.*largeImagePopup\(.|., .*$/,"");print;exit}'

Display your ${PATH}, one directory per line

This works in bash and zsh. You may also want to alias it, if you need to look at it often... $ alias lpath="echo \$PATH | tr : \\\\n" "\$PATH" to make sure to look at your current $PATH

Create a mirror of a local folder, on a remote server

Create a exact mirror of the local folder "/root/files", on remote server 'remote_server' using SSH command (listening on port 22) (all files & folders on destination server/folder will be deleted)

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Commands using uniq (255) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using uniq (255)

sorted by