Commands using uniq sorted by votes

Commands using uniq (255)

sorted by

Show duplicate lines in a file

The following displays only the entries that are duplicates. Show Sample Output
This is sample output - yours may be different.
```
2 Alex Jason:200:Sales
2 Emma Thomas:100:Marketing
```
0

sort namesd.txt | uniq ?cd

ankush108 · 2012-06-26 19:23:58 3
List all the unique extensions in a folder and print their count
This is sample output - yours may be different.
0

find . -type f -print | awk -F'.' '{print $NF}' | sort | uniq -c

vishnu81 · 2012-10-31 12:03:10 5
count & sort one field of the log files

count & sort one field of the log files , such as nginx/apache access log files .
This is sample output - yours may be different.
0

tail -1000 `ls -ltr /var/log/CF* |tail -1|awk '{print $9}'`|cut -d "," -f 17|sort|uniq -c |sort -k2

raindylong · 2012-11-30 16:30:41 5
Find Duplicate Files (based on size first, then MD5 hash)

This is a modified version of the OP, wrapped into a bash function. This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names. It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.
This is sample output - yours may be different.
0

find-duplicates () { find "$@" -not -empty -type f -printf "%s\0" | sort -rnz | uniq -dz | xargs -0 -I{} -n1 find "$@" -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate; }

mpeschke · 2013-01-23 23:20:26 5
Count unique lines in file sorted by instance count (descending) and alphabetically (ascending)

I used to do this sorting with: sort file.txt | uniq -c | sort -nr But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order. Show Sample Output
This is sample output - yours may be different.
```
  24 Poland
  11 Portugal
   8 Australia
   8 USA
   3 Indonesia
   3 Romania
   2 Canada
   2 Slovakia
```
0

sort file.txt | uniq -c | sort -k1nr -k2d

westonruter · 2013-01-28 22:21:05 8
Find Duplicate Files (based on size first, then MD5 hash)

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.
This is sample output - yours may be different.
0

find . -type f -size +0 -printf "%-25s%p\n" | sort -n | uniq -D -w 25 | sed 's/^\w* *$.*$/md5sum "\1"/' | sh | sort | uniq -w32 --all-repeated=separate

jimetc · 2013-02-23 20:44:20 5
Display count of log entries via the previous minute for graphing purposes. Example given is for DHCPREQUESTS on an ISC dhcp service log.
This is sample output - yours may be different.
0

DATE=`date +"%H:%M" --date '-1 min'`; egrep "\ $DATE\:..\ " /var/log/dhcpd.log |awk '/DHCPREQUEST/ {split($3,t,":"); printf("%02d:%02d\n",t[1],t[2]);}' |uniq -c;

jobewan · 2013-03-17 02:23:37 5
list unique file extensions recursively for a path, include extension frequency stats

If you have GNU findutils, you can get only the file name with find /some/path -type f -printf '%f\n' instead of find /some/path -type f | gawk -F/ '{print $NF}' Show Sample Output
This is sample output - yours may be different.
```
100 txt
 25 gz
 10 vim
  3 gitignore
```
0

find /some/path -type f | gawk -F/ '{print $NF}' | gawk -F. '/\./{print $NF}' | sort | uniq -c | sort -rn

skkzsh · 2013-03-18 14:40:26 4
Convert uniq output to json
This is sample output - yours may be different.
0

uniq -c | sed -r 's/([0-9]+)\s(.*)/"\2": \1,/;$s/,/\n}/;1i{'

qistoph · 2013-04-13 16:48:59 7
find top 20 results in apache statistics for a specific month
This is sample output - yours may be different.
0

awk '/Dec\/2012/ {print $1,$8}' logfile | grep -ivE '(.gif|.jpg|.png|favicon|.css|.js|robots.txt|wp-l|wp-term)' | sort | uniq -c | sort -rn | head -n 20

vaaclav · 2013-04-30 06:51:47 6
top 10 of access log

top 10 of access log
This is sample output - yours may be different.
0

awk '{ print $9 }' access.log | sort | uniq -c | sort -nr | head -n 10

cuizhaohua · 2013-05-02 09:23:26 6
how much time restart the wls service?
This is sample output - yours may be different.
0

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

cuizhaohua · 2013-05-16 08:40:57 6
Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded. Show Sample Output
This is sample output - yours may be different.
```
      1 foo@example.com
      3 bar@example.org
    100 baz@example.net
```
0

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

blueskin · 2013-06-04 10:49:14 8
Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits Empty lines are deleted, before the output in format "prefix,ocurrences" Show Sample Output
This is sample output - yours may be different.
```
$ cat SubsxPrefix.csv
52551,392
52842,93
52843,193
```
0

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

neomefistox · 2013-07-17 07:58:56 6
Filter IP's in apache access logs based on use

avoiding UUOC! cut can handle files as well. No neet for a cat.
This is sample output - yours may be different.
0

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

BorneBjoern · 2013-09-17 20:05:03 9
Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts? Show Sample Output
This is sample output - yours may be different.
```
      1 ori
      1 swf
      1 ttf
      1 zip
      2 txt
      4 gz
      8 css
     10 ico
     27 tpl
     28 html
    169 js
    469 php
    590 gif
```
0

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

t3o · 2013-09-26 07:45:19 7
Check where mail was sent from
This is sample output - yours may be different.
0

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

franB · 2013-10-09 20:59:14 7
Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.
This is sample output - yours may be different.
0

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

alafrosty · 2013-10-22 13:34:19 7
Remove duplicate line in a text file.

Remove duplicate line in a text file.
This is sample output - yours may be different.
0

sort in-file.txt | uniq -u > out-file.txt

smed79 · 2014-02-16 07:26:37 11
List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running). Show Sample Output
This is sample output - yours may be different.
```
# /usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head
     17 website899.com
     11 website2.com
      7 website4.com
      7 website8.com
      7 website3.com
      7 website5.com
      6 website7.com
      6 website6.com
      5 website9.com
      5 website79.com
```
0

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 12:31:34 7

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12). Show Sample Output

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 13:24:40 8

最常使用的10个命令

最常使用的10个命令 Show Sample Output
This is sample output - yours may be different.
```
142 cd
    100 ll
     98 tail
     76 vi
     70 service
     53 ps
     52 dstat
     44 cat
     41 stat_peek.sh
     30 ifconfig
```
0

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

cypggs · 2014-04-20 06:40:56 7
Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.
This is sample output - yours may be different.
0

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

metaverse · 2014-04-24 17:18:33 6
Parse logs for IP addresses and how many hits from each IP

Original command: cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100 This is a waste of multiple cats and greps, esp when awk is being used
This is sample output - yours may be different.
0

awk '/text to grep/{print $1}' "log" | sort -n | uniq -c | sort -rn | head -n 100

kln0thing · 2014-07-09 08:48:06 9
Append html-extension to all files in a directory structure that contains html-code.
This is sample output - yours may be different.
0

find . |xargs grep '<html\|<body\|<table' |sed '/~/d;s/:.*//' |sed 's/.*/mv & &.html/' |uniq >run.sh; sh run.sh

mobluse · 2014-07-09 19:20:40 13
‹ First < 6 7 8 9 10 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Set laptop display brightness

Run as root. Path may vary depending on laptop model and video card (this was tested on an Acer laptop with ATI HD3200 video). $ cat /proc/acpi/video/VGA/LCD/brightness to discover the possible values for your display.

Find writable files

Have a grudge against someone on your network? Do a "find -writable" in their directory and see what you can vandalize! But seriously, this is really useful to check the files in your own home directory to make sure they can't inadvertently be changed by someone else's wayward script.

Set up alarm with fade-in, for graceful awakening

There are a lot of commands, which invokes your player at specified time. But I prefer not to jump from by bed, when alarm start to play. Instead, this script increases volume of mpd over time, which much more pleasant when you just woke up :)

Outgoing IP of server

Faster than the curl/wget-approaches.

Advanced python tracing

Trace python statement execution and syscalls invoked during that simultaneously

List only directory names

This command would be useful when it is desirable to list only the directories. Other options Hidden directory $ ls -d .*/ Other path $ ls -d /path/to/top/directory/.*/ Long format: $ ls -ld */

processes per user counter

enumerates the number of processes for each user. ps BSD format is used here , for standard Unix format use : ps -eLf |awk '{$1} {++P[$1]} END {for(a in P) if (a !="UID") print a,P[a]}'

How to establish a remote Gnu screen session that you can re-connect to

Long before tabbed terminals existed, people have been using Gnu screen to open many shells in a single text terminal. Combined with ssh, it gives you the ability to have many open shells with a single remote connection using the above options. If you detach with "Ctrl-a d" or if the ssh session is accidentally terminated, all processes running in your remote shells remain undisturbed, ready for you to reconnect. Other useful screen commands are "Ctrl-a c" (open new shell) and "Ctrl-a a" (alternate between shells). Read this quick reference for more screen commands: http://aperiodic.net/screen/quick_reference

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Detect illegal access to kernel space, potentially useful for Meltdown detection

Based on capsule8 agent examples, not rigorously tested

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Commands using uniq (255) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using uniq (255)

sorted by