Commands using uniq sorted by votes

Commands using uniq (255)

sorted by

Show duplicate lines in a file

The following displays only the entries that are duplicates. Show Sample Output
This is sample output - yours may be different.
```
2 Alex Jason:200:Sales
2 Emma Thomas:100:Marketing
```
0

sort namesd.txt | uniq ?cd

ankush108 · 2012-06-26 19:23:58 3
List all the unique extensions in a folder and print their count
This is sample output - yours may be different.
0

find . -type f -print | awk -F'.' '{print $NF}' | sort | uniq -c

vishnu81 · 2012-10-31 12:03:10 5
count & sort one field of the log files

count & sort one field of the log files , such as nginx/apache access log files .
This is sample output - yours may be different.
0

tail -1000 `ls -ltr /var/log/CF* |tail -1|awk '{print $9}'`|cut -d "," -f 17|sort|uniq -c |sort -k2

raindylong · 2012-11-30 16:30:41 5
Find Duplicate Files (based on size first, then MD5 hash)

This is a modified version of the OP, wrapped into a bash function. This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names. It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.
This is sample output - yours may be different.
0

find-duplicates () { find "$@" -not -empty -type f -printf "%s\0" | sort -rnz | uniq -dz | xargs -0 -I{} -n1 find "$@" -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate; }

mpeschke · 2013-01-23 23:20:26 5
Count unique lines in file sorted by instance count (descending) and alphabetically (ascending)

I used to do this sorting with: sort file.txt | uniq -c | sort -nr But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order. Show Sample Output
This is sample output - yours may be different.
```
  24 Poland
  11 Portugal
   8 Australia
   8 USA
   3 Indonesia
   3 Romania
   2 Canada
   2 Slovakia
```
0

sort file.txt | uniq -c | sort -k1nr -k2d

westonruter · 2013-01-28 22:21:05 8
Find Duplicate Files (based on size first, then MD5 hash)

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.
This is sample output - yours may be different.
0

find . -type f -size +0 -printf "%-25s%p\n" | sort -n | uniq -D -w 25 | sed 's/^\w* *$.*$/md5sum "\1"/' | sh | sort | uniq -w32 --all-repeated=separate

jimetc · 2013-02-23 20:44:20 5
Display count of log entries via the previous minute for graphing purposes. Example given is for DHCPREQUESTS on an ISC dhcp service log.
This is sample output - yours may be different.
0

DATE=`date +"%H:%M" --date '-1 min'`; egrep "\ $DATE\:..\ " /var/log/dhcpd.log |awk '/DHCPREQUEST/ {split($3,t,":"); printf("%02d:%02d\n",t[1],t[2]);}' |uniq -c;

jobewan · 2013-03-17 02:23:37 5
list unique file extensions recursively for a path, include extension frequency stats

If you have GNU findutils, you can get only the file name with find /some/path -type f -printf '%f\n' instead of find /some/path -type f | gawk -F/ '{print $NF}' Show Sample Output
This is sample output - yours may be different.
```
100 txt
 25 gz
 10 vim
  3 gitignore
```
0

find /some/path -type f | gawk -F/ '{print $NF}' | gawk -F. '/\./{print $NF}' | sort | uniq -c | sort -rn

skkzsh · 2013-03-18 14:40:26 4
Convert uniq output to json
This is sample output - yours may be different.
0

uniq -c | sed -r 's/([0-9]+)\s(.*)/"\2": \1,/;$s/,/\n}/;1i{'

qistoph · 2013-04-13 16:48:59 7
find top 20 results in apache statistics for a specific month
This is sample output - yours may be different.
0

awk '/Dec\/2012/ {print $1,$8}' logfile | grep -ivE '(.gif|.jpg|.png|favicon|.css|.js|robots.txt|wp-l|wp-term)' | sort | uniq -c | sort -rn | head -n 20

vaaclav · 2013-04-30 06:51:47 6
top 10 of access log

top 10 of access log
This is sample output - yours may be different.
0

awk '{ print $9 }' access.log | sort | uniq -c | sort -nr | head -n 10

cuizhaohua · 2013-05-02 09:23:26 6
how much time restart the wls service?
This is sample output - yours may be different.
0

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

cuizhaohua · 2013-05-16 08:40:57 6
Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded. Show Sample Output
This is sample output - yours may be different.
```
      1 foo@example.com
      3 bar@example.org
    100 baz@example.net
```
0

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

blueskin · 2013-06-04 10:49:14 8
Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits Empty lines are deleted, before the output in format "prefix,ocurrences" Show Sample Output
This is sample output - yours may be different.
```
$ cat SubsxPrefix.csv
52551,392
52842,93
52843,193
```
0

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

neomefistox · 2013-07-17 07:58:56 6
Filter IP's in apache access logs based on use

avoiding UUOC! cut can handle files as well. No neet for a cat.
This is sample output - yours may be different.
0

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

BorneBjoern · 2013-09-17 20:05:03 9
Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts? Show Sample Output
This is sample output - yours may be different.
```
      1 ori
      1 swf
      1 ttf
      1 zip
      2 txt
      4 gz
      8 css
     10 ico
     27 tpl
     28 html
    169 js
    469 php
    590 gif
```
0

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

t3o · 2013-09-26 07:45:19 7
Check where mail was sent from
This is sample output - yours may be different.
0

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

franB · 2013-10-09 20:59:14 7
Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.
This is sample output - yours may be different.
0

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

alafrosty · 2013-10-22 13:34:19 7
Remove duplicate line in a text file.

Remove duplicate line in a text file.
This is sample output - yours may be different.
0

sort in-file.txt | uniq -u > out-file.txt

smed79 · 2014-02-16 07:26:37 11
List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running). Show Sample Output
This is sample output - yours may be different.
```
# /usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head
     17 website899.com
     11 website2.com
      7 website4.com
      7 website8.com
      7 website3.com
      7 website5.com
      6 website7.com
      6 website6.com
      5 website9.com
      5 website79.com
```
0

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 12:31:34 7

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12). Show Sample Output

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

copocaneta · 2014-03-12 13:24:40 8

最常使用的10个命令

最常使用的10个命令 Show Sample Output
This is sample output - yours may be different.
```
142 cd
    100 ll
     98 tail
     76 vi
     70 service
     53 ps
     52 dstat
     44 cat
     41 stat_peek.sh
     30 ifconfig
```
0

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

cypggs · 2014-04-20 06:40:56 7
Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.
This is sample output - yours may be different.
0

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

metaverse · 2014-04-24 17:18:33 6
Parse logs for IP addresses and how many hits from each IP

Original command: cat "log" | grep "text to grep" | awk '{print $1}' | sort -n | uniq -c | sort -rn | head -n 100 This is a waste of multiple cats and greps, esp when awk is being used
This is sample output - yours may be different.
0

awk '/text to grep/{print $1}' "log" | sort -n | uniq -c | sort -rn | head -n 100

kln0thing · 2014-07-09 08:48:06 9
Append html-extension to all files in a directory structure that contains html-code.
This is sample output - yours may be different.
0

find . |xargs grep '<html\|<body\|<table' |sed '/~/d;s/:.*//' |sed 's/.*/mv & &.html/' |uniq >run.sh; sh run.sh

mobluse · 2014-07-09 19:20:40 13
‹ First < 6 7 8 9 10 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Redirect STDIN

Several times, I find myself hitting my up arrow, and changing the search term. Unfortunately, I find myself wasting too much time typing: $ grep kernel /var/log/messages Redirecting STDIN allows me to put the search term at the end so I less cursor movement to change what I'm searching for: $ < /var/log/messages grep kernel If you're using the emacs keyboard binding, then after you press your up arrow, press CTRL+w to erase the word. If this has already been submitted, I couldn't find it with the search utility.

Submit data to a HTML form with POST method and save the response

Assume that you have a form , in the source look for something similar to : input name="rid" type="TEXT" input name="submit" value="SUBMIT" type="SUBMIT" align="center" Then exec the command to get the response into html More info : www.h3manth.com

Find usb device in realtime

Using this command you can track a moment when usb device was attached.

find previously entered commands (requires configuring .inputrc)

[Click the "show sample output" link to see how to use this keystroke.] Meta-p is one of my all time most used and most loved features of working at the command line. It's also one that surprisingly few people know about. To use it with bash (actually in any readline application), you'll need to add a couple lines to your .inputrc then have bash reread the .inputrc using the bind command: $ echo '"\en": history-search-forward' >> ~/.inputrc $ echo '"\ep": history-search-backward' >> ~/.inputrc $ bind -f ~/.inputrc I first learned about this feature in tcsh. When I switched over to bash about fifteen years ago, I had assumed I'd prefer ^R to search in reverse. Intuitively ^R seemed better since you could search for an argument instead of a command. I think that, like using a microkernel for the Hurd, it sounded so obviously right fifteen years ago, but that was only because the older way had benefits we hadn't known about. I think many of you who use the command line as much as I do know that we can just be thinking about what results we want and our fingers will start typing the commands needed. I assume it's some sort of parallel processing going on with the linguistic part of the brain. Unfortunately, that parallelism doesn't seem to work (at least for me) with searching the history. I realize I can save myself typing using the history shortly after my fingers have already started "speaking". But, when I hit ^R in Bash, everything I've already typed gets ignored and I have to stop and think again about what I was doing. It's a small bump in the road but it can be annoying, especially for long-time command line users. Usually M-p is exactly what I need to save myself time and trouble. If you use the command line a lot, please give Meta-p a try. You may be surprised how it frees your brain to process more smoothly in parallel. (Or maybe it won't. Post here and let me know either way. ☺)

Get the Volume labels all bitlocker volumes had before being encrypted

Get information of volume labels of bitlocker volumes, even if they are encrypted and locked (no access to filesystem, no password provided). Note that the volume labels can have spaces, but only if you name then before encryption. Renaming a bitlocker partition after being encrypted does not have the same effect as doing it before.

Play random music from blip.fm

decoding Active Directory date format

When Ldapsearch queries an Active directory server, all the dates are shown using a timestamp of 18 digits. This perl regexp decodes them in a more human friendly notation. 11644473600 corresponds to some microsoft epoch.

Plot frequency distribution of words from files on a terminal.

Uses the dumb terminal option in gnuplot to plot a graph of frequencies. In this case, we are looking at a frequency analysis of words in all of the .c files.

Open (in vim) all modified files in a git repository

The option --porcelain makes the output of git easier to parse. This one-liner may not work if there is a space in the modified file name.

Find and copy scattered mp3 files into one directory

No problem with word splitting. That should works on many Unix likes.

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Commands using uniq (255) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using uniq (255)

sorted by