Commands using uniq | commandlinefu.com

Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

OpenID

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

UpGuard checks and validates configurations for every major OS, network device, and cloud provider.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using uniq

Hide

News

May 19, 2015 - A Look At The New Commandlinefu: I've put together a short writeup on what kind of newness you can expect from the next iteration of clfu. Check it out here.
March 2, 2015 - New Management: I'm Jon, I'll be maintaining and improving clfu. Thanks to David for building such a great resource!

Hide

Top Tags

awk, bash, curl, find, grep, Linux, sed

Hide

Functions

Terminal - Commands using uniq - 218 results

strings * |grep -v "Apple" |grep http |uniq |sed "s/<[^>]\+>//g"

2014-04-24 17:18:33

User: metaverse

Functions: grep sed strings uniq

Up
Down

Convert all .weblock files in present directory (Apple url) to a url on the stdout.

Convert all .weblock files (Apple url) to a url on the stdout.

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

history |awk '{print $3}' |awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c |sort -rn |head -10

2014-04-20 06:40:56

User: cypggs

Functions: awk head sort uniq

Up
Down

最常使用的10个命令

dmesg | grep -Po 'csum failed ino\S* \d+' | sort | uniq | xargs -n 3 find / -inum 2> /dev/null

2014-03-20 06:27:15

User: Sepero

Functions: dmesg find grep sort uniq xargs

Tags: find inode btrfs

-1

Up
Down

Btrfs: Find file names with checksum errors

Btrfs reports the inode numbers of files with failed checksums. Use `find` to lookup the file names of those inodes.

Comments (7) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12 $14}' | sort | uniq -c | sort -rn | head

2014-03-12 13:24:40

User: copocaneta

Functions: awk grep sort uniq

Tags: process cpanel centos busy script busiest scripts

Up
Down

List the busiest script running on a cPanel server (showing domain)

List the busiest scripts/files running on a cPanel server with domain showing (column $12).

netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' |sed -e 's/::ffff://' | cut -f1 -d: | sort | uniq -c | sort -rn | head

2014-03-12 12:43:07

User: copocaneta

Functions: awk cut grep netstat sed sort uniq

Tags: port 80 cpanel centos

Up
Down

IP addresses connected to port 80

IP addresses and number of connections connected to port 80.

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | awk 'BEGIN { FS = " " } ; { print $12 }' | sed '/^$/d' | sort | uniq -c | sort -n

2014-03-12 12:38:24

User: copocaneta

Functions: awk sed sort uniq

Tags: cpanel connections centos

Up
Down

Number of connections to domains on a cPanel server

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

/usr/bin/lynx -dump -width 500 http://127.0.0.1/whm-server-status | grep GET | awk '{print $12}' | sort | uniq -c | sort -rn | head

2014-03-12 12:31:34

User: copocaneta

Functions: awk grep sort uniq

Up
Down

List the busiest websites on a cPanel server

Easiest way to obtain the busiest website list (sorted by number of process running).

grep -o -P '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\:[0-9]{1,5}\s->\s{5}[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\:[0-9]{1,5}' <capture file> | tr -d ' ' | sed 's/:.....//g' | sort -n | uniq -c | sort -nr

2014-03-05 21:34:42

User: santizo

Functions: grep sed sort tr uniq

Up
Down

Sort netflow packet capture

Sort netflow packet capture by unique connections excluding source port.

git log origin/master..master --name-only --pretty="format:" | sort | uniq | xargs tar -rf mytarfile.tar

2014-03-04 12:17:29

User: sarfraznawaz2005

Functions: sort tar uniq xargs

Up
Down

Export unpushed files list

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

cat /proc/cpuinfo | grep BogoMIPS | uniq | sed 's/^.*://g' | awk '{print($1 / 4) }'

2014-03-01 03:44:03

User: derpat

Functions: awk cat grep sed uniq

Tags: cat sed awk uniq grep /proc/cpuinfo BogoMIPS

Up
Down

calculate how much bogomips one cpu core has (assuming you have 4 cores).

on some distro's you have to replace "BogoMIPS" with "bogomips".

sort in-file.txt | uniq -u > out-file.txt

2014-02-16 07:26:37

User: SMed79

Functions: sort uniq

Tags: sort uniq duplicate text

Up
Down

Remove duplicate line in a text file.

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

2014-01-30 11:37:27

User: icefyre

Functions: grep ls sort tr uniq

Up
Down

count files by type

displays a list of all file extensions in current directory and how many files there are of each type of extension in ascending order (case insensitive)

find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"

2013-10-22 13:34:19

User: alafrosty

Functions: awk cut find sed sort uniq xargs

Tags: size find md5sum fdupes duplicate files

Up
Down

Find Duplicate Files (based on size first, then MD5 hash)

* Find all file sizes and file names from the current directory down (replace "." with a target directory as needed).

* sort the file sizes in numeric order

* List only the duplicated file sizes

* drop the file sizes so there are simply a list of files (retain order)

* calculate md5sums on all of the files

* replace the first instance of two spaces (md5sum output) with a \0

* drop the unique md5sums so only duplicate files remain listed

* Use AWK to aggregate identical files on one line.

* Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit).

>>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

2013-10-09 20:59:14

User: franB

Functions: awk grep sort uniq

Up
Down

Check where mail was sent from

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

find . -type f | perl -ne 'chop(); $ext = substr($_, rindex($_, ".") + 1); print "$ext\n";' | sort | uniq --count | sort -n

2013-09-26 07:45:19

User: t3o

Functions: find perl sort uniq

Up
Down

Find used fileextensions in a project

When trying to find an error in a hosted project it's interesting to find out how the source is organized: Are there .inc files? Or .php files only? Or .xml files that probably contain translated texts?

cut -d " " -f1,4 access_log | sort | uniq -c | sort -rn | head

2013-09-20 21:29:32

User: zlemini

Functions: cut sort uniq

Up
Down

Extract busiest times from apache access log

Analyze an Apache access log for the time period with most activity and display the hit count, requesting IP and the timestamp. May help detect a brute force dos attack.

Comments (1) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n

2013-09-17 20:05:03

User: BorneBjoern

Functions: cut sort uniq

Tags: IP apache2 filter requests

Up
Down

Filter IP's in apache access logs based on use

avoiding UUOC!

cut can handle files as well. No neet for a cat.

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

grep "10/Sep/2013" access.log| cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | awk '{ if ($1 > 10) print $0}'

2013-09-12 08:45:38

User: bormarek

Functions: awk cut grep sort uniq

Up
Down

apache statistics

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

netstat -ntu | awk ' $5 ~ /^(::ffff:|[0-9|])/ { gsub("::ffff:","",$5); print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

2013-09-10 19:28:06

User: mrwulf

Functions: awk cut netstat sort uniq

Tags: sort awk uniq IP netstat cut

Up
Down

netstat with group by (ip adress)

Same as the rest, but handle IPv6 short IPs. Also, sort in the order that you're probably looking for.

cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n

2013-09-07 23:57:31

User: while0pass

Functions: cat cut sort uniq

Tags: IP apache2 filter requests

Up
Down

Filter IP's in apache access logs based on use

The first sort is necessary for ips in a list to be actually unique.

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n

2013-09-02 13:04:47

User: basvdburg

Functions: cat cut sort uniq

Tags: IP apache2 filter requests

Up
Down

Filter IP's in apache access logs based on use

Show's per IP of how many requests they did to the Apache webserver

Comments (3) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

cut -d, -f1 /var/opt/example/dumpfile.130610_subscriber.csv | cut -c3-5 | sort | uniq -c | sed -e 's/^ *//;/^$/d' | awk -F" " '{print $2 "," $1}' > SubsxPrefix.csv

2013-07-17 07:58:56

User: neomefistox

Functions: awk cut sed sort uniq

Tags: Linux UNIX

Up
Down

Create a csv file with '5-digits prefix' phone numbers, as well as occurrences per prefix

dumpfile is a CSV file, which its 1st field is a phone number in format CC+10 digits

Empty lines are deleted, before the output in format "prefix,ocurrences"

sudo /usr/sbin/exim -bp | sed -n '/\*\*\* frozen \*\*\*/,+1!p' | awk '{print $1}' | tr -d [:blank:] | grep @ | sort | uniq -c | sort -n

2013-06-04 10:49:14

User: blueskin

Functions: awk grep sed sort sudo tr uniq

Tags: mail exim mailq

Up
Down

Show counts of messages in exim mail queue, grouped by recipient

Counts of messages by recipient, with frozen messages excluded.

more restart_weblogic.log | grep "LISTEN" | awk '{ print $7 }' | uniq | wc -l

2013-05-16 08:40:57

User: cuizhaohua

Functions: awk grep more uniq wc

Up
Down

how much time restart the wls service?

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

tail -F some.log | perl -ne 'print time(), "\n";' | uniq -c

2013-05-11 04:51:22

User: grana

Functions: perl tail uniq

Tags: tail wc

Up
Down

Lines per second in a log file

Comments (0) | Add to favourites | Report as malicious | Submit alternative | Report as a duplicate

< 1 2 3 4 > Last ›