Commands tagged unique (4)

  • Reads stdin, and outputs each line only once - without sorting ahead of time. This does use more memory than your system's sort utility.


    4
    perl -ne 'print if !$a{$_}++'
    doherty · 2011-02-17 02:18:44 2
  • Once you get into advanced/optimized scripts, functions, or cli usage, you will use the sort command alot. The options are difficult to master/memorize however, and when you use sort commands as much as I do (some examples below), it's useful to have the help available with a simple alias. I love this alias as I never seem to remember all the options for sort, and I use sort like crazy (much better than uniq for example). # Sorts by file permissions find . -maxdepth 1 -printf '%.5m %10M %p\n' | sort -k1 -r -g -bS 20% 00761 drwxrw---x ./tmp 00755 drwxr-xr-x . 00701 drwx-----x ./askapache-m 00644 -rw-r--r-- ./.htaccess # Shows uniq history fast history 1000 | sed 's/^[0-9 ]*//' | sort -fubdS 50% exec bash -lxv export TERM=putty-256color Taken from my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    3
    alias sorth='sort --help|sed -n "/^ *-[^-]/s/^ *\(-[^ ]* -[^ ]*\) *\(.*\)/\1:\2/p"|column -ts":"'
    AskApache · 2010-06-10 21:30:31 0
  • Uniq command is mostly used in combination with sort command, as uniq removes duplicates only from a sorted file. i.e In order for uniq to work, all the duplicate entries should be in the adjacent lines. Show Sample Output


    0
    sort namesd.txt | uniq
    ankush108 · 2012-06-26 19:22:34 0
  • The following displays only the entries that are duplicates. Show Sample Output


    0
    sort namesd.txt | uniq ?cd
    ankush108 · 2012-06-26 19:23:58 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Optimal way of deleting huge numbers of files
Optimal way of deleting huge numbers of files Using -delete is faster than: $ find /path/to/dir -type f -print0 | xargs -0 rm $ find /path/to/dir -type f -exec rm {} + $ find /path/to/dir -type f -exec rm \-f {} \;

Find the package that installed a command

Rename duplicates from MusicBrainz Picard
Renames duplicates from MusicBrainz Picard, so you get the latest copy and not a bunch of duplicates.

Install pip with Proxy
Installs pip packages defining a proxy

List the Sizes of Folders and Directories
I wanted an easy way to list out the sizes of directories and all of the contents of those directories recursively.

Write comments to your history.
A null operation with the name 'comment', allowing comments to be written to HISTFILE. Prepending '#' to a command will *not* write the command to the history file, although it will be available for the current session, thus '#' is not useful for keeping track of comments past the current session.

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Truncate 0.3 sec from an audio file using sox
Using this command one can cut a piece from the end of an audio file.

List the biggest accessible files/dirs in current directory, sorted

Create the signature base string required for a Twitter stream feed
This is the SECOND command in a set for five that are needed for a Twitter stream feed. This command creates variable "b", the so-called "base string" required for oauth in Twitter stream feed requests. (The 256 char limit prevents giving it a better name) We use five environment variables created by a previous step: id, k1, once, ts and k3. The five environment variables are created in a separate command, please see my other commands. For more information on the signature base string, see dev.twitter.com/apps, click on any app (or create a new one) and then go to the "OAuth Tool" tab.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: