Commands tagged unique (4)

  • Reads stdin, and outputs each line only once - without sorting ahead of time. This does use more memory than your system's sort utility.


    4
    perl -ne 'print if !$a{$_}++'
    doherty · 2011-02-17 02:18:44 2
  • Once you get into advanced/optimized scripts, functions, or cli usage, you will use the sort command alot. The options are difficult to master/memorize however, and when you use sort commands as much as I do (some examples below), it's useful to have the help available with a simple alias. I love this alias as I never seem to remember all the options for sort, and I use sort like crazy (much better than uniq for example). # Sorts by file permissions find . -maxdepth 1 -printf '%.5m %10M %p\n' | sort -k1 -r -g -bS 20% 00761 drwxrw---x ./tmp 00755 drwxr-xr-x . 00701 drwx-----x ./askapache-m 00644 -rw-r--r-- ./.htaccess # Shows uniq history fast history 1000 | sed 's/^[0-9 ]*//' | sort -fubdS 50% exec bash -lxv export TERM=putty-256color Taken from my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    3
    alias sorth='sort --help|sed -n "/^ *-[^-]/s/^ *\(-[^ ]* -[^ ]*\) *\(.*\)/\1:\2/p"|column -ts":"'
    AskApache · 2010-06-10 21:30:31 0
  • Uniq command is mostly used in combination with sort command, as uniq removes duplicates only from a sorted file. i.e In order for uniq to work, all the duplicate entries should be in the adjacent lines. Show Sample Output


    0
    sort namesd.txt | uniq
    ankush108 · 2012-06-26 19:22:34 0
  • The following displays only the entries that are duplicates. Show Sample Output


    0
    sort namesd.txt | uniq ?cd
    ankush108 · 2012-06-26 19:23:58 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Decrypt passwords from Google Chrome and Chromium.
Read this before you down voting and comment that it is not working -> Wont work on latest versions ~75> since database file is locked and has to be decrypted. This is useful if you have an old hdd with a chrome installation and want to decrypt your old passwords fast.

list files recursively by size

generate random password
-B flag = don't include characters that can be confused for other characters (this helps when you give someone their password for the first time so they don't cause a lockout with, for example, denyhosts or fail2ban) -s flag = make a "secure", or hard-to-crack password -y flag = include special characters (not used in the example because so many people hate it -- however I recommend it) "1 10" = output 1 password, make it 10 characters in length For even more secure passwords please use the -y flag to include special characters like so: $ pwgen -Bsy 10 1 output>> }&^Y?.>7Wu

Compare two CSV files, discarding any repeated lines
The value for the sort command's -k argument is the column in the CSV file to sort on. In this example, it sorts on the second column. You must use some form of the sort command in order for uniq to work properly.

send echo to socket network
this command will send a message to the socket 25 on host 192.168.1.2 in tcp. works on udp and icmp understand only IP address, not hostname. on the other side (192.168.1.2), you can listen to this socket and test if you receive the message. easy to diagnose a firewall problem or not.

remove password from openssl key file
Strip a password from a openssl key to use with apache httpd server

Monitor cpu in realtime.

Split a file one piece at a time, when using the split command isn't an option (not enough disk space)
bs = buffer size (basically defined the size of a "unit" used by count and skip) count = the number of buffers to copy (16m * 32 = 1/2 gig) skip = (32 * 2) we are grabbing piece 3...which means 2 have already been written so skip (2 * count) i will edit this later if i can to make this all more understandable

pop-up messages on a remote computer
Run this command when you are physically at the computer you wish to send pop-up messages to. Then when you ssh in to it, you can do this: echo "guess who?" > commander guess who? will then pop up on the screen for a few moments, then disappear. You will need to create the commander file first. I mess with my wife all the time with this. i.e. echo "You have given the computer a virus. Computer will be rendered useless in 10 seconds." > commander lol

Rename files in batch


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: