Commands tagged unique (4)

  • uniq without pre-sorting

    Reads stdin, and outputs each line only once - without sorting ahead of time. This does use more memory than your system's sort utility.


    4
    perl -ne 'print if !$a{$_}++'
    doherty · 2011-02-17 02:18:44 2
  • View advanced Sort options, Quick Reference Help Alias

    Once you get into advanced/optimized scripts, functions, or cli usage, you will use the sort command alot. The options are difficult to master/memorize however, and when you use sort commands as much as I do (some examples below), it's useful to have the help available with a simple alias. I love this alias as I never seem to remember all the options for sort, and I use sort like crazy (much better than uniq for example). # Sorts by file permissions find . -maxdepth 1 -printf '%.5m %10M %p\n' | sort -k1 -r -g -bS 20% 00761 drwxrw---x ./tmp 00755 drwxr-xr-x . 00701 drwx-----x ./askapache-m 00644 -rw-r--r-- ./.htaccess # Shows uniq history fast history 1000 | sed 's/^[0-9 ]*//' | sort -fubdS 50% exec bash -lxv export TERM=putty-256color Taken from my http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html Show Sample Output


    3
    alias sorth='sort --help|sed -n "/^ *-[^-]/s/^ *\(-[^ ]* -[^ ]*\) *\(.*\)/\1:\2/p"|column -ts":"'
    AskApache · 2010-06-10 21:30:31 0
  • Unique sort

    Uniq command is mostly used in combination with sort command, as uniq removes duplicates only from a sorted file. i.e In order for uniq to work, all the duplicate entries should be in the adjacent lines. Show Sample Output


    0
    sort namesd.txt | uniq
    ankush108 · 2012-06-26 19:22:34 0
  • Show duplicate lines in a file

    The following displays only the entries that are duplicates. Show Sample Output


    0
    sort namesd.txt | uniq ?cd
    ankush108 · 2012-06-26 19:23:58 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Install pip with Proxy
Installs pip packages defining a proxy

Remove annoying files from recently extracted zip archive
Inspired by http://www.commandlinefu.com/commands/view/2573/remove-all-files-previously-extracted-from-a-tar.gz-file. .... yet for zip files

shell function to underline a given string.
underline() will print $1, followed by a series of '=' characters the width of $1. An optional second argument can be used to replace '=' with a given character. This function is useful for breaking lots of data emitted in a for loop into sections which are easier to parse visually. Let's say that 'xxxx' is a very common pattern occurring in a group of CSV files. You could run $ grep xxxx *.csv This would print the name of each csv file before each matching line, but the output would be hard to parse visually. $ for i in *.csv; do printf "\n"; underline $i; grep "xxxx" $i; done Will break the output into sections separated by the name of the file, underlined.

Update zone file Serial numbers
Will edit *.db files in the same directory with todays date. Useful for doing a mass update to domains on a nameserver, adding spf records, etc. Looks for a string starting with 200 or 201 followed by 7 numbers, and replaces with todays date. This won't overwrite Ip's but i would still do some double checking after running this. Make sure your server's date is correct, otherwise insert your own serial number. $rndc reload should usually follow this command.

Run a long job and notify me when it's finished
You will need libnotify-bin for this to work: $ sudo aptitude install libnotify-bin

Convert CSV to JSON
Replace 'csv_file.csv' with your filename.

Open Remote Desktop (RDP) from command line having a custom screen size
This example uses xfreerdp, which builds upon the development of rdesktop. This example usage will also send you the remote machine's sound.

Merge video files together using mencoder (part of mplayer)
Using mplayer's mencoder, you can merge video files together. '-oac' specifies the audio encoding (here copy, to just copy and not compress) '-ovc' specifies the video encoding (same thing).

List upgrade-able packages on Ubuntu
Taken from apticron and modified.

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: