Commands using uniq

Finds the top ten pages returning an http response code of 404 in an apache log.

Most of the "most used commands" approaches does not consider pipes and other complexities.

This approach considers pipes, process substitution by backticks or $() and multiple commands separated by ;

Perl regular expression breaks up each line using | or < ( or ; or ` or $( and picks the first word (excluding "do" in case of for loops)

note: if you are using lots of perl one-liners, the perl commands will be counted as well in this approach, since semicolon is used as a separator

Get a list of all the unique hostnames from the apache configuration files. Handy to see what sites are running on a server.

Finds all (not just adjacent) repeated lines in a file.

Useful for C projects where header file names must be unique (e.g. when using autoconf/automake), or when diagnosing if the wrong header file is being used (due to dupe file names)

Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv":

A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster.

Parallel is from https://savannah.nongnu.org/projects/parallel/

i wanted to delete all duplicate lines from .bash_history and keep the order of the other lines.

the command cat's the file and adds line numbers, then sorts by the second column. afterwards uniq omits repeated lines, but skips the first field (the line number). then it sorts by the line numbers and at the end cuts the numbers off.

Adapted using your usefull comments !

This command will shorten any URL the user inputs. What makes this command different is that it utilizes 5 different services and gives you 5 different outputs: is.gd, bit.ly, u.nu, geekology.co.za, and tinyurl.

curl -s http://tinyurl.com/create.php?url=$1 \ | sed -n 's/.*\(http:\/\/tinyurl.com\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://bit.ly/?url=$1 \ | sed -n 's/.*\(shortened-url" value="http:\/\/bit.ly\/[a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/p' \ | sed -n 's/.*\(http:\/\/bit.ly\/[a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/p' \ | uniq ; curl -s http://geekology.co.za/shortii/create.php?u=$1 \ | sed -n 's/.*\(http:\/\/geekology.co.za\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://u.nu/unu-api-simple?url=$1 \ | sed -n 's/.*\(http:\/\/u.nu\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://is.gd/api.php?longurl=$1 \ | sed -n 's/.*\(http:\/\/is.gd\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq echo ""

Searches all log files (including archived bzip2 files) for invalid user and PAM authentication errors, both of which are indicative of brute force attempts at logging into computer. A list of all unique IP addresses and domain names is appended to hosts.deny. The command (and grep error messages) will work on Mac OS X 10.6, small adjustments may be needed for other OSs.

Uses the dumb terminal option in gnuplot to plot a graph of frequencies. In this case, we are looking at a frequency analysis of words in all of the .c files.

You'll run into trouble if you have files w/ missing newlines at the end. I tried to use

and even

to force a newline at the end, but as soon as the output is redirected, git seems to ignore the pager.

Figures out total line contribution per author for an entire GIT repo. Includes binary files, which kind of mess up the true count.

If crashes or takes too long, mess with the ls-file option at the start:

git ls-files -x "*pdf" -x "*psd" -x "*tif" to remove really random binary files

git ls-files "*.py" "*.html" "*.css" to only include specific file types

Based off my original SVN version: http://www.commandlinefu.com/commands/view/2787/prints-total-line-count-contribution-per-user-for-an-svn-repository

counts the total (recursive) number of files in the immediate (depth 1) subdirectories as well as the current one and displays them sorted.

Fixed, as per ashawley's comment

This dup finder saves time by comparing size first, then md5sum, it doesn't delete anything, just lists them.

show only the name of the apps that are using internet