Commands using uniq

Useful for C projects where header file names must be unique (e.g. when using autoconf/automake), or when diagnosing if the wrong header file is being used (due to dupe file names)

Improvement of the command "Find Duplicate Files (based on size first, then MD5 hash)" when searching for duplicate files in a directory containing a subversion working copy. This way the (multiple dupicates) in the meta-information directories are ignored.

Can easily be adopted for other VCS as well. For CVS i.e. change ".svn" into ".csv":

A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster.

Parallel is from https://savannah.nongnu.org/projects/parallel/

i wanted to delete all duplicate lines from .bash_history and keep the order of the other lines.

the command cat's the file and adds line numbers, then sorts by the second column. afterwards uniq omits repeated lines, but skips the first field (the line number). then it sorts by the line numbers and at the end cuts the numbers off.

Adapted using your usefull comments !

This command will shorten any URL the user inputs. What makes this command different is that it utilizes 5 different services and gives you 5 different outputs: is.gd, bit.ly, u.nu, geekology.co.za, and tinyurl.

curl -s http://tinyurl.com/create.php?url=$1 \ | sed -n 's/.*\(http:\/\/tinyurl.com\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://bit.ly/?url=$1 \ | sed -n 's/.*\(shortened-url" value="http:\/\/bit.ly\/[a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/p' \ | sed -n 's/.*\(http:\/\/bit.ly\/[a-zA-Z0-9][a-zA-Z0-9]*\).*/\1/p' \ | uniq ; curl -s http://geekology.co.za/shortii/create.php?u=$1 \ | sed -n 's/.*\(http:\/\/geekology.co.za\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://u.nu/unu-api-simple?url=$1 \ | sed -n 's/.*\(http:\/\/u.nu\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq ; curl -s http://is.gd/api.php?longurl=$1 \ | sed -n 's/.*\(http:\/\/is.gd\/[a-z0-9][a-z0-9]*\).*/\1/p' \ | uniq echo ""

Searches all log files (including archived bzip2 files) for invalid user and PAM authentication errors, both of which are indicative of brute force attempts at logging into computer. A list of all unique IP addresses and domain names is appended to hosts.deny. The command (and grep error messages) will work on Mac OS X 10.6, small adjustments may be needed for other OSs.

Uses the dumb terminal option in gnuplot to plot a graph of frequencies. In this case, we are looking at a frequency analysis of words in all of the .c files.

You'll run into trouble if you have files w/ missing newlines at the end. I tried to use

and even

to force a newline at the end, but as soon as the output is redirected, git seems to ignore the pager.

Figures out total line contribution per author for an entire GIT repo. Includes binary files, which kind of mess up the true count.

If crashes or takes too long, mess with the ls-file option at the start:

git ls-files -x "*pdf" -x "*psd" -x "*tif" to remove really random binary files

git ls-files "*.py" "*.html" "*.css" to only include specific file types

Based off my original SVN version: http://www.commandlinefu.com/commands/view/2787/prints-total-line-count-contribution-per-user-for-an-svn-repository

counts the total (recursive) number of files in the immediate (depth 1) subdirectories as well as the current one and displays them sorted.

Fixed, as per ashawley's comment

This dup finder saves time by comparing size first, then md5sum, it doesn't delete anything, just lists them.

show only the name of the apps that are using internet

Can be used to discover what programms create internet traffic. Skip the part after awk to get more details.

Has anyone an idea why the uniq doesn't work propperly here (see sample output)?

Calculates md5 sum of files. sort (required for uniq to work). uniq based on only the hash. use cut ro remove the hash from the result.

The command could show you all conecctions if you skip "grep ESTABLISHED"

I'm working in a group project currently and annoyed at the lack of output by my teammates. Wanting hard metrics of how awesome I am and how awesome they aren't, I wrote this command up.

It will print a full repository listing of all files, remove the directories which confuse blame, run svn blame on each individual file, and tally the resulting line counts. It seems quite slow, depending on your repository location, because blame must hit the server for each individual file. You can remove the -R on the first part to print out the tallies for just the current directory.

This command will generate "CHECK TABLE `db_name.table_name` ;" statements for all tables present in databases on a MySQL server, which can be piped into the mysql command. (Can also be altered to perform OPTIMIZE and REPAIR functions.)

Tested on MySQL 4.x and 5.x systems in a Linux environment under bash.