Commands by alafrosty (1)

  • * Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.


    0
    find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"
    alafrosty · 2013-10-22 13:34:19 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Down for everyone or just me?
Check if a site is down with downforeveryoneorjustme.com

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

fdiff is a 'filtered diff'. Given a text filter and two inputs, will run the filter across the input files and diff the output.
Fdiff will run the command given by the first argument against the input files given as the second and third arguments, and diff the results. It will use 'diff' as the default diff program, but this can be changed by setting $DIFFCMD, e.g. $ export DIFFCMD=vimdiff; $ fdiff zcat 0716_0020005.raw.gz 0716_0030005.raw.gz ... This function will work under bash, but requires the use of command substitution, which is not available under a strict ANSI shell.

a function to find the fastest DNS server
http://public-dns.info gives a list of online dns servers. you need to change the country in url (br in this url) with your country code. this command need some time to ping all IP in list.

copy partition table from /dev/sda to /dev/sdb

Verify/edit bash history command before executing it
Bash history commands are those that begin with the character ! (eg. the most popular 'sudo !!' Explained here => http://www.commandlinefu.com/commands/view/13). By default bash immediately executes the history command. Setting this shell option will make bash first allow you to verify/edit an history command before executing it. To set this option permanently, put this command in ~/.profile or ~/.bashrc file. To unset this option issue following command. $shopt -u histverify

File rotation without rename command
Rotates log files with "gz"-extension in a directory for 7 days and enumerates the number in file name. i.e.: logfile.1.gz > logfile.2.gz I needed this line due to the limitations on AIX Unix systems which do not ship with the rename command.

Rename files in batch

Get ElasticSearch configuration and version details
Replace localhost:9200 with your server location and port. This is the ElasticSearch's default setup for local instances.

Edit a file inside a compressed archive without extracting it
If you vim a compressed file it will list all archive content, then you can pickup any of them for editing and saving. There you have the modified archive without any extra step. It supports many file types such as tar.gz, tgz, zip, etc.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: