Commands tagged fdupes (1)

  • * Find all file sizes and file names from the current directory down (replace "." with a target directory as needed). * sort the file sizes in numeric order * List only the duplicated file sizes * drop the file sizes so there are simply a list of files (retain order) * calculate md5sums on all of the files * replace the first instance of two spaces (md5sum output) with a \0 * drop the unique md5sums so only duplicate files remain listed * Use AWK to aggregate identical files on one line. * Remove the blank line from the beginning (This was done more efficiently by putting another "IF" into the AWK command, but then the whole line exceeded the 255 char limit). >>>> Each output line contains the md5sum and then all of the files that have that identical md5sum. All fields are \0 delimited. All records are \n delimited.


    0
    find . -type f -not -empty -printf "%-25s%p\n"|sort -n|uniq -D -w25|cut -b26-|xargs -d"\n" -n1 md5sum|sed "s/ /\x0/"|uniq -D -w32|awk -F"\0" 'BEGIN{l="";}{if(l!=$1||l==""){printf "\n%s\0",$1}printf "\0%s",$2;l=$1}END{printf "\n"}'|sed "/^$/d"
    alafrosty · 2013-10-22 13:34:19 7

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Backup all MySQL Databases to individual files
No need to loop when we have `xargs`. The sed command filters out the first line of `show databases` output, which is always "Database".

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

create shortcut keys in bash

Recursively remove .svn directories

Periodically run a command without hangups, and send the output to your e-mail
Run "ps -x" (process status) in the background every hour (in this example). The outputs of both "nohup" and "ps -x" are sent to the e-mail (instead of nohup.out and stdout and stderr). If you like it, replace "ps -x" by the command of your choice, replace 3600 (1 hour) by the period of your choice. You can run the command in the loop any time by killing the sleep process. For example $ ps -x 2925 ? S 0:00.00 sh -c unzip E.zip >/dev/null 2>&1 11288 ? O 0:00.00 unzip E.zip 25428 ? I 0:00.00 sleep 3600 14346 pts/42- I 0:00.01 bash -c while true; do ps -x | mail (...); sleep 3600; done 643 pts/66 Ss 0:00.03 -bash 14124 pts/66 O+ 0:00.00 ps -x $ kill 25428 You have mail in /mail/(...)

Download latest NVIDIA Geforce x64 Windows driver
Download latest NVIDIA Geforce x64 Windows7-8 driver from Nvidia's website. Pulls the latest download version (which includes beta). This is the "English" version. The following command includes a 'sed' line to replace "english" with "international" if needed. You can also replace the starting subdomain with "eu." "uk." and others. Enjoy this one liner! 1 character under the max :) $wget "us.download.nvidia.com$(wget -qO- "$(wget -qO- "nvidia.com/Download/processFind.aspx?psid=95&pfid=695&osid=19&lid=1&lang=en-us" | awk '/driverResults.aspx/ {print $4}' | cut -d "'" -f2 | head -n 1)" | awk '/url=/ {print $2}' | sed -e "s/english/international/" | cut -d '=' -f3 | cut -d '&' -f1)"

Get a shell with a not available account

use screen as a terminal emulator to connect to serial consoles
Use GNU/screen as a terminal emulator for anything serial console related. screen /dev/tty eg. screen /dev/ttyS0 9600 MacOSX: http://www.macosxhints.com/article.php?story=20061109133825654 Cheat Sheet: http://www.catonmat.net/blog/screen-terminal-emulator-cheat-sheet/

calculate md5 sums for every file in a directory tree
an alternative

Find last reboot time
Specific to OSX.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: