If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

The pee command is in the moreutils package.

Uses GNU Parallel. Show Sample Output

Parallel does not suffer from the risk of mixing of output that xargs suffers from. -j+0 will run as many jobs in parallel as you have cores. With parallel you only need -0 (and -print0) if your filenames contain a '\n'. Parallel is from https://savannah.nongnu.org/projects/parallel/

xargs -P N spawns up to N worker processes. -n 40 means each grep command gets up to 40 file names each on the command line.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

Make sure to run this command in your git toplevel directory. Modify `-j4` as you like. You can also run any arbitrary command beside `git pull` in parallel on all of your git submodules. Show Sample Output

Mirror a remote directory using some tricks to maximize network speed. lftp:: coolest file transfer tool ever -u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa) -e: execute internal lftp commands set sftp:connect-program: use some specific command instead of plain ssh ssh:: -a -x -T: disable useless things -c arcfour: use the most efficient cipher specification -o Compression=no: disable compression to save CPU mirror: copy remote dir subtree to local dir -v: be verbose (cool progress bar and speed meter, one for each file in parallel) -c: continue interrupted file transfers if possible --loop: repeat mirror until no differences found --use-pget-n=3: transfer each file with 3 independent parallel TCP connections -P 2: transfer 2 files in parallel (totalling 6 TCP connections) sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate) You can play with values for --use-pget-n and/or -P to achieve maximum speed depending on the particular network. If the files are compressible removing "-o Compression=n" can be beneficial. Better create an alias for the command. Show Sample Output

Parallel is from https://savannah.nongnu.org/projects/parallel/ Other examples would be: (echo foss.org.my; echo www.debian.org; echo www.freenetproject.org) | parallel traceroute seq -f %04g 0 9999 | parallel -X rm pict{}.jpg

Do the same as pssh, just in shell syntax. Put your hosts in hostlist, one per line. Command outputs are gathered in output and error directories.

It takes over 5 seconds to scan a single port on a single host using nmap time (nmap -p 80 192.168.1.1 &> /dev/null) real 0m5.109s user 0m0.102s sys 0m0.004s It took netcat about 2.5 minutes to scan port 80 on the class C time (for NUM in {1..255} ; do nc -w 1 -z -v 192.168.1.${NUM} 80 ; done &> /dev/null) real 2m28.651s user 0m0.136s sys 0m0.341s Using parallel, I am able to scan port 80 on the entire class C in under 2 seconds time (seq 1 255 | parallel -j255 'nc -w 1 -z -v 192.168.1.{} 80' &> /dev/null) real 0m1.957s user 0m0.457s sys 0m0.994s

sort is way slow by default. This tells sort to use a buffer equal to half of the available free memory. It also will use multiple process for the sort equal to the number of cpus on your machine (if greater than 1). For me, it is magnitudes faster. If you put this in your bash_profile or startup file, it will be set correctly when bash is started. sort -S1 --parallel=2 <(echo) &>/dev/null && alias sortfast='sort -S$(($(sed '\''/MemF/!d;s/[^0-9]*//g'\'' /proc/meminfo)/2048)) $([ `nproc` -gt 1 ]&&echo -n --parallel=`nproc`)' Alternative echo|sort -S10M --parallel=2 &>/dev/null && alias sortfast="command sort -S$(($(sed '/MemT/!d;s/[^0-9]*//g' /proc/meminfo)/1024-200)) --parallel=$(($(command grep -c ^proc /proc/cpuinfo)*2))" Show Sample Output

Uses parallel processing Reiteration of my earlier command https://www.commandlinefu.com/commands/view/15246/convert-entire-music-library Usage lc Old_Directory New_DIrectory Old_Format New_Format lc ~/Music ~/Music_ogg mp3 ogg Show Sample Output

For instance: find . -type f -name '*.wav' -print0 |xargs -0 -P 3 -n 1 flac -V8 will encode all .wav files into FLAC in parallel. Explanation of xargs flags: -P [max-procs]: Max number of invocations to run at once. Set to 0 to run all at once [potentially dangerous re: excessive RAM usage]. -n [max-args]: Max number of arguments from the list to send to each invocation. -0: Stdin is a null-terminated list. I use xargs to build parallel-processing frameworks into my scripts like the one here: http://pastebin.com/1GvcifYa

Use this to find identify if dirs mostly contain large or small files. Show Sample Output

Ssh to host1, host2, and host3, executing on each host and saving the output in {host}.log. I don't have the 'parallel' command installed, otherwise it sounds interesting and less cryptic.

Backs up all databases, excluding test, mysql, performance_schema, information_schema. Requires parallel to work, install parallel on Ubuntu by running: sudo aptitude install parallel

Define a function that applies bc, the *nix calculator, with the specified expression to all rows of the input CSV. The first column is mapped to {1}, second one to {2}, and so forth. See sample output for an example. This function uses all available cores thanks to GNU Parallel. Requires GNU Parallel Show Sample Output

Shorter version using --tag

First the find command finds all files in your current directory (.). This is piped to xargs to be able to run the next shell pipeline in parallel. The xargs -P argument specifies how many processes you want to run in parallel, you can set this higher than your core count as the duration reading is mainly IO bound. The -print0 and -0 arguments of find and xargs respectively are used to easily handle files with spaces or other special characters. A subshell is executed by xargs to have a shell pipeline for each file that is found by find. This pipeline extracts the duration and converts it to a format easily parsed by awk. ffmpeg reads the file and prints a lot of information about it, grep extracts the duration line. cut and sed cut out the time information, and tr converts the last . to a : to make it easier to split by awk. awk is a specialized programming language for use in shell scripts. Here we use it to split the time elements in 4 variables and add them up. Show Sample Output