Commands tagged parallel

sort is way slow by default. This tells sort to use a buffer equal to half of the available free memory. It also will use multiple process for the sort equal to the number of cpus on your machine (if greater than 1). For me, it is magnitudes faster.

If you put this in your bash_profile or startup file, it will be set correctly when bash is started.

Alternative

You need: pxz for the actual work (http://jnovy.fedorapeople.org/pxz/). The function could be better with better multifile and stdin/out support.

Use this to find identify if dirs mostly contain large or small files.

For instance:

will encode all .wav files into FLAC in parallel.

Explanation of xargs flags:

-P [max-procs]: Max number of invocations to run at once. Set to 0 to run all at once [potentially dangerous re: excessive RAM usage].

-n [max-args]: Max number of arguments from the list to send to each invocation.

-0: Stdin is a null-terminated list.

I use xargs to build parallel-processing frameworks into my scripts like the one here: http://pastebin.com/1GvcifYa

Make sure to run this command in your git toplevel directory. Modify `-j4` as you like. You can also run any arbitrary command beside `git pull` in parallel on all of your git submodules.

It takes over 5 seconds to scan a single port on a single host using nmap

real 0m5.109s

user 0m0.102s

sys 0m0.004s

It took netcat about 2.5 minutes to scan port 80 on the class C

real 2m28.651s

user 0m0.136s

sys 0m0.341s

Using parallel, I am able to scan port 80 on the entire class C in under 2 seconds

real 0m1.957s

user 0m0.457s

sys 0m0.994s

If you have the fdupes command, you'll save a lot of typing. It can do recursive searches (-r,-R) and it allows you to interactively select which of the duplicate files found you wish to keep or delete.

The pee command is in the moreutils package.

Do the same as pssh, just in shell syntax.

Put your hosts in hostlist, one per line.

Command outputs are gathered in output and error directories.

Ssh to host1, host2, and host3, executing on each host and saving the output in {host}.log.

I don't have the 'parallel' command installed, otherwise it sounds interesting and less cryptic.

Parallel does not suffer from the risk of mixing of output that xargs suffers from. -j+0 will run as many jobs in parallel as you have cores.

With parallel you only need -0 (and -print0) if your filenames contain a '\n'.

Parallel is from https://savannah.nongnu.org/projects/parallel/

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that.

This deals nicely with files having special characters in the file name (space ' or ").

Parallel is from https://savannah.nongnu.org/projects/parallel/

xargs deals badly with special characters (such as space, ' and "). To see the problem try this:

touch important_file

touch 'not important_file'

ls not* | xargs rm

Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'.

Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

Both solutions work bad if the number of files is more than the allowed line length of the shell.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this:

touch important_file

touch 'not important_file'

ls not* | xargs rm

Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this:

touch important_file

touch 'not important_file'

ls not* | xargs rm

Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster.

Parallel is from https://savannah.nongnu.org/projects/parallel/

xargs deals badly with special characters (such as space, ' and "). To see the problem try this:

touch important_file

touch 'not important_file'

ls not* | xargs rm

Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.

Parallel is from https://savannah.nongnu.org/projects/parallel/

Other examples would be:

(echo foss.org.my; echo www.debian.org; echo www.freenetproject.org) | parallel traceroute

seq -f %04g 0 9999 | parallel -X rm pict{}.jpg

xargs -P N spawns up to N worker processes. -n 40 means each grep command gets up to 40 file names each on the command line.