xargs -P 3 -n 1 <COMMAND> < <FILE_LIST>

Multi-thread any command

For instance: find . -type f -name '*.wav' -print0 |xargs -0 -P 3 -n 1 flac -V8 will encode all .wav files into FLAC in parallel. Explanation of xargs flags: -P [max-procs]: Max number of invocations to run at once. Set to 0 to run all at once [potentially dangerous re: excessive RAM usage]. -n [max-args]: Max number of arguments from the list to send to each invocation. -0: Stdin is a null-terminated list. I use xargs to build parallel-processing frameworks into my scripts like the one here: http://pastebin.com/1GvcifYa

2
By: h3xx
2011-07-25 22:53:32

These Might Interest You

  • This command will list all threads started by a particular pid along with the start time of each thread. This is very valuable when diagnosing thread leaks.


    1
    ps -o pid,lwp,lstart --pid 797 -L
    jkharness87 · 2010-11-24 17:04:08 0
  • First (and only) argument should be a 4chan thread URL.


    1
    function 4chandl () { wget -e robots=off -nvcdp -t 0 -Hkrl 0 -I \*/src/ -P . "$1" }
    89r · 2013-07-28 11:29:53 0
  • The example is a little bit bogus, but applies to any command that takes a while interactively, or might be a bit of a drag on system resources. Once the command's output is saved to a variable, you can then echo "$OUTPUT" to see it in multi-line glory after that. The use of double-quotes around the backticks and during the variable expansion disables any IFS conversion during those two operations. Very useful for reporting that might pull different lines out, like from dmidecode, inq or any other disk detail command. The only caveat is that storing too much in a variable might make your shell process grow.


    0
    OUTPUT="`find / -type f`" ; echo "$OUTPUT" | grep sysrq ; echo "$OUTPUT" | grep sysctl ; echo "$OUTPUT" | less
    DoNotRememberMe · 2010-03-25 05:02:10 0
  • This does almost the same thing as the original, but it runs the full backtrace for _all_ the threads, which is pretty important when reporting a crash for a multithreaded software, since more often than not, the signal handler is executed in a different thread than the crash happened.


    5
    gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" ${exe} ${corefile}
    Flameeyes · 2010-07-06 14:49:03 1
  • This takes a picture (with the web cam) every 5 minutes, and send the picture to your e-mail. Some systems support mail -a "References: " so that all video surveillance emails are grouped in a single email thread. To keep your inbox clean, it is still possible to filter and move to trash video surveillance emails (and restore these emails only if you really get robbed!) For instance with Gmail, emails sent to me+trash@gmail.com can be filtered with "Matches: DeliveredTo:me+trash@gmail.com" Show Sample Output


    2
    while true ; do fswebcam -d /dev/video0 -r 1280x1024 -F 15 - | uuencode $(date +\%Y\%m\%d\%H\%M).jpeg | mail -s "Video surveillance" me@gmail.com ; sleep 300 ; done
    pascalv · 2016-08-09 14:22:45 0
  • This is a multi-sourced hostname tab completion command you can put in your .bashrc (or your .bash_profile on a MAC). This will not only your .ssh/config, it will parse ALL of the following files for hostnames: .ssh/config .ssh/known_hosts .bash_history (I realize this could be prettied up by defining and calling the function, instead of implicitly referencing it, but this was more breif) Show Sample Output


    0
    complete -W "$( { awk '/^Host / { print $2 }' ~/.ssh/config | egrep -v '\*|,' echo $( grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //' ) while IFS=' ,' read host t; do echo $host; done < ~/.ssh/known_hosts ;} )" ssh
    thelinuxgirl · 2012-02-28 18:08:53 0

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: