commandlinefu.com is the place to record those command-line gems that you return to again and again.
You can sign-in using OpenID credentials, or register a traditional username and password.
Subscribe to the feed for:
parallel can be installed on your central node and can be used to run a command multiple times.
In this example, multiple ssh connections are used to run commands. (-j is the number of jobs to run at the same time). The result can then be piped to commands to perform the "reduce" stage. (sort then uniq in this example).
This example assumes "keyless ssh login" has been set up between the central node and all machines in the cluster.
bashreduce may also do what you want.
There are 4 alternatives - vote for the best!
If you can do better, submit your command here.
You must be signed in to comment.