Commands tagged parallel sorted by votes

Commands tagged parallel (39)

sorted by

Instant threadpool
This is sample output - yours may be different.
0

cat item_list | xargs -n1 -P<n> process_item

sanchan89 · 2013-03-06 20:25:17 0
Calculate different hash sums of one file at the same time

This is to overcome the issue of slow I/O by reading once and forwarding the output to several processes (e. g. 3 in the given command). One could also invoke grep or other programs to work on read data. Show Sample Output
This is sample output - yours may be different.
```
ce67efc071187fe6ee2bd7486cf1307e  -
b16de8c6465bd973188d62e5772f21762d624f0f  -
6c801e3a40ab3e5192420d47f676e7291d8572c4144687e90d59335a47c750ee  -
```
0

dd if=file | tee >(sha1sum) >(md5sum) >(sha256sum) >/dev/null

dubbaluga · 2013-11-07 17:43:54 0
Multi-thread any command
This is sample output - yours may be different.
0

find . -mtime +1000 -type f

cuevas500 · 2015-12-15 06:43:14 0
bulk rename files with GNU Parallel

Works with any file name: space, ', " and even \n is OK. The code in {= =} is a perl expression.
This is sample output - yours may be different.
0

parallel -0 mv {} {=s/foo/bar/g=} ::: *

unixmonkey91589 · 2015-12-25 17:22:43 0
Recompress all files in current directory from gzip to bzip2

Works even if file name contains \n. Spawns one job per core.
This is sample output - yours may be different.
0

parallel -0 'gzip -dc {} | bzip2 > {.}.bz2' ::: *.gz

unixmonkey91589 · 2015-12-25 17:28:03 0
Get the total length of all video / audio in the current dir (and below) in H:m:s

All the other example fail when running in a folder containing too many files due to * being saturated. This command does not use *, allowing me to run it in one folder containing over 300000 audio files. As to running on so many files, I used GNU parallel in order to spawn as many processes as cores, tremendously fasting up the process. Show Sample Output
This is sample output - yours may be different.
```
0:11:3
```
0

ls|grep ".wav"|parallel -j$(nproc) soxi -D {}|awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'

jupiter126 · 2017-05-02 21:37:24 0
Parallel Apache Benchmark

(echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/") | parallel -k 'ab -n 10000 -c 15 {}'
This is sample output - yours may be different.
0

cat url_list.txt | parallel -k 'ab -n 10000 -c 15 {}'

emphazer · 2018-05-17 11:23:28 0
Delete all but latest file in a directory

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-1

ls -t1 | sed 1d | parallel -X rm

unixmonkey8046 · 2010-01-28 12:28:18 0
quick integer CPU benchmark

You could have that little benchmark run on all cores in parallel, as a multi-core benchmark or stress test First find the number of cores, then have parallel iterate over that in, well, parallel Show Sample Output
This is sample output - yours may be different.
```
...
41454625947560171969501001538543925124881846970621917597708878154881\
18278788738248093374936943243527734573872009289119068940335579136

real	0m12,698s
user	0m14,461s
sys	0m0,382s
```
-1

time cat /proc/cpuinfo |grep proc|wc -l|xargs seq|parallel -N 0 echo "2^2^20" '|' bc

kostis · 2018-12-06 05:36:55 0
Remove all files previously extracted from a tar(.gz) file.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

tar -tf <file.tar.gz> | parallel rm

unixmonkey8046 · 2010-01-28 08:28:16 0
add all files not under version control to repository

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

svn status |grep '\?' |awk '{print $2}'| parallel -Xj1 svn add

unixmonkey8046 · 2010-01-28 08:47:54 0
archive all files containing local changes (svn)

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
This is sample output - yours may be different.
-2

svn st | cut -c 9- | parallel -X tar -czvf ../backup.tgz

unixmonkey8046 · 2010-01-28 11:43:16 0
List top ten files/directories sorted by size

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This is sample output - yours may be different.
```
33M     a b
17M     libxnee
3.6M    doc
3.0M    gnee
```
-2

du -s * | sort -nr | head | cut -f2 | parallel -k du -sh

unixmonkey8046 · 2010-01-28 12:59:14 3
backup and remove files with access time older than 5 days.

This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
-3

tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5 -type f` 2> /dev/null | parallel -X rm -f

unixmonkey8046 · 2010-01-28 12:41:41 0
< 1 2

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

colorize comm output

It just colorizes the line based on if it has 0, 1 or 2 tabs at the beginning of the line. Won't work so well if lines already begin with tabs (too bad comm doesn't have an option to substitute \t for something else). Don't forget comm needs input files to be sorted. You can use a shortcut like this with bash: comm

Clean up poorly named TV shows.

Replace 'SHOWNAME' with the name of the TV show. Add -n to test the command without renaming files. Check the 'sample output'.

Single use vnc-over-ssh connection

This command 1. SSH into a machine 2. Tunnels VNC port to your local computer ("-L 5900:localhost:5900") 3. Runs a single use vnc server ("x11vnc -safer -localhost -nopw -once -display :0") 4. Goes into the background ("-f") 5. Runs VNC viewer on the local computer connecting to the remote machine via the newly created SSH tunnel ("vinagre localhost:5900")

Edit video by cutting the part you like without transcoding.

Examples: The following will take frames starting at 15.2 seconds for a total of 45.9 seconds: $ mencoder -ss 15.2 -endpos 30.7 -oac copy -ovc copy mymovie.avi -o myeditedmovie.avi Keep in mind -endpos is the total time, i.e. the output video in this is 3 minutes 3 seconds in length: $ mencoder -ss 1 minute -endpos 2 minutes 3 seconds -oac copy -ovc copy mymovie.avi -o myeditedmovie.avi

Randomize lines in a file

This appends a random number as a first filed of all lines in SOMEFILE then sorts by the first column and finally cuts of the random numbers.

Generate random valid mac addresses

Ruby version. Also, a perl version: $perl -e 'printf("%.2x.",rand(255))for(1..5);printf("%.2x\n",rand(255))'

Display unique values of a column

the column number is '6'

Limit bandwidth usage by apt-get

Limits the usage of bandwidth by apt-get, in the example the command will use 30Kb/s ;) It should work for most apt-get actions (install, update, upgrade, dist-upgrade, etc.)

Mplayer save stream to file

To save any streaming url content to a file.

Blue Matrix

Same as original, but works in bash

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged parallel

Commands tagged parallel (39) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands tagged parallel (39)

sorted by