Commands tagged parallel sorted by votes

Commands tagged parallel (39)

sorted by

Instant threadpool
This is sample output - yours may be different.
0

cat item_list | xargs -n1 -P<n> process_item

sanchan89 · 2013-03-06 20:25:17 0
Calculate different hash sums of one file at the same time

This is to overcome the issue of slow I/O by reading once and forwarding the output to several processes (e. g. 3 in the given command). One could also invoke grep or other programs to work on read data. Show Sample Output
This is sample output - yours may be different.
```
ce67efc071187fe6ee2bd7486cf1307e  -
b16de8c6465bd973188d62e5772f21762d624f0f  -
6c801e3a40ab3e5192420d47f676e7291d8572c4144687e90d59335a47c750ee  -
```
0

dd if=file | tee >(sha1sum) >(md5sum) >(sha256sum) >/dev/null

dubbaluga · 2013-11-07 17:43:54 0
Multi-thread any command
This is sample output - yours may be different.
0

find . -mtime +1000 -type f

cuevas500 · 2015-12-15 06:43:14 0
bulk rename files with GNU Parallel

Works with any file name: space, ', " and even \n is OK. The code in {= =} is a perl expression.
This is sample output - yours may be different.
0

parallel -0 mv {} {=s/foo/bar/g=} ::: *

unixmonkey91589 · 2015-12-25 17:22:43 0
Recompress all files in current directory from gzip to bzip2

Works even if file name contains \n. Spawns one job per core.
This is sample output - yours may be different.
0

parallel -0 'gzip -dc {} | bzip2 > {.}.bz2' ::: *.gz

unixmonkey91589 · 2015-12-25 17:28:03 0
Get the total length of all video / audio in the current dir (and below) in H:m:s

All the other example fail when running in a folder containing too many files due to * being saturated. This command does not use *, allowing me to run it in one folder containing over 300000 audio files. As to running on so many files, I used GNU parallel in order to spawn as many processes as cores, tremendously fasting up the process. Show Sample Output
This is sample output - yours may be different.
```
0:11:3
```
0

ls|grep ".wav"|parallel -j$(nproc) soxi -D {}|awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'

jupiter126 · 2017-05-02 21:37:24 0
Parallel Apache Benchmark

(echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/") | parallel -k 'ab -n 10000 -c 15 {}'
This is sample output - yours may be different.
0

cat url_list.txt | parallel -k 'ab -n 10000 -c 15 {}'

emphazer · 2018-05-17 11:23:28 0
Delete all but latest file in a directory

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-1

ls -t1 | sed 1d | parallel -X rm

unixmonkey8046 · 2010-01-28 12:28:18 0
quick integer CPU benchmark

You could have that little benchmark run on all cores in parallel, as a multi-core benchmark or stress test First find the number of cores, then have parallel iterate over that in, well, parallel Show Sample Output
This is sample output - yours may be different.
```
...
41454625947560171969501001538543925124881846970621917597708878154881\
18278788738248093374936943243527734573872009289119068940335579136

real	0m12,698s
user	0m14,461s
sys	0m0,382s
```
-1

time cat /proc/cpuinfo |grep proc|wc -l|xargs seq|parallel -N 0 echo "2^2^20" '|' bc

kostis · 2018-12-06 05:36:55 0
Remove all files previously extracted from a tar(.gz) file.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

tar -tf <file.tar.gz> | parallel rm

unixmonkey8046 · 2010-01-28 08:28:16 0
add all files not under version control to repository

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

svn status |grep '\?' |awk '{print $2}'| parallel -Xj1 svn add

unixmonkey8046 · 2010-01-28 08:47:54 0
archive all files containing local changes (svn)

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
This is sample output - yours may be different.
-2

svn st | cut -c 9- | parallel -X tar -czvf ../backup.tgz

unixmonkey8046 · 2010-01-28 11:43:16 0
List top ten files/directories sorted by size

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This is sample output - yours may be different.
```
33M     a b
17M     libxnee
3.6M    doc
3.0M    gnee
```
-2

du -s * | sort -nr | head | cut -f2 | parallel -k du -sh

unixmonkey8046 · 2010-01-28 12:59:14 3
backup and remove files with access time older than 5 days.

This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
-3

tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5 -type f` 2> /dev/null | parallel -X rm -f

unixmonkey8046 · 2010-01-28 12:41:41 0
< 1 2

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Check RAM size

To create files with specific permission:

Generate a (compressed) pdf from images

use imagemagik convert

Expand shortened URLs

curl(1) is more portable than wget(1) across Unices, so here is an alternative doing the same thing with greater portability. This shell function uses curl(1) to show what site a shortened URL is pointing to, even if there are many nested shortened URLs. This is a great way to test whether or not the shortened URL is sending you to a malicious site, or somewhere nasty that you don't want to visit. The sample output is from: $ expandurl http://t.co/LDWqmtDM

Append last argument to last command

Just like "!$", except it does it instantly. Then you can hit enter if you want.

Which processes are listening on a specific port (e.g. port 80)

swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

reduce mp3 bitrate (and size, of course)

Useful if you have to put some mp3 files into mobile devices (ie mobile phones with no much memory)

prevents replace an existing file by mistake

Use set +o noclobber and you will be able to replace files again

Find usb device in realtime

Using this command you can track a moment when usb device was attached.

Display which distro is installed

Works on nearly all linux distros

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged parallel

Commands tagged parallel (39) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands tagged parallel (39)

sorted by