Commands tagged parallel sorted by votes

Commands tagged parallel (40)

sorted by

Instant threadpool
This is sample output - yours may be different.
0

cat item_list | xargs -n1 -P<n> process_item

sanchan89 · 2013-03-06 20:25:17 9
Calculate different hash sums of one file at the same time

This is to overcome the issue of slow I/O by reading once and forwarding the output to several processes (e. g. 3 in the given command). One could also invoke grep or other programs to work on read data. Show Sample Output
This is sample output - yours may be different.
```
ce67efc071187fe6ee2bd7486cf1307e  -
b16de8c6465bd973188d62e5772f21762d624f0f  -
6c801e3a40ab3e5192420d47f676e7291d8572c4144687e90d59335a47c750ee  -
```
0

dd if=file | tee >(sha1sum) >(md5sum) >(sha256sum) >/dev/null

dubbaluga · 2013-11-07 17:43:54 0
Multi-thread any command
This is sample output - yours may be different.
0

find . -mtime +1000 -type f

cuevas500 · 2015-12-15 06:43:14 0
bulk rename files with GNU Parallel

Works with any file name: space, ', " and even \n is OK. The code in {= =} is a perl expression.
This is sample output - yours may be different.
0

parallel -0 mv {} {=s/foo/bar/g=} ::: *

unixmonkey91589 · 2015-12-25 17:22:43 0
Recompress all files in current directory from gzip to bzip2

Works even if file name contains \n. Spawns one job per core.
This is sample output - yours may be different.
0

parallel -0 'gzip -dc {} | bzip2 > {.}.bz2' ::: *.gz

unixmonkey91589 · 2015-12-25 17:28:03 0
Get the total length of all video / audio in the current dir (and below) in H:m:s

All the other example fail when running in a folder containing too many files due to * being saturated. This command does not use *, allowing me to run it in one folder containing over 300000 audio files. As to running on so many files, I used GNU parallel in order to spawn as many processes as cores, tremendously fasting up the process. Show Sample Output
This is sample output - yours may be different.
```
0:11:3
```
0

ls|grep ".wav"|parallel -j$(nproc) soxi -D {}|awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'

jupiter126 · 2017-05-02 21:37:24 0
Parallel Apache Benchmark

(echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/") | parallel -k 'ab -n 10000 -c 15 {}'
This is sample output - yours may be different.
0

cat url_list.txt | parallel -k 'ab -n 10000 -c 15 {}'

emphazer · 2018-05-17 11:23:28 1
Check a directory of PNG files for errors

Useful for checking if a large number of PNG files was downloaded successfully by verifying the built-in CRC checksum. For incomplete files, the command will print: "00002309.png EOF while reading IDAT data ERROR: 00002309.png" The process is very fast; checking 21,000 files of 5MB in size took only five minutes on a 2011 Intel mobile dual-core.
This is sample output - yours may be different.
0

ls *.png |parallel --nice 19 --bar --will-cite "pngcheck -q {}"

Negate · 2021-09-05 00:37:07 51
Delete all but latest file in a directory

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-1

ls -t1 | sed 1d | parallel -X rm

unixmonkey8046 · 2010-01-28 12:28:18 0
quick integer CPU benchmark

You could have that little benchmark run on all cores in parallel, as a multi-core benchmark or stress test First find the number of cores, then have parallel iterate over that in, well, parallel Show Sample Output
This is sample output - yours may be different.
```
...
41454625947560171969501001538543925124881846970621917597708878154881\
18278788738248093374936943243527734573872009289119068940335579136

real	0m12,698s
user	0m14,461s
sys	0m0,382s
```
-1

time cat /proc/cpuinfo |grep proc|wc -l|xargs seq|parallel -N 0 echo "2^2^20" '|' bc

kostis · 2018-12-06 05:36:55 0
Remove all files previously extracted from a tar(.gz) file.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

tar -tf <file.tar.gz> | parallel rm

unixmonkey8046 · 2010-01-28 08:28:16 0
add all files not under version control to repository

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

svn status |grep '\?' |awk '{print $2}'| parallel -Xj1 svn add

unixmonkey8046 · 2010-01-28 08:47:54 0
archive all files containing local changes (svn)

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
This is sample output - yours may be different.
-2

svn st | cut -c 9- | parallel -X tar -czvf ../backup.tgz

unixmonkey8046 · 2010-01-28 11:43:16 0
List top ten files/directories sorted by size

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This is sample output - yours may be different.
```
33M     a b
17M     libxnee
3.6M    doc
3.0M    gnee
```
-2

du -s * | sort -nr | head | cut -f2 | parallel -k du -sh

unixmonkey8046 · 2010-01-28 12:59:14 3
backup and remove files with access time older than 5 days.

This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
-3

tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5 -type f` 2> /dev/null | parallel -X rm -f

unixmonkey8046 · 2010-01-28 12:41:41 0
< 1 2

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

dump a single table of a database to file

Shows size of dirs and files, hidden or not, sorted.

Enhanced version: fixes sorting by human readable numbers, and filters out non MB or GB entries that have a G or an M in their name.

Check (partial) runtime-dependencies of Gentoo ebuilds

The output is only partial because runtime dependencies should count in also commands executed via system() and libraries loaded with dlopen(), but at least it gives an idea of what a package directly links to. Note: this is meaningful *only* if you're using -Wl,--as-needed in your LDFLAGS, otherwise it'll bring you a bunch of false positives.

list files recursively by size

Shows how many percents of all avaliable packages are installed in your gentoo system

docker kill all containers

Limit memory usage per script/program

When I'm testing some scripts or programs, they end up using more memory than anticipated. In that case, computer nearly halts due to swap space usage, and sometimes I have to press Magic SysRq+REISUB to reboot. So, I was looking for a way to limit memory usage per script and found out that ulimit can limit memory. If you run it this way: $ $ ulimit -v 1000000 . $ $ scriptname Then the new memory limit will be valid for that shell. I think changing the limit within a subshell is much more flexible and it won't interfere with your current shell ulimit settings. note: -v 1000000 corresponds to approximately 1GB of RAM

Grep syslog today last hour

Uses date to grep de logfile for today and uses it to get the last hour logs. Can be used to get last minute logs or today's logs.

Convert all .flac from a folder subtree in 192Kb mp3

find . -type f -iname '*.flac' # searches from the current folder recursively for .flac audio files | # the output (a .flac audio files with relative path from ./ ) is piped to while read FILE; do FILENAME="${FILE%.*}"; flac -cd "$FILE" | lame -b 192 - "${FILENAME}.mp3"; done # for each line on the list: # FILE gets the file with .flac extension and relative path # FILENAME gets FILE without the .flac extension # run flac for that FILE with output piped to lame conversion to mp3 using 192Kb bitrate

Leap year calculation

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged parallel

Commands tagged parallel (40) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands tagged parallel (40)

sorted by