Commands tagged parallel sorted by votes

Commands tagged parallel (43)

sorted by

Find Duplicate Files (based on size first, then MD5 hash)

A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
0

find -not -empty -type f -printf "%s\n" | sort | uniq -d | parallel find -type f -size {}c | parallel md5sum | sort | uniq -w32 --all-repeated=separate

unixmonkey8046 · 2010-01-28 08:40:18 3
Parallel XZ with progress bar

You need: pxz for the actual work (http://jnovy.fedorapeople.org/pxz/). The function could be better with better multifile and stdin/out support.
This is sample output - yours may be different.
0

function xzv() { THREADS=`grep processor /proc/cpuinfo | wc -l`; for file in $*; do pv -s `stat -c%s $file` < $file | pxz -q -T $THREADS > $file.xz ; done; }

oernii2 · 2011-12-14 08:22:08 4
Instant threadpool
This is sample output - yours may be different.
0

cat item_list | xargs -n1 -P<n> process_item

sanchan89 · 2013-03-06 20:25:17 12
Calculate different hash sums of one file at the same time

This is to overcome the issue of slow I/O by reading once and forwarding the output to several processes (e. g. 3 in the given command). One could also invoke grep or other programs to work on read data. Show Sample Output
This is sample output - yours may be different.
```
ce67efc071187fe6ee2bd7486cf1307e  -
b16de8c6465bd973188d62e5772f21762d624f0f  -
6c801e3a40ab3e5192420d47f676e7291d8572c4144687e90d59335a47c750ee  -
```
0

dd if=file | tee >(sha1sum) >(md5sum) >(sha256sum) >/dev/null

dubbaluga · 2013-11-07 17:43:54 4
Multi-thread any command
This is sample output - yours may be different.
0

find . -mtime +1000 -type f

cuevas500 · 2015-12-15 06:43:14 5
bulk rename files with GNU Parallel

Works with any file name: space, ', " and even \n is OK. The code in {= =} is a perl expression.
This is sample output - yours may be different.
0

parallel -0 mv {} {=s/foo/bar/g=} ::: *

unixmonkey91589 · 2015-12-25 17:22:43 5
Recompress all files in current directory from gzip to bzip2

Works even if file name contains \n. Spawns one job per core.
This is sample output - yours may be different.
0

parallel -0 'gzip -dc {} | bzip2 > {.}.bz2' ::: *.gz

unixmonkey91589 · 2015-12-25 17:28:03 3
Get the total length of all video / audio in the current dir (and below) in H:m:s

All the other example fail when running in a folder containing too many files due to * being saturated. This command does not use *, allowing me to run it in one folder containing over 300000 audio files. As to running on so many files, I used GNU parallel in order to spawn as many processes as cores, tremendously fasting up the process. Show Sample Output
This is sample output - yours may be different.
```
0:11:3
```
0

ls|grep ".wav"|parallel -j$(nproc) soxi -D {}|awk '{SUM += $1} END { printf "%d:%d:%d\n",SUM/3600,SUM%3600/60,SUM%60}'

jupiter126 · 2017-05-02 21:37:24 8
Parallel Apache Benchmark

(echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/") | parallel -k 'ab -n 10000 -c 15 {}'
This is sample output - yours may be different.
0

cat url_list.txt | parallel -k 'ab -n 10000 -c 15 {}'

emphazer · 2018-05-17 11:23:28 20
Run 10 curl commands in parallel via xargs

Useful for download mulitple files Show Sample Output
This is sample output - yours may be different.
```
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
xx.xxx.xxx.xx
```
0

NUM="10";seq ${NUM}|time xargs -I % -n1 -P${NUM} curl -sL ifconfig.co

wuseman1 · 2022-06-02 02:08:05 170
Unzip 25 zip files files at once
This is sample output - yours may be different.
0

find . -maxdepth 1 -name '*.zip' -print0 | xargs -0 -I {} -P 25 unzip {}

wuseman1 · 2022-11-20 14:07:44 92
Delete all but latest file in a directory

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-1

ls -t1 | sed 1d | parallel -X rm

unixmonkey8046 · 2010-01-28 12:28:18 2
quick integer CPU benchmark

You could have that little benchmark run on all cores in parallel, as a multi-core benchmark or stress test First find the number of cores, then have parallel iterate over that in, well, parallel Show Sample Output
This is sample output - yours may be different.
```
...
41454625947560171969501001538543925124881846970621917597708878154881\
18278788738248093374936943243527734573872009289119068940335579136

real	0m12,698s
user	0m14,461s
sys	0m0,382s
```
-1

time cat /proc/cpuinfo |grep proc|wc -l|xargs seq|parallel -N 0 echo "2^2^20" '|' bc

kostis · 2018-12-06 05:36:55 459
Remove all files previously extracted from a tar(.gz) file.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

tar -tf <file.tar.gz> | parallel rm

unixmonkey8046 · 2010-01-28 08:28:16 2
add all files not under version control to repository

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

svn status |grep '\?' |awk '{print $2}'| parallel -Xj1 svn add

unixmonkey8046 · 2010-01-28 08:47:54 2
archive all files containing local changes (svn)

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
This is sample output - yours may be different.
-2

svn st | cut -c 9- | parallel -X tar -czvf ../backup.tgz

unixmonkey8046 · 2010-01-28 11:43:16 2
List top ten files/directories sorted by size

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This is sample output - yours may be different.
```
33M     a b
17M     libxnee
3.6M    doc
3.0M    gnee
```
-2

du -s * | sort -nr | head | cut -f2 | parallel -k du -sh

unixmonkey8046 · 2010-01-28 12:59:14 5
backup and remove files with access time older than 5 days.

This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
-3

tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5 -type f` 2> /dev/null | parallel -X rm -f

unixmonkey8046 · 2010-01-28 12:41:41 2
< 1 2

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Export unpushed files list

kill process by name

Or even easier, if it's available: $ killall firefox I have no idea why you would want to rely on two unusual dependencies to do something that can be done a hundred ways from coreutils...

Kill all processes belonging to a user

Convert records in columns to csv

Get your Firefox bookmarks

Extracts yours bookmarks out of sqlite with the format: dateAdded|url

Pick a random line from a file

List programs with open ports and connections

I prefer to use this and not the -n variety, so I get DNS-resolved hostnames. Nice when I'm trying to figure out who's got that port open.

Change the primary group of a user

Rename files in batch

List latest 5 modified files recursively

The output format is given by the -printf parameter: %T@ = modify time in seconds since Jan. 1, 1970, 00:00 GMT, with fractional part. Mandatory, hidden in the end. %TY-%Tm-%Td %TH:%TM:%.2TS = modify time as YYYY-MM-DD HH:MM:SS. Optional. %p = file path Refer to http://linux.die.net/man/1/find for more about -printf formatting. ------------------------ sort -nr = sort numerically and reverse (higher values - most recent timestamp - first) head -n 5 = get only 5 first lines (change 5 to whatever you want) cut -f2- -d" " = trim first field (timestamp, used only for sorting) ------------------------ Very useful for building scripts for detecting malicious files upload and malware injections.

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands tagged parallel

Commands tagged parallel (43) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands tagged parallel (43)

sorted by