Useful for checking if a large number of PNG files was downloaded successfully by verifying the built-in CRC checksum. For incomplete files, the command will print: "00002309.png EOF while reading IDAT data ERROR: 00002309.png" The process is very fast; checking 21,000 files of 5MB in size took only five minutes on a 2011 Intel mobile dual-core.
A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/
You need: pxz for the actual work (http://jnovy.fedorapeople.org/pxz/). The function could be better with better multifile and stdin/out support.
This is to overcome the issue of slow I/O by reading once and forwarding the output to several processes (e. g. 3 in the given command). One could also invoke grep or other programs to work on read data. Show Sample Output
Works with any file name: space, ', " and even \n is OK. The code in {= =} is a perl expression.
Works even if file name contains \n. Spawns one job per core.
All the other example fail when running in a folder containing too many files due to * being saturated. This command does not use *, allowing me to run it in one folder containing over 300000 audio files. As to running on so many files, I used GNU parallel in order to spawn as many processes as cores, tremendously fasting up the process. Show Sample Output
(echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/"; echo "https://example.com/") | parallel -k 'ab -n 10000 -c 15 {}'
Useful for download mulitple files Show Sample Output
xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
You could have that little benchmark run on all cores in parallel, as a multi-core benchmark or stress test First find the number of cores, then have parallel iterate over that in, well, parallel Show Sample Output
xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for: