If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency. To speed up you transfers you need to compress the data so you will have less to transfer. So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer. Using ssh "-C" option is not compatible with every ssh version (ssh2 for instance).
Part of the "atool" package
- recompresses all gz files to bz2 files from this point and below in the directory tree
- output shows the size of the original file, and the size of the new file. Useful.
- conceptually easier to understand than playing tricks with awk and sed.
- don't like output? Use the following line:
for gz in `find . -type f -name '*.gz' -print`; do f=`basename $gz .gz` && d=`dirname $gz` && gunzip -c $gz | bzip2 - -c > $d/$f.bz2 && rm -f $gz ; done
Show Sample Output
Works even if file name contains \n. Spawns one job per core.
Sets the size of the disk to $DISKSIZE so that the percentage readout of pv is correct. set /dev/sdb to whatever your disk is /dev/sdX. Next pipe dd to pv, then pipe pv to gzip so that you get a gzipped image file. Show Sample Output
untar in place with out creating a temporary file
Create a single tar.gz archive I know it's a very basic one, but it's one I keep forgetting. Show Sample Output
use this command to gzip the file and write to stdout and from the stdout redirect to the another file
This solution is similar to [1] except that it does not have any dependency on GNU Parallel. Also, it tries to minimize the impact on the running system (using ionice and nice). [1] http://www.commandlinefu.com/commands/view/7009/recompress-all-.gz-files-in-current-directory-using-bzip2-running-1-job-per-cpu-core-in-parallel
Should do exactly the same - compress every file in the current directory. You can even use it recursively:
gzip -r .
Where foodir is the directory you want to zip up.
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for: