copy working directory and compress it on-the-fly while showing progress

tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz
What happens here is we tell tar to create "-c" an archive of all files in current dir "." (recursively) and output the data to stdout "-f -". Next we specify the size "-s" to pv of all files in current dir. The "du -sb . | awk ?{print $1}?" returns number of bytes in current dir, and it gets fed as "-s" parameter to pv. Next we gzip the whole content and output the result to out.tgz file. This way "pv" knows how much data is still left to be processed and shows us that it will take yet another 4 mins 49 secs to finish. Credit: Peteris Krumins http://www.catonmat.net/blog/unix-utilities-pipe-viewer/
Sample Output
253MB 0:00:05 [46.7MB/s] [>     ]  1% ETA 0:04:49

25
2009-12-18 17:09:08

What Others Think

credit to peteris krumins
opertinicy · 465 weeks and 2 days ago
As tar can also compress you can save a pipe here. Also 'out' is being redirected to pwd, which is the pwd for tar. I would probably do the following (using zsh glob qualifier for the output filename ... but for bash you could use ${key##*/} or what-have-you): tar cfJ - . |pv --size $(du -sb jargon-wl|cut -f1) > ~/backup/${$(pwd):t}.tar.xz (fingers crossed the content manager here don't munge the above command)
khayyam · 293 weeks and 4 days ago
of course you can compress and save a pipe, but then pv won't know hoy much is still left as it's "zipped"... and du won't be.
pulketo · 173 weeks and 4 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: