Pipe viewer is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. Source: http://www.catonmat.net/blog/unix-utilities-pipe-viewer/ Show Sample Output

What happens here is we tell tar to create "-c" an archive of all files in current dir "." (recursively) and output the data to stdout "-f -". Next we specify the size "-s" to pv of all files in current dir. The "du -sb . | awk ?{print $1}?" returns number of bytes in current dir, and it gets fed as "-s" parameter to pv. Next we gzip the whole content and output the result to out.tgz file. This way "pv" knows how much data is still left to be processed and shows us that it will take yet another 4 mins 49 secs to finish. Credit: Peteris Krumins http://www.catonmat.net/blog/unix-utilities-pipe-viewer/ Show Sample Output

usage: sitepass MaStErPaSsWoRd example.com description: An admittedly excessive amount of hashing, but this will give you a pretty secure password, It also eliminates repeated characters and deletes itself from your command history. tr '!-~' 'P-~!-O' # this bit is rot47, kinda like rot13 but more nerdy rev # this avoids the first few bytes of gzip payload, and the magic bytes. Show Sample Output

Useful to move many files (thousands or millions files) over ssh. Faster than scp because this way you save a lot of tcp connection establishments (syn/ack packets). If using a fast lan (I have just tested gigabyte ethernet) it is faster to not compress the data so the command would be: tar -cf - /home/user/test | ssh user@sshServer 'cd /tmp; tar xf -'

Use `zless` to read the content of your *rss.gz file: zless commandlinefu-contribs-backup-2009-08-10-07.40.39.rss.gz Show Sample Output

Useful for backing up old files, custom logs, etc. via a cronjob. Show Sample Output

It grabs all the database names granted for the $MYSQLUSER and gzip them to a remote host via SSH.

The command uses ssh(1) to get to a remote host, uses tar(1) to archive a remote directory, prints the result to STDOUT, which is piped to gzip(1) to compress to a local file. In other words, we are archiving and compressing a remote directory to our local box.

Useful for a cron job that runs nightly, gzipping or alternatively deleting files from a specific directory that are older than 10 days (in this case)

check your load with top... Start more of these jobs to get an multi-core cpu busy... Show Sample Output

You set the file/dirname transfer variable, in the end point you set the path destination, this command uses pipe view to show progress, compress the file outut and takes account to change the ssh cipher. Support dirnames with spaces. Merged ideas and comments by http://www.commandlinefu.com/commands/view/4379/copy-working-directory-and-compress-it-on-the-fly-while-showing-progress and http://www.commandlinefu.com/commands/view/3177/move-a-lot-of-files-over-ssh Show Sample Output

This command is meant to be used to make a lightweight backup, for when you want to know which files might be missing or changed, but you don't care about their contents (because you have some way to recover them). Explanation of parts: "ls -RFal /" lists all files in and below the root directory, along with their permissions and some other metadata. I think sudo is necessary to allow ls to read the metadata of certain files. "| gzip" compresses the result, from 177 MB to 16 MB in my case. "> all_files_list.txt.gz" saves the result to a file in the current directory called all_files_list.txt.gz. This name can be changed, of course. Show Sample Output

In this example we convert a .tar.bz2 file to a .tar.gz file. If you don't have Pipe Viewer, you'll have to download it via apt-get install pv, etc. Show Sample Output

Opens a snapshot of a live UFS2 filesystem, runs dump to generate a full filesystem backup which is run through gzip. The filesystem must support snapshots and have a .snap directory in the filesystem root. To restore the backup, one can do zcat /path/to/adXsYz.dump.gz | restore -rf -

This is a useful command to backup an sd card with relative total size for piping to pv with a progressbar

Ran as the postgres user, dumps each database individually. It dumps with the create statements as well, so you can just 'zcat $x-nightly.dmp.gz | psql' to reimport/recreate a database from a backup.