Commands tagged mt-st (1)

  • This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !


    0
    bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
    johnr · 2014-07-22 15:47:50 8

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Show crontabs for all users
This is flatcaps tweaked command to make it work on SLES 11.2

print crontab entries for all the users that actually have a crontab
This is how I list the crontab for all the users on a given system that actually have a crontab. You could wrap it with a function block and place it in your .profile or .bashrc for quick access. There's prolly a simpler way to do this. Discuss.

Easy to extend one-liner for cron scripts that automate filesystem checking
This one-liner is for cron jobs that need to provide some basic information about a filesystem and the time it takes to complete the operation. You can swap out the di command for df or du if that's your thing. The |& redirections the stderr and stdout to the mail command. How to configure the variables. TOFSCK=/path/to/mount FSCKDEV=/dev/path/device or FSCKDEV=`grep $TOFSCK /proc/mounts | cut -f1 -d" "` MAILSUB="weekly file system check $TOFSCK "

Insert a line at the top of a text file without sed or awk or bash loops
Just use '-' to use STDIN as an additional input to 'cat'

Backup files older than 1 day on /home/dir, gzip them, moving old file to a dated file.
Useful for backing up old files, custom logs, etc. via a cronjob.

Create a simple video contact sheet using the vcs bash script
Assumes you've downloaded Toni Corvera's vcs script (http://p.outlyer.net/vcs), have it in your PATH, and have installed the script's dependencies. Generates a video contact sheet of 24 thumbnails and 3 thumbnails per column. The bold font and white-on-black color scheme keeps the text readable at the chosen 70% JPEG compression quality, which keeps the file size at a manageable level. You can go even lower with the quality and get a good looking result.

reset an hanging terminal session
when your terminal session seems unrensponsive (this normally happen after outputting some binary data directly on your standard output) it may me saned by hitting: CTRL+J tput sgr0 CTRL+J Note: don't press the Enter key, just ctrl+j

Write on the console without being registered
just use a space to prevent commands from being recorded in bash's history on most systems

Replace underscores with spaces in filenames and dirnames, recursively into subdirs.
Everyone wants to take spaces out of filenames. Forget that. I want to put them back in. We've got tools and filesystems that support spaces, they look better, so I'm going to use them. Because of how find works I find I need to run this multiple times, if it's renaming subdirs. But it can be re-run without issues. I got this version of the command from a comment in this underscore-generating command. http://www.commandlinefu.com/commands/view/760/find-recursively-from-current-directory-down-files-and-directories-whose-names-contain-single-or-multiple-whitespaces-and-replace-each-such-occurrence-with-a-single-underscore. All I did was change the regex.

Calculate days on which Friday the 13th occurs (inspired from the work of the user justsomeguy)
Friday is the 5th day of the week, monday is the 1st. Output may be affected by locale.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: