Commands tagged mt-st (1)

  • This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !


    0
    bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
    johnr · 2014-07-22 15:47:50 8

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

most used commands in history (comprehensive)
Most of the "most used commands" approaches does not consider pipes and other complexities. This approach considers pipes, process substitution by backticks or $() and multiple commands separated by ; Perl regular expression breaks up each line using | or < ( or ; or ` or $( and picks the first word (excluding "do" in case of for loops) note: if you are using lots of perl one-liners, the perl commands will be counted as well in this approach, since semicolon is used as a separator

check open ports without netstat or lsof

Move a folder and merge it with another folder
This will move a folder and merge it with another folder which may contain duplicates. Technically it's just creating hardlinks of everything in the folder, and after it's done, delete the source (with rm -r source/ ) to complete the move. This is much faster than, for example, using rsync to merge folders which would actually copy the entire contents and so for a lot of files would take much longer. This uses macutils gcp port of cp so it can be used on osx/MacOS. If using in linux or some unix where cp includes the ability to create links with -l you can just use cp instead of gcp.

download a list of urls

Find the package that installed a command

Generate a random password 30 characters long

Generate a random left-hand password
Generates a random 8-character password that can be typed using only the left hand on a QWERTY keyboard. Useful to avoid taking your hand off of the mouse, especially if your username is left-handed. Change the 8 to your length of choice, add or remove characters from the list based on your preferences or kezboard layout, etc.

vi a new file with execution mode
$ vix /tmp/script.sh Open a file directly with execution permission. Put the function in your .bashrc You can also put this in your vimrc: $ command XX w | set ar | silent exe "!chmod +x %" | redraw! and open a new file like this: $ vi +XX /tmp/script.sh

Extract tarball from internet without local saving

Swap a file or dir with quick resotre
This lets you replace a file or directory and quickly revert if something goes wrong. For example, the current version of a website's files are in public_html. Put a new version of the site in public_html~ and execute the command. The names are swapped. If anything goes wrong, execute it again (up arrow or !!).


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: