commandlinefu.com is the place to record those command-line gems that you return to again and again.
You can sign-in using OpenID credentials, or register a traditional username and password.
Subscribe to the feed for:
This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray.
Go to the directory where you want to write the output files : cd /bklogs
Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www".
It will create a tar and write it to the tape drive on /dev/nst0.
In the process, it will
1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact
2) generate a filelist of the content of the tar with filesize to $bkname.lst
3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz.
4) show buffer in/out speed and used space in the buffer
5) show progress bar with time approximation using pv
To eject the tape :
; sleep 75; mt-st -f /dev/nst0 rewoffl
1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it."
2) Fix filesize approximation
3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?)
4) Support spanning on multiple tape.
5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project.
1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing.
2) Remove the v in tar argument cvf to prevent listing all files added to the archive.
3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/)
and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !
show off how big your disks are
With this command, you can check the difference between the volumes mounted and the volume in /etc/fstab.
Show disk space info, grepping out the uninteresting ones beginning with ^none while we're at it.
The main point of this submission is the way it maintains the header row with the command grouping, by removing it from the pipeline before it gets fed into the sort command. (I'm surprised sort doesn't have an option to skip a header row, actually..)
It took me a while to work out how to do this, I thought of it as I was drifting off to sleep last night!
Display the size (human reading) of all the directories in your home path (~).
Reports all local partitions having more than 90% usage.
Just add it in a crontab and you'll get a mail when a disk is full.
(sending mail to the root user must work for that)
Put into some file. No special purpouse, just for fun...
For disk space constraint testing. Leaves a little space available for creating temp files, etc. Easily free up the used disk space again by deleting the dummy00 file. Can tailor the testing by building smaller 'blocks' to suit the needs of the testing.
WARNING: do not do this to the '/' (root) filesystem unless you know what you are doing... on some systems it could crash the OS.
Oracle DBA remove some logfiles which are still open by the database and he is complaining the space has not been reclaimed? Use the above command to find out what PID needs to be stopped. Or alternatively recover the file via:
cp /proc/pid/fd/filehandle /new/file.txt