Backup to LTO Tape with progress, checksums and buffering

bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !

0
By: johnr
2014-07-22 15:47:50

What Others Think

just to be sure: this doens't save file list and checksum to tape right?
MSN · 456 weeks and 5 days ago
where to get shrooms dried mushrooms african pyramid mushroom african transkei mushroom albino penis envy b+ mushrooms blue meanies mushroom brazilian magic mushroom burmese mushroom golden teacher mushroom ko samui super strain malabar coast martinique mexican mushroom mushroom british columbia super thai mushroom nepalese chitwan mushroom penis envy texas penis envy 6 texas yellow caps vietnamese white jedi mind fuck tri coloured ecuadorian mushroom magic mushsroom edibles mexican cubensis psychedelic chocolate bar where to get shrooms dried mushrooms african pyramid mushroom african transkei mushroom albino penis envy b+ mushrooms blue meanies mushroom brazilian magic mushroom burmese mushroom golden teacher mushroom ko samui super strain malabar coast martinique mexican mushroom mushroom british columbia super thai mushroom nepalese chitwan mushroom penis envy texas penis envy 6 texas yellow caps vietnamese white jedi mind fuck tri coloured ecuadorian mushroom magic mushsroom edibles mexican cubensis psychedelic chocolate bar midnight mint psychedelic chocolate bar origin mango psilo gummies 0.25g-1g origin strawberry psilo gummies 0.25g-1g penis envy psychedelic chocolate bar psilocybin cube gummies vegan dark chocolate psychedelic chocolate bar microdoses brain capsules 50mg-200mg chill capsules 50mg-200mg clarity capsules 50mg-200mg health capsules 50mg-200mg love capsules 50mg-200mg mood capsules 50mg-200mg pure capsules 50mg-200mg social capsules 30 pack 350mg social capsules 5 pack 350mg buy weed online without medical card marijuana flowers types of concentrates cannabis cartridges buy edibles online legal marijuana shatter cannabis wax marijuana hash buy psychedelic online tinctures for anxiety cannabis seeds bank weed smoker accessories birthday cake kush acdc strain ace killer og strain ak 47 strain alaskan thunder fuck alien kush strain ambrosia kush strain atomik moon rocks cbd 40.57% atomik moon rocks diamond 58.77% banana og strain bc big bud strain birthday cake kush blue cheese strain blue dream og kush blue moonrock kush blue rhino strain blueberry kush strain bruce banner strain cannabis trim cherry pie strain cannatonic strain citrix strain dark star kush durban poison strain eleven roses strain g13 haze strain gelato strain girl scout cookies flavors girl scout cookies online granddad purple strain grape nana weed strains grape pie strain green crack strain guava kush strain hawaiian skunk jack herer strain mendo breath weed mighty mango mr nice guy weed blue mystic strain northern lights strain northern lights moon rock platinum og kush gas og strain pink og kush platinum hawaiian strain pootie tang strain purple mimosa strain purple white widow strain smart buds sour diesel weed strain sour moonrock strain space monkey tins strawberry banana weed sun rocks super citrus haze strain super silver haze wedding cake strain exotic white widows strain zombie kush strain all natural pistachio crunch apple fritter vape cartridge aura edibles gummy-candy banana og cartridge birthday cake dank vape pen cartridg biscotti vape cartridge blackberry kush cartridge bliss edibles 300mg thc blue dream cartridge 180mlg berry blue raspberry 500mg-chocolate chip- cookies atomic wheelchair activated thc distillate strike gold blue dream shatter bubba kush shatter cbd dried organic ginger blueberry kush vape cartridges chemdawg shatter 50mg choco chunk cookies cookies and cream vape delta 8 thc capsules do-si-dos cartridges durban poison vape cartridge eds and bills edible candy bags forbidden fruit cartridge fruity pebbles cartridge gelato cartridge 1000mg girl scout cookies shatter girl scout cookies wax gorilla glue #4 wax gorilla glue cartridge gorilla glue shatter grand daddy purple kush wax gsc cartridge hardcore og cartridge jack herer cartridge jack herer wax jet fuel cartridge juicy melon cartridge khalifa kush shatter kosher kush cartridge labanese camel hash lemon berry cartridge
johnson20007 · 148 weeks and 1 day ago
Amazing! This blog looks just like my old one! It's on a completely different subject but it has pretty much the same layout and design. Wonderful choice of colors! Pug Puppies for Sale Near Me PUG PUPPY FOR SALE NEAR ME PUG PUPPIES FOR SALE pug puppies for sale in kentucky Pug Puppies for Sale Under $500 Near Me pug puppies for sale in texas pugs puppies for sale teacup pugs for sale pug puppies for sale by owner pug puppies ohio
rahimhh21 · 119 weeks and 4 days ago
Pug Puppies for Sale Near Me pugs puppies for sale teacup pugs for sale pug puppies for sale by owner pug puppies ohio PUG PUPPY FOR SALE NEAR ME PUG PUPPIES FOR SALE pug puppies for sale in kentucky Pug Puppies for Sale Under $500 Near Me pug puppies for sale in texas pug puppies for sale $200 pugs for sale near me under $500 pugs for sale under $400 near me pugs for sale near me puppies for sale near me under $500 pug puppies for sale under $1,000 near me pug for sale pug puppies for sale under $300 Brindle Pug Pitbull Pug Mix Pugs for sale cheap Cheap pug affordable pug puppies for sale near me black pugs for sale near me White Pugs for sale pug dog for sale free pug puppies pug puppies for sale in my area mn pug breeders pug puppies indiana pugs for sale michigan PUG PUPPY ADOPTION Pug puppies for sale Pug puppies for sale near me Pug puppies near me Pug Puppies for Sale Under $500 Near Me Cute Pug Puppies Black pug puppies Black pug puppies for sale pug puppies for adoption black pug puppies for sale near me chihuahua pug mix puppies how much is a pug puppy teacup pug puppies baby pug puppies pictures of pug puppies pug puppies for sale in Ohio pug puppies price pug mix puppies teacup pug puppies for sale best food for pug puppy newborn pug puppies pug puppies craigslist pug puppies for sale craigslist adorable pug puppies how much does a pug puppy cost Pitbull pug mix puppies pug pit mix puppy pug puppies for sale $200 pug puppies for sale in NJ Pug puppies for sale in Wisconsin pug puppy cost pug puppy food royal canin pug puppy royal canin pug puppy food fawn pug puppy pug puppies for sale florida pug puppies for sale in Indiana pug puppies for sale in KY/A> pug puppies for sale in NC pug dog puppy AKC Registered Pug Puppies For sale cheap pug puppies for sale near me cheap pug puppies for sale in California cheap pug puppies for sale in nj Black Pug Puppies for sale pugs puppies for sale
AnthonyLewis9201 · 113 weeks and 5 days ago
Pug Puppies for Sale Near Me pugs puppies for sale teacup pugs for sale pug puppies for sale by owner pug puppies ohio PUG PUPPY FOR SALE NEAR ME PUG PUPPIES FOR SALE pug puppies for sale in kentucky Pug Puppies for Sale Under $500 Near Me pug puppies for sale in texas pug puppies for sale $200 pugs for sale near me under $500 pugs for sale under $400 near me pugs for sale near me puppies for sale near me under $500 pug puppies for sale under $1,000 near me pug for sale pug puppies for sale under $300 Brindle Pug Pitbull Pug Mix Pugs for sale cheap Cheap pug affordable pug puppies for sale near me black pugs for sale near me White Pugs for sale pug dog for sale free pug puppies pug puppies for sale in my area mn pug breeders pug puppies indiana pugs for sale michigan PUG PUPPY ADOPTION Pug puppies for sale Pug puppies for sale near me Pug puppies near me Pug Puppies for Sale Under $500 Near Me Cute Pug Puppies Black pug puppies Black pug puppies for sale pug puppies for adoption black pug puppies for sale near me chihuahua pug mix puppies how much is a pug puppy teacup pug puppies baby pug puppies baby pug puppies for sale pictures of pug puppies pug puppies for sale in Ohio pug puppies price pug mix puppies teacup pug puppies for sale best food for pug puppy newborn pug puppies pug puppies craigslist pug puppies for sale craigslist adorable pug puppies how much does a pug puppy cost Pitbull pug mix puppies pug pit mix puppy pug puppies for sale $200 pug puppies for sale in NJ Pug puppies for sale in Wisconsin pug puppy cost pug puppy food royal canin pug puppy royal canin pug puppy food fawn pug puppy pug puppies for sale florida pug puppies for sale in Indiana pug puppies for sale in KY pug puppies for sale in NC pug dog puppy AKC Registered Pug Puppies For sale cheap pug puppies for sale near me cheap pug puppies for sale in California cheap pug puppies for sale in nj Black Pug Puppies for sale pugs puppies for sale
Perfecthomepugs · 109 weeks and 5 days ago
Perfect House Of Pugs acquired a passion for animals, and the Pug puppy in particular, at a young age since I grew up in a house with lots of dogs. It was a natural ability to know how to grow, nurture, and care for animals. I knew what the future held when I met my wife, who essentially had the same enthusiasm as I have. Being a breeder is more of a passion for me than a profession since I get to witness people's faces light up when they successfully adopt a pug puppy from our facility. It is not possible to find happiness in anything else. PLEASE VISIT OUR WEBSITE LINK BELLOW FOR MORE INFORMATION ABOUT PUG PUPPIES. PUG PUPPY ADOPTION Pug puppies for sale PUG PUPPIES FOR SALE NEAR ME PUG PUPPIES NEAR ME PUG PUPPIES FOR SALE UNDER 500 NEAR ME CUTE PUG PUPPIES BLACK PUG PUPPIES BLACK PUG PUPPIES FOR SALE PUG PUPPIES FOR ADOPTION BLACK PUG PUPPIES FOR SALE NEAR ME CHIHUAHUA PUG MIX PUPPIES TEACUP PUG PUPPIES HOW MUCH IS A PUG PUPPY BABY PUG PUPPIES PICTURES OF PUG PUPPIES PUG PUPPIES FOR SALE IN OHIO PUG PUPPIES FOR SALE IN PA PUG PUPPIES PRICE PUG MIX PUPPIES TEACUP PUG PUPPIES FOR SALE BABY PUG PUPPIES FOR SALE BEST FOOD FOR PUG PUPPY NEWBORN PUG PUPPIES PUG PUPPIES CRAIGSLIST PUG PUPPIES FOR SALE CRAIGSLIST WHITE PUG PUPPIES PUG PUPPIES FOR SALE IN MICHIGAN PUG PUPPIES FOR SALE UNDER 500 ADORABLE PUG PUPPIES BRINDLE PUG PUPPY HOW MUCH DOES A PUG PUPPY COST PITBULL PUG MIX PUPPIES PUGS FOR SALE NEAR ME PUG PUPPIES FOR SALE 200 PUG PUPPIES FOR SALE IN NJ PUG PUPPIES FOR SALE IN WISCONSIN PUG PUPPIES MICHIGAN PUG PUPPY COST MERLE PUG PUPPY FOR ADOPTION BEST PUG PUPPY FOOD ROYAL CANIN PUG PUPPY FOOD FAWN PUG PUPPY PUG PUPPIES FOR SALE FLORIDA PUG PUPPIES FOR SALE IN INDIANA PUG PUPPIES FOR SALE IN KY PUG PUPPIES FOR SALE IN NC PUG PUPPIES FOR SALE IN TEXAS PUG PUPPIES FOR SALE MN PUG DOG PUPPY AKC REGISTERED PUG PUPPIES FOR SALE PUG ALL BLACK CHEAP PUG PUPPIES FOR SALE NEAR ME PUGS PUPPIES FOR SALE CHEAP PUG PUPPIES FOR SALE IN CALIFORNIA MINIATURE PUG PUPPIES FOR SALE NEAR ME MINIATURE PUG PUPPIES FOR SALE CHEAP PUG PUPPIES FOR SALE IN NJ KNOWN FACTS ABOUT PUG PUPPIES ADOPT PUG PUPPIS RECUE A PUG PUPPY PERFECT HOUSE OF PUGS PUG PUPPIES INSTAGRAM PINTEREST QUORA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA USA
pugpuppies95 · 71 weeks and 6 days ago
Acquista Ossicodone ItaliaAcquista Ossicodone Italia  Acquista cocaina crack in linea cocaina a squame di pesce italia Acquista cocaina boliviana in linea Acquista Saxenda in Linea Acquista fentermina in linea italia Acquista Ozempic In Linea Acquista Stilnox in linea italia Acquista Ambien in linea italia Acquista Quaaludes in linea Acquista Metaqualone In Linea Acquista subutex in linea italia Acquista Adderall In linea italia Acquista Ossicodone 10 mg Sandoz italia Acquista Laudano droga online Acquista ossicodone pillole Acquista desoxyn in linea italia Acquista buprenorfina in linea italia Acquista ritalin in linea italia Acquista dilaudid in linea italia Acquista ritalin in linea italia Acquista Ossicodone 30 mg Sandoz italia Acquista fentanil in linea italia Acquista Ossicodone Italia Acquista Ossicodone Italia Acquista la morfina iniettabile Italia Acquista Ossicodone 80 mg Sandoz italia Acquista Xanax in linea italia Acquista strisce di Suboxone italia Acquista vyvanse in linea italia Acquista oxycontin in linea italia Acquista Percocet in linea italia Acquista ossicodone pillole in linea italia Acquista morfina in linea italia Acquista ossicodone in linea italia Buy wood online Buy wood pellets online  dried-pods for sale  opium-pods  for sale dried-papaver-pods  dried poppy pods in USA free dried podsdried poppy pods   körkort sverigekörkort sverige köp körkort köp falsk pass BESTÄLLA KÖRKORT ONLINE buy dried poppy pods online buy opium pods buy opium pods Buy dried poppies online dried poppy pods unwashed poppy pods for sale Dried poppy pods near me Poppy seed tea dried poppy pods for craft wholesale dried flowers USA Giant dried poppy heads dried poppy pods dried poppy pods   Führerschein kaufen ohne prüfungkaufen MPU GutachtenKaufen bootsführerschein onlineSchweizer Führerschein kaufen führerschein polen kaufen ohne prüfung FÜHRERSCHEIN ONLINE KAUFEN https://eu-fuhrerscheinlegal.com/product/kaufen-deutschland-fuhrerschein/ Buy wood online Buy wood pellets online
donperi1 · 64 weeks and 5 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: