Commands tagged pv (38)

  • NOTE: When doing these commands when asked for questions there might be flowing text from the pv doing the progress bar just continue typing as if its not there, close your eyes if it helps, there might be a yes or no question, type "yes" and ENTER to it, and also it will ask for a password, just put in your password and ENTER I talk alot more about this and alot of other variations of this command on my site: http://www.kossboss.com/linuxtarpvncssh Show Sample Output


    0
    cd /srcfolder; tar -czf - . | pv -s `du -sb . | awk '{print $1}'` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com "tar -xzvf - -C /dstfolder"
    bhbmaster · 2013-05-30 07:21:06 2
  • forgot to use a pv or rsync and want to know how much has been copied. Show Sample Output


    0
    watch ls -lh /path/to/folder
    vonElfensenf · 2014-03-27 10:51:36 4
  • This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !


    0
    bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
    johnr · 2014-07-22 15:47:50 5

  • 0
    (pv -n centos-7.0-1406-x86_64-DVD.img | dd of=/dev/disk4 bs=1m conv=notrunc,noerror) 2>&1 | dialog --gauge "Copying CentOS to USB Stick in /dev/disk4" 10 70 0
    BoxingOctopus · 2015-01-19 19:36:15 8
  • Create a file with random binary content. Required pv, units packages. It use openssl to encrypt zeros using aes-256 and time stamp as password to generate a pseudo-random file. Show Sample Output


    0
    s=1G bs=16K; count=`units ${s}iB ${bs}iB -1 -t --out="%.f"`; openssl enc -aes-256-ctr -pass pass:`date +%s%N` -nosalt < /dev/zero 2>/dev/null | dd iflag=fullblock bs=$bs count=$count | tee $s | pv -s $s | md5sum | sed -e "s/-/$s/" > ${s}.md5
    jcppkkk · 2015-09-30 06:27:39 5
  • Due to @tremby, here: http://unix.stackexchange.com/a/172088/58343. I'm looking for a way to parallelize openssl and feed that to dd since openssl is the bottleneck on my machine: http://unix.stackexchange.com/questions/253466/parallelize-openssl-as-input-to-dd


    0
    openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | pv --progress --eta --rate --bytes --size 8000632782848 | dd of=/dev/md0 bs=2M
    diagon · 2016-01-05 19:36:03 13
  • first line is the speed the uncompressed data is read, second line is the compressed data sent over ssh. change sdb to your target drive/partition to be backed up. change pbzip -c1 to suit your compression. and ssh to your target file. don't forget to run zerofree/fstrim first! Show Sample Output


    0
    dd if=/dev/sdb | pv -rabc | pbzip2 -c1 | pv -rabc | ssh user@192.168.0.1 'cat > /dump.bz2'
    sexyrms · 2016-06-19 23:27:03 6
  • Sets the size of the disk to $DISKSIZE so that the percentage readout of pv is correct. set /dev/sdb to whatever your disk is /dev/sdX. Next pipe dd to pv, then pipe pv to gzip so that you get a gzipped image file. Show Sample Output


    0
    DISKSIZE=`sudo blockdev --getsize64 /dev/sdb` && sudo dd bs=4096 if=/dev/sdb | pv -s $DISKSIZE | sudo gzip -9 > ~/USBDRIVEBACKUP.img.gz
    frame45 · 2016-08-31 00:03:56 7
  • uses the wonderful 'pv' command to give a progress bar when copying one partition to another. Amazing for long running dd commands Show Sample Output


    0
    pv -tpreb /dev/sdc2 | dd of=/dev/sdb2 bs=64K conv=noerror,sync
    4fthawaiian · 2016-12-22 03:18:09 6

  • 0
    pv file.txt | tee >(sha1sum > file.sha1) > file-copy.txt
    bugmenot · 2022-11-24 20:23:02 49
  • Change your drive letter as you wish. Using pv command for speed detect.First of all you must install pv command for usage. http://www.bayner.com/ kerim@bayner.com Show Sample Output


    -1
    cat /dev/sda | pv -r > /dev/null
    kerim · 2011-01-23 22:58:56 4
  • Only works on single files, doesn't preserve permissions/timestamps/ownership. Show Sample Output


    -6
    pv file1 > file2
    ppaschka · 2010-02-25 19:18:32 2
  • the f is for file and - stdout, This way little shorter. I Like copy-directory function It does the job but looks like SH**, and this doesn't understand folders with whitespaces and can only handle full path, but otherwise fine, function copy-directory () { ; FrDir="$(echo $1 | sed 's:/: :g' | awk '/ / {print $NF}')" ; SiZe="$(du -sb $1 | awk '{print $1}')" ; (cd $1 ; cd .. ; tar c $FrDir/ )|pv -s $SiZe|(cd $2 ; tar x ) ; } Show Sample Output


    -11
    (cd /source/dir ; tar cv .)|(cd /dest/dir ; tar xv)
    marssi · 2009-07-19 10:31:13 11
  •  < 1 2

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

random xkcd comic
i sorta stole this from http://www.shell-fu.org/lister.php?id=878#MTC_form but it didn't work, so here it is, fixed. --- updated to work with jpegs, and to use a fancy positive look behind assertion.

Download full FLAC albums from archive.org
The only zipped version of an album available for download is the lossy mp3 version. To download lossless files, because of their size, you must download them individually. This command scrapes the page for all the FLAC (or also SHN) files.

MSDOS command to check existance of command and exit batch if failed
This is a command to be used inside of MS-DOS batch files to check existence of commands as preconditions before actual batch processing can be started. If the command is found, batch script continues execution. If not, a message is printed on screen, script then waits for user pressing a key and exits. An error message of the command itself is suppressed for clarity purpose.

Play radio stream with mplayer
Above command will play Virgin Radio Dubai

Show directories
Show only the subdirectories in the current directory. In the example above, /lib has 135 files and directories. With this command, the 9 dirs jump out.

Save your open windows to a file so they can be opened after you restart
This will save your open windows to a file (~/.windows). To start those applications: $ cat ~/.windows | while read line; do $line &; done Should work on any EWMH/NetWM compatible X Window Manager. If you use DWM or another Window Manager not using EWMH or NetWM try this: $ xwininfo -root -children | grep '^ ' | grep -v children | grep -v '' | sed -n 's/^ *\(0x[0-9a-f]*\) .*/\1/p' | uniq | while read line; do xprop -id $line _NET_WM_PID | sed -n 's/.* = \([0-9]*\)$/\1/p'; done | uniq -u | grep -v '^$' | while read line; do ps -o cmd= $line; done > ~/.windows

Convert encoding of given files from one encoding to another

Find common lines between two files

Find the package that installed a command

extract email addresses from some file (or any other pattern)


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: