Commands tagged dd (72)

  • Assuming we have a disk image, created by dd if=/dev/sda of=image.dd we can check the image's partition layout with fdisk -ul image.dd, then substitute "x" with starting sector of the partition we want to mount. This example assumes that the disk uses 512Byte sectors


    0
    mount -o loop,offset=$((512*x)) /path/to/dd/image /mount/path
    0disse0 · 2011-06-14 19:30:54 7
  • Show running time. eta, progressbar Show Sample Output


    0
    pv -petrs $(stat -c %s file.iso) file.iso | dd bs=1M oflag=sync of=/dev/sdX
    f4m8 · 2012-01-30 07:16:29 5

  • 0
    pv file.iso >/dev/sdX
    qiet72 · 2012-01-30 09:38:15 3
  • The comp.unix.shell posting by St?phane Chazelas also lists the following offsets: type 32768 (1 byte) id 32769 (5 bytes) version 32774 (1 byte) system_id 32776 (32 bytes) volume_id 32808 (32 bytes) volume_space_size 32848 (8 bytes) escape_sequences 32856 (32 bytes) volume_set_size 32888 (4 bytes) volume_sequence_number 32892 (4 bytes) logical_block_size 32896 (4 bytes) path_table_size 32900 (8 bytes) type_l_path_table 32908 (4 bytes) opt_type_l_path_table 32912 (4 bytes) type_m_path_table 32916 (4 bytes) opt_type_m_path_table 32920 (4 bytes) root_directory_record 32924 (34 bytes) volume_set_id 32958 (128 bytes) publisher_id 33086 (128 bytes) preparer_id 33214 (128 bytes) application_id 33342 (128 bytes) copyright_file_id 33470 (37 bytes) abstract_file_id 33507 (37 bytes) bibliographic_file_id 33544 (37 bytes) creation_date 33581 (17 bytes) modification_date 33598 (17 bytes) expiration_date 33615 (17 bytes) effective_date 33632 (17 bytes) file_structure_version 33649 (1 byte) application_data 33651 (512 bytes)


    0
    dd if=/dev/cdrom bs=1 skip=32808 count=32 conv=unblock cbs=32 2>/dev/null
    mlk · 2013-04-14 20:50:58 6
  • This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !


    0
    bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
    johnr · 2014-07-22 15:47:50 8
  • display dd status on OSX (coreutils) every 10 seconds Show Sample Output


    0
    watch -n 10 sudo kill -INFO $(pgrep -l '^dd$' | cut -d ' ' -f 1)
    nfarrar · 2014-11-23 21:21:08 7

  • 0
    (pv -n centos-7.0-1406-x86_64-DVD.img | dd of=/dev/disk4 bs=1m conv=notrunc,noerror) 2>&1 | dialog --gauge "Copying CentOS to USB Stick in /dev/disk4" 10 70 0
    BoxingOctopus · 2015-01-19 19:36:15 12
  • Output appears in terminal where dd is running, not where command is executed. Send a USR1 signal to dd. Finds the most recently run dd command


    0
    sudo pkill -USR1 -n -x dd
    texasflood · 2015-01-25 12:31:56 6
  • Observe the process of your dd command on Mac the Mac bash when you burn an .iso or .img on a SD-Card. This command avoids permission errors.


    0
    while pgrep ^dd; do sudo pkill -INFO dd; sleep 20; done
    kassnik · 2015-12-06 20:16:34 10
  • Due to @tremby, here: http://unix.stackexchange.com/a/172088/58343. I'm looking for a way to parallelize openssl and feed that to dd since openssl is the bottleneck on my machine: http://unix.stackexchange.com/questions/253466/parallelize-openssl-as-input-to-dd


    0
    openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | pv --progress --eta --rate --bytes --size 8000632782848 | dd of=/dev/md0 bs=2M
    diagon · 2016-01-05 19:36:03 20
  • 1. Sends an email to announce the start of the command 2. Raw data copy of block device over ssh to a new host 3. Sends an email to announce the end of the command Make sure the lvm device at the other side exists and has the same amount of logical units.


    0
    echo "dd start" | mail -s ready pete@example.com; dd if=/dev/system/machine-disk | ssh -c blowfish root@destination.example.com dd of=/dev/system/machine-disk ; echo "dd ready" | mail -s ready pete@example.com
    mipmip · 2016-03-01 13:13:20 12
  • Coreutils 8.24 added status=progress


    0
    dd if=foo of=bar status=progress
    ptman · 2016-04-15 08:43:40 12
  • first line is the speed the uncompressed data is read, second line is the compressed data sent over ssh. change sdb to your target drive/partition to be backed up. change pbzip -c1 to suit your compression. and ssh to your target file. don't forget to run zerofree/fstrim first! Show Sample Output


    0
    dd if=/dev/sdb | pv -rabc | pbzip2 -c1 | pv -rabc | ssh user@192.168.0.1 'cat > /dump.bz2'
    sexyrms · 2016-06-19 23:27:03 11
  • Sets the size of the disk to $DISKSIZE so that the percentage readout of pv is correct. set /dev/sdb to whatever your disk is /dev/sdX. Next pipe dd to pv, then pipe pv to gzip so that you get a gzipped image file. Show Sample Output


    0
    DISKSIZE=`sudo blockdev --getsize64 /dev/sdb` && sudo dd bs=4096 if=/dev/sdb | pv -s $DISKSIZE | sudo gzip -9 > ~/USBDRIVEBACKUP.img.gz
    frame45 · 2016-08-31 00:03:56 14
  • uses the wonderful 'pv' command to give a progress bar when copying one partition to another. Amazing for long running dd commands Show Sample Output


    0
    pv -tpreb /dev/sdc2 | dd of=/dev/sdb2 bs=64K conv=noerror,sync
    4fthawaiian · 2016-12-22 03:18:09 14
  • Recent versions of dd provides status flag. Setting that flag to progress shows progress in realtime.


    0
    dd status=progress if=infile of=outfile bs=512
    tankmohit11 · 2017-03-15 05:46:25 21

  • -1
    dd of=output.txt if=input.txt ibs=1 skip=$(expr `stat -c%s input.txt` / 2)
    kev · 2011-07-10 12:04:48 7
  • This example is taken from Cygwin running on Win7Ent-64. Device names will vary by platform. Both commands resulted in identical files per the output of md5sum, and ran in the same time down to the second (2m45s), less than 100ms apart. I timed the commands with 'time', which added before 'dd' or 'readom' gives execution times after the command completes. See 'man time' for more info...it can be found on any Unix or Linux newer than 1973. Yeah, that means everywhere. readom is supposed to guarantee good reads, and does support flags for bypassing bad blocks where dd will either fail or hang. readom's verbosity gave more interesting output than dd. On Cygwin, my attempt with 'readom' from the first answer actually ended up reading my hard drive. Both attempts got to 5GB before I killed them, seeing as that is past any CD or standard DVD. dd: 'bs=1M' says "read 1MB into RAM from source, then write that 1MB to output. I also tested 10MB, which shaved the time down to 2m42s. 'if=/dev/scd0' selects Cygwin's representation of the first CD-ROM drive. 'of=./filename.iso' simply means "create filename.iso in the current directory." readom: '-v' says "be a little noisy (verbose)." The man page implies more verbosity with more 'v's, e.g. -vvv. dev='D:' in Cygwin explicitly specifies the D-drive. I tried other entries, like '/dev/scd0' and '2,0', but both read from my hard drive instead of the CD-ROM. I imagine my LUN-foo (2,0) was off for my system, but on Cygwin 'D:' sort of "cut to the chase" and did the job. f='./filename.iso' specifies the output file. speed=2 simply sets the speed at which the CD is read. I also tried 4, which ran the exact same 2m45s. retries=8 simply means try reading a block up to 8 times before giving up. This is useful for damaged media (scratches, glue lines, etc.), allowing you to automatically "get everything that can be copied" so you at least have most of the data. Show Sample Output


    -1
    dd bs=1M if=/dev/scd0 of=./filename.iso OR readom -v dev='D:' f='./filename.iso' speed=2 retries=8
    scotharkins · 2013-10-23 15:53:27 6
  • For disk space constraint testing. Leaves a little space available for creating temp files, etc. Easily free up the used disk space again by deleting the dummy00 file. Can tailor the testing by building smaller 'blocks' to suit the needs of the testing. WARNING: do not do this to the '/' (root) filesystem unless you know what you are doing... on some systems it could crash the OS.


    -3
    dd if=/dev/zero of=/fs/to/fill/dummy00 bs=8192 count=$(df --block-size=8192 / | awk 'NR!=1 {print $4-100}')
    arcege · 2009-12-03 15:20:18 5

  • -4
    dd if=/dev/zero of=/tmp/bigfile bs=1024k count=100
    wincus · 2010-01-15 00:44:44 4
  • This is an useful command for when your OS is reporting less free RAM than it actually has. In case terminated processes did not free their variables correctly, the previously allocated RAM might make a bit sluggis over time. This command then creates a huge file made out of zeroes and then removes it, thus freeing the amount of memory occupied by the file in the RAM. In this example, the sequence will free up to 1GB(1M * 1K) of unused RAM. This will not free memory which is genuinely being used by active processes.


    -11
    dd if=/dev/zero of=junk bs=1M count=1K
    guedesav · 2009-11-01 23:45:51 6
  • If you want to delete lines fast then all you need to do is vi/vim a text file, type in the amount of lines you want to delete (in my example I wanted to delete 10056 lines) followed by dd (no spaces). There will be no output so becareful with what number you type.


    -14
    10056dd
    techie · 2013-04-10 10:34:39 10
  •  < 1 2 3

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

list block level layout

awk date convert
Convert readable date/time with `date` command

Alias TAIL for automatic smart output
Run the alias command, then issue $ps aux | tail and resize your terminal window (putty/console/hyperterm/xterm/etc) then issue the same command and you'll understand. $ ${LINES:-`tput lines 2>/dev/null||echo -n 12`} Insructs the shell that if LINES is not set or null to use the output from `tput lines` ( ncurses based terminal access ) to get the number of lines in your terminal. But furthermore, in case that doesn't work either, it will default to using the default of 80. The default for TAIL is to output the last 10 lines, this alias changes the default to output the last x lines instead, where x is the number of lines currently displayed on your terminal - 7. The -7 is there so that the top line displayed is the command you ran that used TAIL, ie the prompt. Depending on whether your PS1 and/or PROMPT_COMMAND output more than 1 line (mine is 3) you will want to increase from -2. So with my prompt being the following, I need -7, or - 5 if I only want to display the commandline at the top. ( http://www.askapache.com/linux/bash-power-prompt.html ) 275MB/748MB [7995:7993 - 0:186] 06:26:49 Thu Apr 08 [askapache@n1-backbone5:/dev/pts/0 +1] ~ $ In most shells the LINES variable is created automatically at login and updated when the terminal is resized (28 linux, 23/20 others for SIGWINCH) to contain the number of vertical lines that can fit in your terminal window. Because the alias doesn't hard-code the current LINES but relys on the $LINES variable, this is a dynamic alias that will always work on a tty device.

generate a unique and secure password for every website that you login to
usage: sitepass MaStErPaSsWoRd example.com description: An admittedly excessive amount of hashing, but this will give you a pretty secure password, It also eliminates repeated characters and deletes itself from your command history. tr '!-~' 'P-~!-O' # this bit is rot47, kinda like rot13 but more nerdy rev # this avoids the first few bytes of gzip payload, and the magic bytes.

Route outbound SMTP connections through a addtional IP address rather than your primary

View all images
So you are in directory with loads of pictures laying around and you need to quickly scan through them all

Renaming a file without overwiting an existing file name
Sometimes in a hurry you may move or copy a file using an already existent file name. If you aliased the cp and mv command with the -i option you are prompted for a confirmation before overwriting but if your aliases aren't there you will loose the target file! The -b option will force the mv command to check if the destination file already exists and if it is already there a backup copy with an ending ~ is created.

Extract the MBR ID of a device
Useful when you want to know the mbrid of a device - for the purpose of making it bootable. Certain hybridiso distros, for eg the OpenSUSE live ISO uses the mbrid to find the live media. Use this command to find out the mbrid of your USB drive and then edit the /grub/mbrid file to match it.

Convert CSV to JSON
Replace 'csv_file.csv' with your filename.

Find files that were modified by a given command
Traces the system calls of a program. See http://linuxhelp.blogspot.com/2006/05/strace-very-powerful-troubleshooting.html for more information.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: