Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.


If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

Hide

News

2011-03-12 - Confoo 2011 presentation
Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands
To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner
Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed
Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.
Hide

Tags

Hide

Functions

Commands tagged dd from sorted by
Terminal - Commands tagged dd - 61 results
bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
2014-07-22 15:47:50
User: johnr
Functions: cut dd du tail tar tee
0

This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray.

Go to the directory where you want to write the output files : cd /bklogs

Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www".

It will create a tar and write it to the tape drive on /dev/nst0.

In the process, it will

1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact

2) generate a filelist of the content of the tar with filesize to $bkname.lst

3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz.

4) show buffer in/out speed and used space in the buffer

5) show progress bar with time approximation using pv

ADD :

To eject the tape :

; sleep 75; mt-st -f /dev/nst0 rewoffl

TODO:

1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it."

2) Fix filesize approximation

3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?)

4) Support spanning on multiple tape.

5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project.

TIP:

1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing.

2) Remove the v in tar argument cvf to prevent listing all files added to the archive.

3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/)

and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !

ctrl-t
while pgrep ^dd; do pkill -INFO dd; sleep 10; done
2014-01-28 03:09:25
User: sternebrau
Functions: sleep
Tags: dd progress
0

While a dd is running in one terminal, open another and enter the while loop. The sample output will be displayed in the window running the dd and the while loop will exit when the dd is complete. It's possible that a "sudo" will need to be inserted before "pkill", depending on your setup, for example:

while pgrep ^dd; do sudo pkill -INFO dd; sleep 10; done
dd bs=1M if=/dev/scd0 of=./filename.iso OR readom -v dev='D:' f='./filename.iso' speed=2 retries=8
2013-10-23 15:53:27
User: scotharkins
Functions: dd
-1

This example is taken from Cygwin running on Win7Ent-64. Device names will vary by platform.

Both commands resulted in identical files per the output of md5sum, and ran in the same time down to the second (2m45s), less than 100ms apart. I timed the commands with 'time', which added before 'dd' or 'readom' gives execution times after the command completes. See 'man time' for more info...it can be found on any Unix or Linux newer than 1973. Yeah, that means everywhere.

readom is supposed to guarantee good reads, and does support flags for bypassing bad blocks where dd will either fail or hang.

readom's verbosity gave more interesting output than dd.

On Cygwin, my attempt with 'readom' from the first answer actually ended up reading my hard drive. Both attempts got to 5GB before I killed them, seeing as that is past any CD or standard DVD.

dd:

'bs=1M' says "read 1MB into RAM from source, then write that 1MB to output. I also tested 10MB, which shaved the time down to 2m42s.

'if=/dev/scd0' selects Cygwin's representation of the first CD-ROM drive.

'of=./filename.iso' simply means "create filename.iso in the current directory."

readom:

'-v' says "be a little noisy (verbose)." The man page implies more verbosity with more 'v's, e.g. -vvv.

dev='D:' in Cygwin explicitly specifies the D-drive. I tried other entries, like '/dev/scd0' and '2,0', but both read from my hard drive instead of the CD-ROM. I imagine my LUN-foo (2,0) was off for my system, but on Cygwin 'D:' sort of "cut to the chase" and did the job.

f='./filename.iso' specifies the output file.

speed=2 simply sets the speed at which the CD is read. I also tried 4, which ran the exact same 2m45s.

retries=8 simply means try reading a block up to 8 times before giving up. This is useful for damaged media (scratches, glue lines, etc.), allowing you to automatically "get everything that can be copied" so you at least have most of the data.

pv -tpreb /dev/sda | dd of=/dev/sdb bs=1M
2013-08-19 23:04:15
User: sc0ttyd
Functions: dd
Tags: dd progress pv,
0

Your platform may not have pv by default. If you are using Homebew on OSX, simply 'brew install pv'.

dd if=/dev/cdrom bs=1 skip=32808 count=32 conv=unblock cbs=32 2>/dev/null
2013-04-14 20:50:58
User: mlk
Functions: dd
Tags: dd cd iso9660
0

The comp.unix.shell posting by St?phane Chazelas also lists the following offsets:

type 32768 (1 byte)

id 32769 (5 bytes)

version 32774 (1 byte)

system_id 32776 (32 bytes)

volume_id 32808 (32 bytes)

volume_space_size 32848 (8 bytes)

escape_sequences 32856 (32 bytes)

volume_set_size 32888 (4 bytes)

volume_sequence_number 32892 (4 bytes)

logical_block_size 32896 (4 bytes)

path_table_size 32900 (8 bytes)

type_l_path_table 32908 (4 bytes)

opt_type_l_path_table 32912 (4 bytes)

type_m_path_table 32916 (4 bytes)

opt_type_m_path_table 32920 (4 bytes)

root_directory_record 32924 (34 bytes)

volume_set_id 32958 (128 bytes)

publisher_id 33086 (128 bytes)

preparer_id 33214 (128 bytes)

application_id 33342 (128 bytes)

copyright_file_id 33470 (37 bytes)

abstract_file_id 33507 (37 bytes)

bibliographic_file_id 33544 (37 bytes)

creation_date 33581 (17 bytes)

modification_date 33598 (17 bytes)

expiration_date 33615 (17 bytes)

effective_date 33632 (17 bytes)

file_structure_version 33649 (1 byte)

application_data 33651 (512 bytes)

10056dd
2013-04-10 10:34:39
User: techie
Tags: dd
-14

If you want to delete lines fast then all you need to do is vi/vim a text file, type in the amount of lines you want to delete (in my example I wanted to delete 10056 lines) followed by dd (no spaces). There will be no output so becareful with what number you type.

CTRL + T
2012-12-19 02:21:41
Tags: dd progress
16

Sends SIGINFO to the process. This is a BSD feature OS X inherited. You must have the terminal window executing dd selected when entering CTRL + T for this to work.

watch -n 1 pkill -USR1 "^dd$"
2012-08-31 05:15:45
Functions: watch
Tags: dd progress speed
0

Sends the "USR1" signal every 1 second (-n 1) to a process called exactly "dd".

The signal in some systems can be INFO or SIGINFO ...

look at the signals list in: man kill

dd if=/dev/cdrom of=~/cdrom_image.iso
2012-07-10 06:03:25
User: o0110o
Functions: dd
Tags: dd cd iso dvd convert
5

An easy method to generate ISOs from CD/DVD media.

kill -SIGUSR1 xxxx
2012-04-12 09:32:24
User: netaxiz
Functions: kill
Tags: dd pv
1

run this in another terminal, were xxxx is the process ID of the running dd process.

the progress will report on the original terminal that you ran dd on

pv -tpreb /dev/urandom | dd of=file.img
2012-04-11 22:32:52
User: marrowsuck
Functions: dd
Tags: dd pv
9

This version was mentioned in the comments. Credits go to flatcap.

dd if=/dev/cdrom of=~/cdimage.iso
x=1024; y=32768; cat <(echo -e "P5\n$x $y\n255\n") <(dd if=/dev/sda1 bs=$x count=$y) > sda1.pgm
2012-03-06 03:09:16
User: adeverteuil
Functions: cat dd echo
Tags: dd images pnm pgm
1

Keep width to a power of 2 to see patterns emerge. 512 is good. So is 4096 for huge maps.

PNM headers are super basic.

http://netpbm.sourceforge.net/doc/pbm.html

SIZE=`fdisk -s /dev/sdx`; dd if=/dev/sdx bs=1M | pv -s "$SIZE"k > hdd.img
dd if=/dev/urandom of=file.img bs=4KB& sleep 1 && pid=`pidof dd`; while [[ -d /proc/$pid ]]; do kill -USR1 $pid && sleep 10 && clear; done
2012-02-23 01:45:53
Functions: dd kill sleep
1

The previously-posted one-liner didn't work for me for whatever reason, so I ended up doing this instead.

pv -petrs $(stat -c %s file.iso) file.iso | dd bs=1M oflag=sync of=/dev/sdX
time (pv file.iso | dd bs=1M oflag=sync of=/dev/sdX 2>/dev/null)
dd if=<device> | pv | nc <target> <port>
2012-01-27 18:37:36
Functions: dd
Tags: dd nc pv 7z
8

Create an image of "device" and send it to another machine through the network ("target" and "port" sets the ip and port the stream will be sent to), outputting a progress bar

On the machine that will receive, compress and store the file, use:

nc -l -p <port> | 7z a <filename> -si -m0=lzma2 -mx=9 -ms=on

Optionally, add the -v4g switch at the end of the line in order to split the file every 4 gigabytes (or set another size: accepted suffixes are k, m and g).

The file will be compressed using 7z format, lzma2 algorithm, with maximum compression level and solid file activated.

The compression stage will be executed on the machine which will store the image. It was planned this way because the processor on that machine was faster, and being on a gigabit network, transfering the uncompressed image wasn't much of a problem.

dd of=output.txt if=input.txt ibs=1 skip=$(expr `stat -c%s input.txt` / 2)
dd if=/dev/urandom of=file.img bs=4KB& pid=$!; while [[ -d /proc/$pid ]]; do kill -USR1 $pid && sleep 1 && clear; done
2011-06-24 21:49:10
Functions: dd kill sleep
Tags: dd progress
2

Only slightly different than previous commands. The benefit is that your "watch" should die when the dd command has completed. (Of course this would depend on /proc being available)

mount -o loop,offset=$((512*x)) /path/to/dd/image /mount/path
2011-06-14 19:30:54
User: 0disse0
Functions: mount
Tags: dd mount fdisk
0

Assuming we have a disk image, created by dd if=/dev/sda of=image.dd we can check the image's partition layout with fdisk -ul image.dd, then substitute "x" with starting sector of the partition we want to mount. This example assumes that the disk uses 512Byte sectors

ghc -e "mapM_ (\_->Data.ByteString.Char8.putStr (Data.ByteString.Char8.replicate (1024*1024) '\\255')) [1..24]"
2011-04-05 14:28:54
User: MaskRay
Tags: dd tr
2

I'm both a one-liner fan and a haskell learner

tr '\0' '\377' < /dev/zero|dd count=$((<bytes>/512))
2011-04-05 14:26:02
User: cfy
Functions: dd tr
Tags: dd tr
4

the speed is about 500MB/s on my machine.

i think it's fast enough to output not too many bytes.

while a C program may output 1GB per sencond on my machine.

if the size is not the power of 512,you may change the bs and count in dd.