Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.


If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

Hide

News

2011-03-12 - Confoo 2011 presentation
Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands
To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner
Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed
Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.
Hide

Tags

Hide

Functions

Commands tagged tar from sorted by
Terminal - Commands tagged tar - 75 results
On target: "nc -l 4000 | lz4c -d - | tar xvf -" On source: "tar -cf - . | lz4c | nc target_ip 4000"
2014-08-02 05:09:30
User: baitisj
Tags: tar nc lz4
0

Adds high-performance, lightweight lz4 compression to speed the transfer of files over a trusted network link. Using (insecure) netcat results in a much faster transfer than using a ssh tunnel because of the lack of overhead. Also, LZ4 is as fast or faster than LZ0, much faster than gzip or LZMA, an in a worst-case scenario, incompressible data gets increased by 0.4% in size. Using LZMA or gzip compressors makes more sense in cases where the network link is the bottleneck, whereas LZ4 makes more sense if CPU time is more of a bottleneck.

bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
2014-07-22 15:47:50
User: johnr
Functions: cut dd du tail tar tee
0

This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray.

Go to the directory where you want to write the output files : cd /bklogs

Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www".

It will create a tar and write it to the tape drive on /dev/nst0.

In the process, it will

1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact

2) generate a filelist of the content of the tar with filesize to $bkname.lst

3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz.

4) show buffer in/out speed and used space in the buffer

5) show progress bar with time approximation using pv

ADD :

To eject the tape :

; sleep 75; mt-st -f /dev/nst0 rewoffl

TODO:

1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it."

2) Fix filesize approximation

3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?)

4) Support spanning on multiple tape.

5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project.

TIP:

1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing.

2) Remove the v in tar argument cvf to prevent listing all files added to the archive.

3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/)

and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !

tar -cj / -X /tmp/exclude.txt | cstream -v 1 -c 3 -T 10 | ssh user@host 'tar -xj -C /backupDestination'
2014-07-21 18:52:19
User: fantleas
Functions: ssh tar
0

The files are automatically uncompressed when they reach the destination machine. This is a fast way to backup your server to your local computer while it's running (shutting down services is recommended).

A file named "exclude.txt" is needed at /tmp/ containing the following :

/dev/*

/media/*

/mnt/*

/proc/*

/sys/*

/tmp/*

/home/*/.local/share/Trash

/home/*/.gvfs

/home/*/.cache

/home/*/.thumbnails

/etc/fstab

/lib/modules/*/volatile/.mounted

/var/run/*

/var/lock/*

/var/tmp/*

/var/cache/apt/archives/*

/lost+found/*

tar -xvpf file.tar.gz
2014-04-25 10:23:03
User: shajeen
Functions: tar
1

-x, --extract, --get

extract files from an archive

-p, --preserve-permissions, --same-permissions

extract information about file permissions (default for superuser)

-f, --file=ARCHIVE

use archive file or device ARCHIVE

-v, --verbose

verbosely list files processed

tar -cf - file1 dir1/ dir2/ | md5sum
2014-04-17 14:33:44
User: snipertyler
Functions: tar
-3

Doesn't create a file

Make sure to list the files / directories in the same order every time.

tar -cvf - /path/to/tar/up | xz - > myTarArchive.tar.xz
2014-03-18 19:51:50
User: razerwolf
Functions: tar
1

compress directory archive with xz compression, if tar doesn't have the -J option (OSX tar doesn't have -J)

tar -cJf myarchive.tar.xz /path/to/archive/
2014-03-13 03:34:18
User: Sepero
Functions: tar
1

Compress files or a directory to xz format. XZ has superior and faster compression than bzip2 in most cases. XZ is superior to 7zip format because it can save file permissions and other metadata data.

tar --exclude='patternToExclude' --use-compress-program=pbzip2 -cf 'my-archive.tar.bz2' directoyToZip/
tar -axf fileNameHere.tgz
2014-02-01 16:14:22
User: toro
Functions: tar
Tags: tar
1

With -a you don't care about file type (bz2, gzip, etc.)

tar zxvf fileNameHere.tgz
2014-01-28 10:33:51
User: Jonas_E
Functions: tar
Tags: tar unpack
-2

tar command options:

-z : Uncompress the resulting archive with gzip command.

-x : Extract to disk from the archive.

-v : Produce verbose output i.e. show progress and file names while extracting files.

-f backup.tgz : Read the archive from the specified file called backup.tgz.

-C /tmp/data : Unpack/extract files in /tmp/data instead of the default current directory.

sudo tar -zcvf $(hostname)-etc-back-`date +%d`-`date +%m`-`date +%y`.tar.gz /etc && sudo chown $USER:$USER $(hostname)-etc-back*
2013-08-18 12:49:11
User: thanosme
Functions: chown sudo tar
Tags: backup tar date
0

Back up /etc directory with a name based on the current date and the hostname of the machine, then chown the file for the current user for use.

cd /srcfolder; tar -czf - . | pv -s `du -sb . | awk '{print $1}'` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com "tar -xzvf - -C /dstfolder"
2013-05-30 07:21:06
User: bhbmaster
Functions: awk cd ssh tar
Tags: ssh tar pv
0

NOTE: When doing these commands when asked for questions there might be flowing text from the pv doing the progress bar just continue typing as if its not there, close your eyes if it helps, there might be a yes or no question, type "yes" and ENTER to it, and also it will ask for a password, just put in your password and ENTER

I talk alot more about this and alot of other variations of this command on my site:

http://www.kossboss.com/linuxtarpvncssh

while true; do nc -l -p 50002 | pv | tar -xf -; done
2013-05-30 07:17:23
User: bhbmaster
Functions: tar
Tags: tar nc pv
0

Do above at the Destination aka The Server.

Do the following at the Source aka The Client:

tar -cf - /srcfolder | pv | nc www.home.com 50002

If you want ETAs and stuff:

tar -cf - /srcfolder | pv -s `du -sb /srcfolder | awk '{print $1}'` | nc www.home.com 50002

If you dont care about progress bars @ server/destination:

tar -cf - /srcfolder | pv | nc www.home.com 50002

If you dont care about progress bars @ client/source:

tar -cf - /srcfolder | pv -s `du -sb /srcfolder | awk '{print $1}'` | nc www.home.com 50002

I have this in alot better detail where there is more room to talk about it on my site:

http://www.kossboss.com/linuxtarpvncssh

XZ_OPT=-9 tar cJf tarfile.tar.xz directory
2013-03-30 06:00:39
Functions: tar
0

is preserving creation time, modification time, permission, the directory structure, etc.

tar zcf - foo | gpg -c --cipher-algo aes256 -o foo.tgz.gpg
2013-03-13 09:44:39
User: skkzsh
Functions: gpg tar
0

Decrypt with:

gpg -o- foo.tgz.gpg | tar zxvf -
tar --create --file - --posix --gzip -- <dir> | openssl enc -e -aes256 -out <file>
2012-11-27 15:33:45
User: seb1245
Functions: tar
19

Create an AES256 encrypted and compressed tar archive.

User is prompted to enter the password.

Decrypt with:

openssl enc -d -aes256 -in <file> | tar --extract --file - --gzip
nc -l $PORT | pv -b > archive.tar.bz2
2012-11-13 16:47:45
Tags: netcat tar pv
0

Receives bzip'd tar archive via netcat (openbsd nc) and stores locally. Displays size with pv. Start this receiver first, then the sender.

tar -cjf - $DIR | nc $HOST $PORT
2012-11-13 16:44:26
Functions: tar
Tags: netcat tar
0

Tar's up $DIR locally (w/bzip2) and sends remotely to $HOST:$PORT where netcat listens (using openbsd netcat). Start up receiving side command first, then execute this.

tar --lzma -xvf /path/to/archive
tar -zcvf file.tgz ./
tar cvzf - /folder/ | ssh root@192.168.0.1 "dd of=/dest/folder/file.tar.gz"
2012-07-13 17:54:51
User: kruspemsv
Functions: ssh tar
1

You can ran this also with cat for example:

tar zcvf - /folder/ | ssh root@192.168.0.1 "cat > /dest/folder/file.tar.gz"

Or even run other command's:

tcpdump | ssh root@10.0.0.1 "cat > /tmp/tcpdump.log"
als some.jar
aunpack foo.tar.bz2
GZIP="--rsyncable" tar -czf something.tgz /something
vim some-archive.tar.gz
2012-04-20 02:37:28
User: ktonga
Functions: vim
5

If you vim a compressed file it will list all archive content, then you can pickup any of them for editing and saving. There you have the modified archive without any extra step. It supports many file types such as tar.gz, tgz, zip, etc.