Commands tagged tar (84)

  • Tar's up $DIR locally (w/bzip2) and sends remotely to $HOST:$PORT where netcat listens (using openbsd netcat). Start up receiving side command first, then execute this.


    0
    tar -cjf - $DIR | nc $HOST $PORT
    taintedkernel · 2012-11-13 16:44:26 5
  • Receives bzip'd tar archive via netcat (openbsd nc) and stores locally. Displays size with pv. Start this receiver first, then the sender.


    0
    nc -l $PORT | pv -b > archive.tar.bz2
    taintedkernel · 2012-11-13 16:47:45 4
  • Decrypt with: gpg -o- foo.tgz.gpg | tar zxvf -


    0
    tar zcf - foo | gpg -c --cipher-algo aes256 -o foo.tgz.gpg
    skkzsh · 2013-03-13 09:44:39 5
  • is preserving creation time, modification time, permission, the directory structure, etc. Show Sample Output


    0
    XZ_OPT=-9 tar cJf tarfile.tar.xz directory
    unixmonkey51864 · 2013-03-30 06:00:39 7
  • Do above at the Destination aka The Server. Do the following at the Source aka The Client: tar -cf - /srcfolder | pv | nc www.home.com 50002 If you want ETAs and stuff: tar -cf - /srcfolder | pv -s `du -sb /srcfolder | awk '{print $1}'` | nc www.home.com 50002 If you dont care about progress bars @ server/destination: tar -cf - /srcfolder | pv | nc www.home.com 50002 If you dont care about progress bars @ client/source: tar -cf - /srcfolder | pv -s `du -sb /srcfolder | awk '{print $1}'` | nc www.home.com 50002 I have this in alot better detail where there is more room to talk about it on my site: http://www.kossboss.com/linuxtarpvncssh Show Sample Output


    0
    while true; do nc -l -p 50002 | pv | tar -xf -; done
    bhbmaster · 2013-05-30 07:17:23 10
  • NOTE: When doing these commands when asked for questions there might be flowing text from the pv doing the progress bar just continue typing as if its not there, close your eyes if it helps, there might be a yes or no question, type "yes" and ENTER to it, and also it will ask for a password, just put in your password and ENTER I talk alot more about this and alot of other variations of this command on my site: http://www.kossboss.com/linuxtarpvncssh Show Sample Output


    0
    cd /srcfolder; tar -czf - . | pv -s `du -sb . | awk '{print $1}'` | ssh -c arcfour,blowfish-cbc -p 50005 root@destination.com "tar -xzvf - -C /dstfolder"
    bhbmaster · 2013-05-30 07:21:06 7
  • Back up /etc directory with a name based on the current date and the hostname of the machine, then chown the file for the current user for use.


    0
    sudo tar -zcvf $(hostname)-etc-back-`date +%d`-`date +%m`-`date +%y`.tar.gz /etc && sudo chown $USER:$USER $(hostname)-etc-back*
    thanosme · 2013-08-18 12:49:11 19
  • The files are automatically uncompressed when they reach the destination machine. This is a fast way to backup your server to your local computer while it's running (shutting down services is recommended). A file named "exclude.txt" is needed at /tmp/ containing the following : /dev/* /media/* /mnt/* /proc/* /sys/* /tmp/* /home/*/.local/share/Trash /home/*/.gvfs /home/*/.cache /home/*/.thumbnails /etc/fstab /lib/modules/*/volatile/.mounted /var/run/* /var/lock/* /var/tmp/* /var/cache/apt/archives/* /lost+found/* Show Sample Output


    0
    tar -cj / -X /tmp/exclude.txt | cstream -v 1 -c 3 -T 10 | ssh user@host 'tar -xj -C /backupDestination'
    fantleas · 2014-07-21 18:52:19 9
  • This will write to TAPE (LTO3-4 in my case) a backup of files/folders. Could be changed to write to DVD/Blueray. Go to the directory where you want to write the output files : cd /bklogs Enter a name in bkname="Backup1", enter folders/files in tobk="/home /var/www". It will create a tar and write it to the tape drive on /dev/nst0. In the process, it will 1) generate a sha512 sum of the tar to $bkname.sha512; so you can validate that your data is intact 2) generate a filelist of the content of the tar with filesize to $bkname.lst 3) buffer the tar file to prevent shoe-shining the tape (I use 4GB for lto3(80mb/sec), 8gb for lto4 (120mb/sec), 3Tb usb3 disks support those speed, else I use 3x2tb raidz. 4) show buffer in/out speed and used space in the buffer 5) show progress bar with time approximation using pv ADD : To eject the tape : ; sleep 75; mt-st -f /dev/nst0 rewoffl TODO: 1) When using old tapes, if the buffer is full and the drive slows down, it means the tape is old and would need to be replaced instead of wiping it and recycling it for an other backup. Logging where and when it slows down could provide good information on the wear of the tape. I don't know how to get that information from the mbuffer output and to trigger a "This tape slowed down X times at Y1gb, Y2gb, Y3gb down to Zmb/s for a total of 30sec. It would be wise to replace this tape next time you want to write to it." 2) Fix filesize approximation 3) Save all the output to $bkname.log with progress update being new lines. (any one have an idea?) 4) Support spanning on multiple tape. 5) Replace tar format with something else (dar?); looking at xar right now (https://code.google.com/p/xar/), xml metadata could contain per file checksum, compression algorithm (bzip2, xv, gzip), gnupg encryption, thumbnail, videopreview, image EXIF... But that's an other project. TIP: 1) You can specify the width of the progressbar of pv. If its longer than the terminal, line refresh will be written to new lines. That way you can see if there was speed slowdown during writing. 2) Remove the v in tar argument cvf to prevent listing all files added to the archive. 3) You can get tarsum (http://www.guyrutenberg.com/2009/04/29/tarsum-02-a-read-only-version-of-tarsum/) and add >(tarsum --checksum sha256 > $bkname_list.sha256) after the tee to generate checksums of individual files !


    0
    bkname="test"; tobk="*" ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee >(sha512sum > $bkname.sha512) >(tar -tv > $bkname.lst) | mbuffer -m 4G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k
    johnr · 2014-07-22 15:47:50 8
  • Adds high-performance, lightweight lz4 compression to speed the transfer of files over a trusted network link. Using (insecure) netcat results in a much faster transfer than using a ssh tunnel because of the lack of overhead. Also, LZ4 is as fast or faster than LZ0, much faster than gzip or LZMA, an in a worst-case scenario, incompressible data gets increased by 0.4% in size. Using LZMA or gzip compressors makes more sense in cases where the network link is the bottleneck, whereas LZ4 makes more sense if CPU time is more of a bottleneck.


    0
    On target: "nc -l 4000 | lz4c -d - | tar xvf -" On source: "tar -cf - . | lz4c | nc target_ip 4000"
    baitisj · 2014-08-02 05:09:30 7
  • extract () { if [ -f $1 ] ; then case $1 in *.tar.bz2) tar xvjf $1 ;; *.tar.gz) tar xvzf $1 ;; *.tar.xz) tar Jxvf $1 ;; *.bz2) bunzip2 $1 ;; *.rar) unrar x $1 ;; *.gz) gunzip $1 ;; *.tar) tar xvf $1 ;; *.tbz2) tar xvjf $1 ;; *.tgz) tar xvzf $1 ;; *.zip) unzip $1 ;; *.Z) uncompress $1 ;; *.7z) 7z x $1 ;; *) echo "don't know how to extract '$1'..." ;; esac read -r -p "Delete the compressed file? [Y/N] " response response=${response,,} # tolower if [[ $response =~ ^([Yy]es|YES|[Yy])$ ]]; then echo "rm '$1'" rm $1 fi else echo "'$1' is not a valid file!" fi }


    0
    extract file.tar.gz
    asdzxc · 2015-02-08 11:17:12 9
  • compress(){ # compress [FIle/Folder] [NewFileName].[Suffix] # compress image.jpg pictures.tar.bz2 # compress Document/ folder.rar if [ -f $1 ] || [ -d $1 ]; then case $2 in *.tar.bz2) tar -jcvf $2 $1 ;; *.tar.gz) tar -zcvpf $2 $1 ;; *.tar) tar -cvpf $2 $1 ;; *.zip) zip -r $2 $1 ;; *.rar) rar a -r -rr10 $2 $1 ;; *) echo "don't know how to compres '$1'..." ;; esac else echo "'$1' is not a valid file or folder2" fi }


    0
    compress pictures/ pictures.tar.gz
    asdzxc · 2015-02-08 11:28:30 13
  • This command tars/gz all the folders contained in a directory. Only applies to top level directory. Very handy when you have to transfer mny folders containing lots of stuff. Can also work with tar only, zip...


    0
    find . -type d -maxdepth 1 -mindepth 1 -exec tar czf {}.tar.gz {} \;
    deadbird · 2015-05-25 13:03:05 11
  • The function had to be cut down to meet the maximum command length requirements. The full version of the function is: extract() { if [ -f $1 ]; then case $1 in *.tar.bz2) tar xvjf $1 ;; *.tar.gz) tar xvzf $1 ;; *.bz2) bunzip2 $1 ;; *.rar) unrar x $1 ;; *.gz) gunzip $1 ;; *.tar) tar xvf $1 ;; *.tbz2) tar xvjf $1 ;; *.tgz) tar xvzf $1 ;; *.zip) unzip $1 ;; *.Z) uncompress $1 ;; *.7z) 7z x $1 ;; *) echo "'$1' cannot be extracted via >extract<" ;; esac else echo "'$1' is not a valid file!" fi } Note: This is not my original code. I came across it in a forum somewhere a while ago, and it's been such a useful addition to my .bashrc file, that I thought it worth sharing. Show Sample Output


    0
    extract() {if [ -f $1 ];then case $1 in *.tar.bz2) tar xvjf $1;;*.tar.gz) tar xvzf $1;;*.bz2) bunzip2 $1;;*.rar) unrar x $1;;*.gz) gunzip $1;;*.tar) tar xvf $1;;*.tbz2) tar xvjf $1;;*.tgz) tar xvzf $1;;*.zip) unzip $1;;*.7z) 7z x $1;;esac fi}
    SamuelWN · 2016-02-08 14:52:05 12
  • untar in place with out creating a temporary file


    0
    ssh user@host "tar -zcf - /path/to/dir" | tar -xvz
    sandeep048 · 2017-10-07 11:37:51 18
  • `tar xfzO` extracts to STDOUT which got redirected directly to mysql. Really helpful, when your hard drive can't fit two copies of non-compressed database :)


    -1
    tar xfzO <backup_name>.tar.gz | mysql -u root <database_name>
    alecnmk · 2011-02-10 22:18:42 5

  • -1
    cp -av source dest
    Vilemirth · 2011-02-19 18:43:49 3
  • Sometimes you might need to have two copies of data that is in tar. You might unpack, and then copy, but if IO is slow, you might lower it by automatically writing it twice (or more times)


    -1
    mkdir copy{1,2}; gzip -dc file.tar.gz | tee >( tar x -C copy1/ ) | tar x -C copy2/
    depesz · 2011-04-14 17:02:05 5
  • Simple Compressed Backup of the /etc Linux compatible


    -1
    tar jcpf /home/[usuario]/etc-$(hostname)-backup-$(date +%Y%m%d-%H%M%S).tar.bz2 /etc
    mack · 2011-04-29 22:53:11 4
  • Similar, but uses tarball instead of zip file


    -1
    git archive HEAD | gzip > ~/Dropbox/archive.tar.gz
    tamouse · 2011-06-08 09:44:07 3
  • backup your files in tar archive + timestamp of backup Show Sample Output


    -1
    tar -cvf bind9-config-`date +%s`.tar *
    Fuonum · 2014-10-29 05:15:15 9
  • Create a single tar.gz archive I know it's a very basic one, but it's one I keep forgetting. Show Sample Output


    -2
    tar -pczf archive_name.tar.gz /path/to/dir/or/file
    ryuslash · 2009-07-17 19:53:02 30
  • This script will list all the files in the tarballs present on any folder or subfolder of the provided path. The while loop is for echoing the file name of the tarball before listing the files, so the tarball can be identified


    -2
    find <path> -name "*.tgz" -or -name "*.tar.gz" | while read file; do echo "$file: "; tar -tzf $file; done
    polaco · 2009-11-10 20:39:04 36
  • The magic is performed by the parameter -t Show Sample Output


    -2
    for F in $(find ./ -name "*.tgz") ; do tar -tvzf $F ; done
    alchandia · 2009-11-11 00:50:52 3
  • This may seem like a long command, but it is great for making sure all file permissions are kept in tact. What it is doing is streaming the files in a sub-shell and then untarring them in the target directory. Please note that the -z command should not be used for local files and no perfomance increase will be visible as overhead processing (CPU) will be evident, and will slow down the copy. You also may keep simple with, but you don't have the progress info: cp -rpf /some/directory /other/path Show Sample Output


    -2
    dir='path to file'; tar cpf - "$dir" | pv -s $(du -sb "$dir" | awk '{print $1}') | tar xpf - -C /other/path
    starchox · 2010-01-19 19:05:45 3
  •  < 1 2 3 4 > 

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Rename files in batch

Download an Entire website with wget

Run a program transparently, but print a stack trace if it fails
For automated unit tests I wanted my program to run normally, but if it crashed, to add a stack trace to the output log. I came up with this command so I wouldn't have to mess around with core files. The one downside is that it does smoosh your program's stderr and stdout together.

Get AWS temporary credentials ready to export based on a MFA virtual appliance
You might want to secure your AWS operations requiring to use a MFA token. But then to use API or tools, you need to pass credentials generated with a MFA token. This commands asks you for the MFA code and retrieves these credentials using AWS Cli. To print the exports, you can use: `awk '{ print "export AWS_ACCESS_KEY_ID=\"" $1 "\"\n" "export AWS_SECRET_ACCESS_KEY=\"" $2 "\"\n" "export AWS_SESSION_TOKEN=\"" $3 "\"" }'` You must adapt the command line to include: * $MFA_IDis ARN of the virtual MFA or serial number of the physical one * TTL for the credentials

a short counter
Maybe you know shorter ?

Log colorizer for OSX (ccze alternative)
Download colorizer by @raszi @ http://github.com/raszi/colorize

Hold off any screensavers/timeouts
Moves the mouse 1 pixel down and to the right, then immediately back again, every 4 minutes. This keeps screensavers from turning on. I have used this extensively and I've never even noticed the mouse movement because it is so subtle.

back ssh from firewalled hosts
host B (you) redirects a modem port (62220) to his local ssh. host A is a remote machine (the ones that issues the ssh cmd). once connected port 5497 is in listening mode on host B. host B just do a ssh 127.0.0.1 -p 5497 -l user and reaches the remote host'ssh. This can be used also for vnc and so on.

Get your outgoing IP address


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: