Commands using rsync (88)

  • It can resume a failed secure copy ( usefull when you transfer big files like db dumps through vpn ) using rsync. It requires rsync installed in both hosts. rsync --partial --progress --rsh=ssh $file_source $user@$host:$destination_file local -> remote or rsync --partial --progress --rsh=ssh $user@$host:$remote_file $destination_file remote -> local


    23
    rsync --partial --progress --rsh=ssh $file_source $user@$host:$destination_file
    dr_gogeta86 · 2009-04-01 13:13:14 10
  • If your user has sudo on the remote box, you can rsync data as root without needing to login as root. This is very helpful if the remote box does not allow root to login over SSH (which is a common security restriction).


    18
    rsync --rsync-path 'sudo rsync' username@source:/folder/ /local/
    Alioth · 2009-03-25 21:18:55 11
  • The command copies a file from remote SSH host on port 8322 with bandwidth limit 100KB/sec; --progress shows a progress bar --partial turns partial download on; thus, you can resume the process if something goes wrong --bwlimit limits bandwidth by specified KB/sec --ipv4 selects IPv4 as preferred I find it useful to create the following alias: alias myscp='rsync --progress --partial --rsh="ssh -p 8322" --bwlimit=100 --ipv4' in ~/.bash_aliases, ~/.bash_profile, ~/.bash_login or ~/.bashrc where appropriate. Show Sample Output


    17
    rsync --progress --partial --rsh="ssh -p 8322" --bwlimit=100 --ipv4 user@domain.com:~/file.tgz .
    ruslan · 2011-02-10 14:25:22 9

  • 15
    rsync --progress file1 file2
    fletch · 2010-02-25 19:57:01 15
  • Put it into your sh startup script (I use alias scpresume='rsync --partial --progress --rsh=ssh' in bash). When a file transfer via scp has aborted, just use scpresume instead of scp and rsync will copy only the parts of the file that haven't yet been transmitted. Show Sample Output


    14
    rsync --partial --progress --rsh=ssh SOURCE DESTINATION
    episodeiv · 2009-02-16 16:22:10 653
  • copy files to a ssh server with gzip compression


    14
    rsync -az /home/user/test user@sshServer:/tmp/
    peshay · 2009-08-25 10:45:15 6
  • This will backup the _contents_ of /media/SOURCE to /media/TARGET where TARGET is formatted with ntfs. The --modify-window lets rsync ignore the less accurate timestamps of NTFS.


    13
    rsync -rtvu --modify-window=1 --progress /media/SOURCE/ /media/TARGET/
    0x2142 · 2009-07-05 07:40:10 11
  • This creates an archive that does the following: rsync:: (Everyone seems to like -z, but it is much slower for me) -a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files. -H: preserves hard-links -A: preserves ACLs -X: preserves extended attributes -x: don't cross file-system boundaries -v: increase verbosity --numeric-ds: don't map uid/gid values by user/group name --delete: delete extraneous files from dest dirs (differential clean-up during sync) --progress: show progress during transfer ssh:: -T: turn off pseudo-tty to decrease cpu load on destination. -c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination. -o Compression=no: Turn off SSH compression. -x: turn off X forwarding if it is on by default. Flip: rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" [source_dir] [dest_host:/dest_dir]


    12
    rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>
    somaddict · 2012-12-26 13:46:23 7
  • use find with rsync


    11
    find . -name "whatever.*" -print0 | rsync -av --files-from=- --from0 ./ ./destination/
    grokskookum · 2009-10-15 20:06:38 20
  • This command works by rsyncing the target directory (containing the files you want to delete) with an empty directory. The '--delete' switch instructs rsync to remove files that are not present in the source directory. Since there are no files there, all the files will be deleted. I'm not clear on why it's faster than 'find -delete', but it is. Benchmarks here: https://web.archive.org/web/20130929001850/http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html


    11
    rsync -a --delete empty-dir/ target-dir/
    malathion · 2016-06-07 16:56:55 18
  • -r for recursive (if you want to copy entire directories) src for the source file (or wildcards) dst for the destination --progress to show a progress bar


    10
    rsync -rv <src> <dst> --progress
    fecub · 2011-08-05 09:29:12 14
  • connect to a remote server using ftp protocol over FUSE file system, then rsync the remote folder to a local one and then unmount the remote ftp server (FUSE FS) it can be divided to 3 different commands and you should have curlftpfs and rsync installed


    9
    curlftpfs ftp://YourUsername:YourPassword@YourFTPServerURL /tmp/remote-website/ && rsync -av /tmp/remote-website/* /usr/local/data_latest && umount /tmp/remote-website
    nadavkav · 2009-03-31 18:01:00 10
  • Assumed dir A, B, C are subdirs of the current dir Exact syntax of the command is: rsync -v -r --size-only --compare-dest=/path_to_A/A/ /path_to_B/B/ /path_to_C/C/ (do not omit end-slashes, since that would copy only the names and not the contents of subdirs of dir B to dir C) You can replace --size-only with --checksum for more thorough file differences validation Useful switch: -n, --dry-run perform a trial run with no changes made


    9
    rsync -v -r --size-only --compare-dest=../A/ B/ C/
    knoppix5 · 2013-09-10 21:41:16 7
  • Create a exact mirror of the local folder "/root/files", on remote server 'remote_server' using SSH command (listening on port 22) (all files & folders on destination server/folder will be deleted)


    8
    rsync -e "/usr/bin/ssh -p22" -a --progress --stats --delete -l -z -v -r -p /root/files/ user@remote_server:/root/files/
    storm · 2009-02-11 02:44:00 19
  • rsync from source to dest all between >30


    8
    rsync -avz -e ssh --files-from=<(find -mtime +30 -mtime -60) source dest
    voyeg3r · 2009-03-13 12:58:28 18
  • Yes, rsync(1) supports local directories. And, should anything change, it's trivial to run the command again, and grab only the changes, instead of the full directory.


    8
    rsync -a /etc /destination
    atoponce · 2011-10-18 13:07:55 4
  • transfer files from localhost to a remotehost.


    7
    rsync -av --progress ./file.txt user@host:/path/to/dir
    aoiaoi · 2009-02-06 11:51:51 13
  • Copies the complete root-dir of a linux server to another one, where the new harddisks formated and mountet. Very useful to migrate a root-server to another one.


    7
    rsync -ayz -e ssh --exclude=/proc --exclude=/sys --exclude=/dev / root@NEWHOST:/MNTDIR
    bones · 2012-11-06 09:43:42 698
  • dname is a directory named something like 20090803 for Aug 3, 2009. lastbackup is a soft link to the last backup made - say 20090802. $folder is the folder being backed up. Because this uses hard linking, files that already exist and haven't changed take up almost no space yet each date directory has a kind of "snapshot" of that day's files. Naturally, lastbackup needs to be updated after this operation. I must say that I can't take credit for this gem; I picked it up from somewhere on the net so long ago I don't remember where from anymore. Ah, well... Systems that are only somewhat slicker than this costs hundreds or even thousands of dollars - but we're HACKERS! We don't need no steenkin' commercial software... :)


    6
    rsync -a --delete --link-dest=../lastbackup $folder $dname/
    pamirian · 2009-08-04 07:08:54 12
  • If you have lots of remote hosts sitting "behind" an ssh proxy host, then there is a special-case use of "rsynch" that allows one to easily copy directories and files across the ssh proxy host, without having to do two explicit copies: the '-e' option allows for a replacement "rsh" command. We use this option to specify an "ssh" tunnel command, with the '-A' option that causes authentication agent requests to be forwarded back to the local host. If you have ssh set up correctly, the above command can be done without any passwords being entered.


    5
    rsync -avz -e 'ssh -A sshproxy ssh' srcdir remhost:dest/path/
    totoro · 2009-03-25 21:29:07 11
  • Creates files in $DATE and hardlinks existing files to $PREVDATE. Thus full backup in each directory. Only drawback is changed modification time. Recommend a wrapper script to determine $DATE and $PREVDATE. Works like a charm. (Dirvish handrolled)


    5
    rsync -a --link-dest=/media/backup/$HOSTNAME/$PREVDATE '--exclude=/[ps][ry][os]' --exclude=/media/backup/$HOSTNAME / /media/backup/$HOSTNAME/$DATE/
    danam · 2010-01-30 08:36:41 37
  • With this cron, rsync begins to sinchronize the contents of the local directory on /[VIPdirectory] with the directory /backup/[VIPdirectory] on the remote server X.X.X.X. Previously we need working on public/private-keys ssh to guarantee the acces to the remote server on X.X.X.X Show Sample Output


    5
    0 10 * * * rsync -rau /[VIPdirectory] X.X.X.X:/backup/[VIPdirectory]
    mack · 2010-03-02 17:48:54 3

  • 5
    rsync -rv --include '*/' --include '*.txt' --exclude '*' srcDir/ desDir/
    anhpht · 2011-10-23 07:45:16 5

  • 5
    rsync -avxHAXW --info=progress2 /old-disk /new-disk/
    bugmenot · 2016-03-01 19:17:08 16
  • Apart from an exact copy of your recent contents, also keep all earlier versions of files and folders that were modified or deleted. Inspired by EVACopy http://evacopy.sourceforge.net Show Sample Output


    5
    backup() { source=$1 ; rsync --relative --force --ignore-errors --no-perms --chmod=ugo=rwX --delete --backup --backup-dir=$(date +%Y%m%d-%H%M%S)_Backup --whole-file -a -v $source/ ~/Backup ; } ; backup /source_folder_to_backup
    pascalvaucheret · 2018-08-02 21:27:29 274
  •  1 2 3 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Get the full path of a bash script's Git repository head.
Rather than complicated and fragile paths relative to a script like "../../other", this command will retrieve the full path of the file's repository head. Safe with spaces in directory names. Works within a symlinked directory. Broken down: $cd "$(dirname "${BASH_SOURCE[0]}")" temporarily changes directories within this expansion. Double quoted "$(dirname" and ")" with unquoted ${BASH_SOURCE[0]} allows spaces in the path. $git rev-parse --show-toplevel gets the full path of the repository head of the current working directory, which was temporarily changed by the "cd".

Tail a log file with long lines truncated
This truncates any lines longer than 80 characters. Also useful for looking at different parts of the line, e.g. cut -b 50-100 shows columns 50 through 100.

Make redirects to localhost via /etc/hosts more interesting
Normally when a site is blocked through /etc/hosts, traffic is just being redirected to a non-existent server that isn't going to respond. This helps get your point across a little more clearly than a browser timeout. Of course you could use any number of codes: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes Obviously, this command can be added to init-rc.d, and more sophisticated responses can be given. Seems noteworthy to mention that the information sent from the browser can be parsed using the bash READ builtin (such as 'while read -t 1 statement; do parsing'), and the connection stays open until the script exits. Take care that you must use EXEC:'bash -c foo.sh', as 'execvp' (socat's method for executing scripts) invokes 'sh', not 'bash'.

mysqlcheck --defaults-file=/etc/mysql/debian.cnf --auto-repair --all-databases

Poor man's nmap for a class C network from rfc1918
What do you do when nmap is not available and you want to see the hosts responding to an icmp echo request ? This one-liner will print all hosts responding with their ipv4 address.

Get AWS temporary credentials ready to export based on a MFA virtual appliance
You might want to secure your AWS operations requiring to use a MFA token. But then to use API or tools, you need to pass credentials generated with a MFA token. This commands asks you for the MFA code and retrieves these credentials using AWS Cli. To print the exports, you can use: `awk '{ print "export AWS_ACCESS_KEY_ID=\"" $1 "\"\n" "export AWS_SECRET_ACCESS_KEY=\"" $2 "\"\n" "export AWS_SESSION_TOKEN=\"" $3 "\"" }'` You must adapt the command line to include: * $MFA_IDis ARN of the virtual MFA or serial number of the physical one * TTL for the credentials

positions the mysql slave at a specific master position
say you want to reinitialize the slave database without resetting the master positions. You stop the slave, dump the master database with --master-data=2 then execute the command on the slave and wait for it to stop at the exact position of the dump. reinit the slave db and start the slave. enjoy.

Create a 100MB file for testing transfer speed

generate a telephone keypad
seems a useless command ...

Delete empty directories
Recursively delete empty directories. Use with care.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: