create an incremental backup of a directory using hard links

rsync -a --delete --link-dest=../lastbackup $folder $dname/
dname is a directory named something like 20090803 for Aug 3, 2009. lastbackup is a soft link to the last backup made - say 20090802. $folder is the folder being backed up. Because this uses hard linking, files that already exist and haven't changed take up almost no space yet each date directory has a kind of "snapshot" of that day's files. Naturally, lastbackup needs to be updated after this operation. I must say that I can't take credit for this gem; I picked it up from somewhere on the net so long ago I don't remember where from anymore. Ah, well... Systems that are only somewhat slicker than this costs hundreds or even thousands of dollars - but we're HACKERS! We don't need no steenkin' commercial software... :)

6
By: pamirian
2009-08-04 07:08:54

These Might Interest You

  • 'data' is the directory to backup, 'backup' is directory to store snapshots. Backup files on a regular basis using hard links. Very efficient, quick. Backup data is directly available. Same as explained here : http://blog.interlinked.org/tutorials/rsync_time_machine.html in one line. Using du to check the size of your backups, the first backup counts for all the space, and other backups only files that have changed. Show Sample Output


    1
    rsync -av --link-dest=$(ls -1d /backup/*/ | tail -1) /data/ /backup/$(date +%Y%m%d%H%M)/
    dooblem · 2010-08-05 19:36:24 0
  • This will create an exact duplicate image of your hard drive that you can then restore by simply reversing the "if" & "of" locations. sudo dd if=/media/disk/backup/sda.backup of=/dev/sda Alternatively, you can use an SSH connection to do your backups: dd if=/dev/sda | ssh user@ssh.server.com dd of=~/backup/sda.backup


    15
    sudo dd if=/dev/sda of=/media/disk/backup/sda.backup
    bandit36 · 2009-02-27 20:23:37 2
  • This command will backup the entire / directory, excluding /dev, /proc, /sys, /tmp, /run, /mnt, /media, /lost+found directories. Let us break down the above command and see what each argument does. rsync: A fast, versatile, local and remote file-copying utility -aAXv: The files are transferred in ?archive? mode, which ensures that symbolic links, devices, permissions, ownerships, modification times, ACLs, and extended attributes are preserved. -/: Source directory -exclude: Excludes the given directories from backup. -/mnt: It is the backup destination folder. Please be mindful that you must exclude the destination directory, if it exists in the local system. It will avoid the an infinite loop. To restore the backup, just reverse the source and destination paths in the above command.


    0
    rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/home/*","/lost+found/*"} <backup path> > <path_of log file>
    vinabb · 2017-07-26 13:33:50 0
  • Usage example: display output of a command running in the background at desired time The example in details: report disk quotas and that backup process will start soon In my /etc/crontab file I added following four lines for weekly automatic incremental backup: . 52 13 * * 7 root mount /dev/sda3 /media/da2dc69c-92cc-4249-b2c3-9b00847e7106 . 53 13 * * 7 knoppix5 df -h >~/df.txt . 54 13 * * 7 knoppix5 env DISPLAY=:0 /usr/bin/gedit ~/df.txt && wmctl -a gedit . 55 13 * * 7 root /home/knoppix5/rdiff-backup.sh . line one: as root mount media for backup on Sunday 13:52 line two: as user knoppix5 write out to text file in home directory the free space of all mounted disks on Sunday 13:53 line three: in front of you open and display a very simple text editor (I prefer gedit) with content of previously reported disk usage at Sunday 13:54 wmctl -a gedit means (from the manual): -a Switch to the desktop containing the window , raise the window, and give it focus. line four: as root run incremental backup script rdiff-backup.sh as root on Sunday 13:54 . my rdiff-backup.sh, with root permissions backups in short time (writes only changes from the last backup) the etire linux system (except excluded - i.e. you don't want backup recursively your backup disk), looks like this (Show sample output): Show Sample Output


    0
    env DISPLAY=:0 /usr/bin/gedit ~/df.txt && wmctl -a gedit
    knoppix5 · 2015-04-12 13:48:31 4

What Others Think

this command is completely useless as the hardlinked "backuped" files will change, if the original file changes. Only use is to recoveraccidentally deleted files - but by the option "--delete" these will be deleted in yout backup also next time you run this ... DOWN DOWN DOWN
sneaker · 459 weeks and 6 days ago
No, removing the --delete part doesn't remove the hardlinked files, it just maintains the integrity of the backup that the snapshot reflects. The originals are still maintained in the previous backup. The assertion by sneaker is true but misleading that when the hardlinked files change this modifies the file to which it links. You see, a new version of the file is created in each "incremental" backup by the rsync if any changes have occurred, so there's no worry that you are going to retrieve a backup of a file with modifications that have happened at a future date. The failed assumption is that you are always backing up to the same location (directory), which is false, friends. Read pamirian's explanation of the command more carefully. Each backup gets its own directory, which builds on the backup before it UNLESS there is a change to any given file in the backup in which case a new copy is created rather than hardlinked back. If a file has been deleted at the location of origin, that file is simply omitted from the incremental backup you're doing. It isn't "deleted" in all previous backups. UP UP UP This is the same technology used by microsoft shadow copy technology for their fileshares, and also by netapp SANs that have LUNs with snapshot volumes enabled. It's a good thing though that sneaker pointed out his/her reservations, so that those common misconceptions could be revealed and dispelled. Now the otherwise unseen argument could never have been resolved. Kudos to both pamirian and sneaker.
linuxrawkstar · 459 weeks and 5 days ago
as long as you remember that --link-dest requires rotation and an empty target dir, you'll be fine. If you still don't get it, try using a system that hardlinks and rotates for you, such as rsnapshot.
rkulla · 416 weeks and 4 days ago
You find the concept beautifully explained with a script to put those backups on a remote location here: http://blog.interlinked.org/tutorials/rsync_time_machine.html
joedhon · 370 weeks and 2 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: