Efficient Backups Using rsync And An External Disk

[Seth W. Klein]

I was fixing my backups the other day, went looking, and discovered rsync --link-dest. With it, rsync substitutes hard links for files that haven't changed. The result:

tria tria # du -hs backup-*1.7G backup-2008-03-19
82M backup-2008-03-20

40M backup-2008-03-21

40M backup-2008-03-22

76M backup-2008-03-23

37M backup-2008-03-24

63M backup-2008-03-25

206M backup-2008-03-26

42M backup-2008-03-27

101M backup-2008-03-28

70M backup-2008-03-29

52M backup-2008-03-30

64M backup-2008-03-31

101M backup-2008-04-01

tria tria # du -s backup-* | tail -n +2 | cut -f 1 \

| sed '1s/^/0 0 /; s/$/ + r 1 + r/; $s|$| 1 k r / p 1024 / p|' | dc

76286.1

74.4

tria tria # du -s backup-* | cut -f 1 \

| sed '1s/^/0 0 /; s/$/ + r 1 + r/; $s|$| 1 k r / p 1024 / p|' | dc

196095.7

191.4

tria tria # df /mnt/ext1/

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/sdd1 307663800 46438044 245597324 16% /mnt/ext1

tria tria # dc

307663800 76286 / p

4033

1 k

365 / p

11.0

q

tria tria #

Average of 74M a day, 191M and dropping if you include the first. Nearly 11 years before I'll have to delete one, and I expect disk failure will force me to replace the drive with one that will inevitably be larger long before I get this one full. If I want to clear space, I can delete any of them, even the first. The drive cost less than 200 USD.

Accessing stuff from backups is only seconds slower than from master—locate even indexes the whole thing while I sleep.

Scripted and run from cron, the process costs me 10 or 20 seconds every day to review the status email, and it would cost nothing if I was naive.

Here's relevant portions of the backup script :

#!/bin/sh#args:
ensure_mounted() {

mount | cut -d ' ' -f 3 | grep -Fx "$1" >/dev/null || {

echo "$2" 1>&2

exit 1

} || exit $?

}

#args:

ensure_dir() {

test -d "$1" || {

echo "$2" 1>&2

exit 1

} || exit $?

}

#args:

#output: code to set $dest and $prev

directories() {

local dest=$1/`date "+$2-%Y-%m-%d"`

local prev=`find "$1" -maxdepth 1 -name "$2-*" -type d | sort | tail -n 1`

case $prev in '')

prev=$dest

esac

echo "dest='$dest'"

echo "prev='$prev'"

}



local_backup() {

local base=/mnt/ext1/tria

ensure_dir "$base" \

"Error: cannot run local backup (disk likely not mounted)"

eval `directories "$base" backup`

echo "using $prev for hardlinks"

rsync -aRv --link-dest "$prev" \

/etc \

/var \

--exclude '/var/cache' \

--exclude '/var/db' \

--exclude '/var/tmp' \

/home/sk \

--exclude '/home/sk/.mozilla/firefox/*.default/Cache/*' \

"$dest"

rsync -av --link-dest "$prev" \

/usr/src/linux/.config \

"$dest/kernel.config"

}



case $1 in

-l) local_backup ;;

*) echo "Usage: backup [-l]" 1>&2; exit 1 ;;

esac