1

I have a home server running Ubuntu 14.04.3 LTS with an ext4 file system, to which I regularly backup data from various devices.

Now I have set-up an external USB HDD with ZFS file system and would like to backup all of the data from the home server onto that USB drive. And since I am using ZFS on the external drive, I would also want to use ZFS's snapshot abilities.

Because I would be copying files from ext4 to ZFS, I cannot use the zfs send command. Therefore I am stuck with rsync.

I have been using rsync for a while, so I am quite familiar with it. But, since I will be performing snapshots on the target file system, I want to waste as little space as possible.

I came across this blog post which uses the following command:

/usr/bin/rsync -axH --no-whole-file --inplace --delete / /$backup_target/root/

Aside from the usual flags like -a, -x, and --delete it also uses:

  • -H (hard links),
  • --no-whole-file (force enable delta-xfer algorithm), and
  • --inplace (update folder instead of making a new backup).

My question: are those flags sufficient or should I add or remove some?

The backup flow will be as follows:

  1. Backup with rsync to the same directory every time
  2. Create snapshot of that directory using zfs snapshot backup@'date +%Y%m%d'

I am guessing that --inplace is a must (because of backuping to the same directory every time). Same probably goes for --no-whole-file since it only copies changed block of data. But what about -H? Any other flags I should add?

Improve this question
4
  • 1
    I run rsync in a similar scenario with -aAHX --delete --numeric-ids --relative --delete-excluded plus some --exclude parameters. You may want to have a look at that parameter set as a starting point. Using -x may or may not be appropriate depending on how your file systems are set up.
    – user
    Commented Sep 5, 2015 at 12:01
  • Everything is on the same file system, so I guess -x doesn't make much sense. I also don't use ACL's in fstab so -A is probably not useful as well. I also don't need --relative and will add --delete-excluded if I exclude anything. On the other hand, -X and --numeric-ids look useful, so I will probably use them. Would you agree that --inplace and --no-whole-file are also good to use?
    – alesc
    Commented Sep 5, 2015 at 12:26
  • 1
    I don't know about the latter two, but I suspect they can't really hurt. If everything is on the same file system, then -x is probably what you want (unless you want to copy the backup target directory into itself... recursively.)
    – user
    Commented Sep 5, 2015 at 12:39
  • Thanks again for the tip. I will stay with x then. I have been using it all the time, but wasn't aware that it prevents recursive copying :) If you wish, you can put everything into an answer, so I can reward you.
    – alesc
    Commented Sep 5, 2015 at 12:46

1 Answer 1

Reset to default
3

The command line you found provides a reasonable starting point. I run rsync for backups (through rsnapshot, which isn't really relevant because you will be using ZFS snapshots for history; I started using that before I started using ZFS, and have kept using it because it works well for me) with this set of parameters:

-aAHX --delete --numeric-ids --relative --delete-excluded

plus some --exclude parameters for things I don't want backed up. The major difference between my setup and the one you found is probably the use of -A (--acls) and --delete-excluded (also delete excluded files from dest dirs).

As we discussed in the comments, --inplace and --no-whole-file are options that probably can't really hurt in your use case.

-x (--one-file-system) is primarily useful when you have only one or a small number of file systems you want backed up. (Since I use ZFS file systems extensively on the source side, I instead --exclude the backup target location.) -x tells rsync to not recurse into file systems other than those listed on its command line as sources. One way or another, you will need to keep the target location out of rsync's idea of source locations; otherwise, on the first backup you will get (say) /* copied into /backup/, then next time you will get /* and as part of that /backup copied into /backup, and so on. This is what I meant when I mentioned in the comments recursively copying the backup target directory into itself.

Create snapshot of that directory using zfs snapshot backup@'date +%Y%m%d'

You want to make that zfs snapshot backup@$(date +'%Y%m%d'), obviously. (Either use $() or use backticks; not apostrophes. I prefer the $() syntax; for one thing, it's nestable.)

Also, make sure to have a snapshot pruning strategy in place for when you start running low on disk space.

Improve this answer
2
  • I was using backtics for inserting the date but it wasn't compatible with this site's markup (which also uses backtics for inline code). So I changed them to apostrophes and forgot to mention that. But $() looks better so I will probably use that. For now, I won't have any automated pruning strategy but will manually delete older snapshots as needed. Is zfs destroy backup@some-date enough to free the space or do I need to perform some other actions?
    – alesc
    Commented Sep 5, 2015 at 13:27
  • @alesc zfs destroy backup@some-date should be enough if you have only a single file system in the pool. Otherwise, consider using something like zfs destroy backup@some-date -r. Be aware that destroying snapshots is a potentially lengthy operation, even though the zfs destroy command may exit quickly. Check man zpool-features for async_destroy and consider enabling it on your pool if it isn't already enabled.
    – user
    Commented Sep 5, 2015 at 13:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .