Finding an appropriate backup strategy (borgbackup, restic etc)

Question

I am fully aware that questions like this one have been asked multiple times. However, since reading lots of documentation of various backup tools didn't answer all my questions, I think its fine to describe my scenario here to get some input.

I want to develop a backup strategy that suits my needs. Here is a quick rundown of the requirements:

• Open-Source Software
• Commandline support (Not just a GUI)
• I want to backup and restore from/to multiple devices. (All devices are running linux)
• I want to be able to backup specific files/folders, and ignore others
• I want to be able to backup to multiple destinations, e.g.:
  • An external HDD
  • Some off-site server (In case my house burns down)
• I would like to automate backups, e.g. using cron
• I would like that backups can be encrypted properly
• I would like the backups to be deduplicating, so, only the delta to the previous backup is stored.
• When restoring from a backup I'd like to be able to directly use the restored backup as a new point to backup new changes. (Think git-repository, where you can directly push new changes after you cloned the repository to a new system).

After conducting some research, there are two programs that for the most part seem to be a good fit:

• borgbackup
• restic

With borgbackup I see two problems:

• I can restore files in two ways: Either I fuse-mount the backup or directly extract the backup. However, when restoring, I always get just get the files. When I restore a backup to a new system, I have to set all the repositories up fresh again, in order to run new backups from the new host? That is very inconvenient. (Again: I would love a similar behavior like a git repository, where the restored data/repository directly can be used to backup (push) new data.)
• I don't think its possible to specify multiple endpoints for backups. E.g. I can backup my data to a remote server, but I can't also sync to my external HDD. (See: https://borgbackup.readthedocs.io/en/stable/faq.html#can-i-backup-from-multiple-servers-into-a-single-repository). The way I see it, cold storage backups, like an external HDD doesn't seem to go to well with borg.

With restic:

• The same as with borg, essentially.

So, my question is, how can I achieve a backup solution that works for the scenario I described above. Are borg and restic the right tools for the job here? If so, how do I deal with the problems I described? If not, is there other backup software that would be better here?

Answer 1

rclone I think would meet all of your requirements:

• Open-Source Software
• Commandline support
• backup to and restore from multiple devices (multiple OS support)
• backup specific files/folders, and ignore others
• backup to multiple destinations (just not multiple in one run, would have to be combined with either xargs or GNU parallel to achieve that)
• automate backups, e.g. using cron
• backups can be encrypted
• only the delta to the previous backup is stored (=> incremental)
• use restored backup as a new backup target

Maybe you want to try rclone sync src dest --backup-dir incr$(date "+%Y%m%d.%H%M%S")

As you mentioned git: You surely also want to have a look at git-annex. The above points are claimed for that, too. (To be honest: while I know rclone to be rock solid from years long personal experience (has never failed me so far), I have never tried git-annex to date, despite its features looking very intriguing)

In general it really depends on your environment. The actual question is rather: Are you going to establish a solution as a continuous service for multiple users and lots of machines, then better use a full grown system like suggested in https://unix.stackexchange.com/a/683353/118985 and create a dedicated backup environment.

However, if that thing is for your private system and only to cover few computers - which I suspect from your writing above - the administrative effort and resources required are far smaller, and you will be a lot faster and experience less frustration in getting the system running with rclone: It is only a single executable to be put in your search path, and - in most cases, to use encrypted or cloud backup - one small configuration file in your home directory, which is created by the program).

Answer 2

set up two linux (i.e. centos) systems and use rsnapshot

• Open-Source Software
  • yes
• Commandline support (Not just a GUI)
  • there is no gui, rsnapshot is a simple perl script and everything is configured in /etc/rsnapshot.conf
• I want to backup and restore from/to multiple devices. (All devices are running linux)
  • can do
• I want to be able to backup specific files/folders, and ignore others
  • can do
• I want to be able to backup to multiple destinations, e.g.: An external HDD or Some off-site server (In case my house burns down)
  • can do, need internet connection obviously, and some backup system wherever else to receive
• I would like to automate backups, e.g. using cron
  • call rsnapshot from within cron however frequently you want backup to happen, very simple
• I would like that backups can be encrypted properly
  • read rsnapshot website, not sure about this one. I believe an SSH connection between you and your backup server would be the first level of encryption; if you need everything on the backup server "encrypted" then that's kind of a different matter, and not necessarily the job for rsnapshot.
• I would like the backups to be deduplicating, so, only the delta to the previous backup is stored.
  • read the rsnapshot website, it may offer an acceptable solution if you configure rsnapshot to do incremental backups. There might be other software you can run, in linux, on the backup server, that accomplishes deduplication to meet your needs.
• When restoring from a backup I'd like to be able to directly use the restored backup as a new point to backup new changes. (Think git-repository, where you can directly push new changes after you cloned the repository to a new system).
  • rsnapshot is based on rsync so you would have a folder structure on your backup server exactly like you have on your main server, and then if you configure NFS or anything similiar on the backup server you can export that backup folder structure back to your main (or any other) server to be used.

I run rsnapshot on one server having 16 drive bays, first 7 bays is raid-5 mounted as /data. Second 7 bays is raid-5 mounted as /backup that is root only access. I nfs export /backup as readonly to localhost (just my server) and mount /backup as read-only under a /backup2 folder that user's can access. The folder/file permissions are mirror that of data, so if a user has permission to do so they can navigate under /backup2 to copy out anything they want when they've accidentally deleted their original under /data. In your case would need to configure a second system to receive the rsnapshot backup. There are many online articles regarding rsnapshot.

• https://troy.jdmz.net/rsnapshot/index.html
• https://rsnapshot.org/
• https://rsnapshot.org/downloads/
  • rsnapshot-1.4.4.tar.gz

Answer 3

My advice is to use bacula. I have set it up in a fairly large scale, using the specifics you require. While it can be a bit tricky to understand at first, you will be very happy with it when you set it up. The community is also quite good. Do not waste your time with buffed up rsync clones.