systemd: Retrying a timer

Question

I'm using borgbackup to back up a few systems. They all use the same borgbackup repository. I have written a systemd timer that starts a backup job using borgmatic. I have an Ansible playbook that deploys everything and sets up the timer. My timer file looks like this:

[Unit]
Description=Run borgmatic backup

[Timer]
OnCalendar=*-*-* 00:00:00
RandomizedDelaySec=10000
OnBootSec=1200
Persistent=true

[Install]
WantedBy=timers.target

All systems have the same OnCalendar expression. The issue I'm trying to solve is that even with the RandomizedDelaySec set, the backups often run close enough to the same time that one has a lock and the second job fails.

I know that I can solve this in a few different ways, I could use separate repos, I could change the OnCalendar expression with Ansible when I deploy, or I could manually change it. The solution that I'm trying to pursue, for the sake of learning systemd better, is one where I create a configuration that retries after some time on the occasion that a backup job fails. It seems to me like systemd must provide a way to do this. Is it possible? If so, how?

Answer 1

You could use borgs --lock-wait or the equivalent setting in borgmatics configuration (storage -> lock_wait) to make the job wait longer for the lock. This will need some tuning or experimentation, of course, to make sure all backups run with the given timing.