systemctl Persistent timer and service, when computer turned off

Question

How do the systemctl timer works when the computer is turned off at the given trigger time? There is the option "Persistent", but when exactly is the command executed? In how far is it guaranteed that the command will be safely executed, e.g. that a maximum of given time shall not pass between two executions?

status:

$ systemctl status mintupdate-automation-upgrade.timer
● mintupdate-automation-upgrade.timer - Update Manager automatic upgrades
     Loaded: loaded (/lib/systemd/system/mintupdate-automation-upgrade.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Fri 2021-01-22 20:20:24 CET; 4 days ago
    Trigger: Thu 2021-01-28 00:44:21 CET; 12h left
   Triggers: ● mintupdate-automation-upgrade.service

configuration files

$ systemctl cat mintupdate-automation-upgrade.*
# /lib/systemd/system/mintupdate-automation-upgrade.timer
[Unit]
Description=Update Manager automatic upgrades

[Timer]
OnCalendar=daily
OnStartupSec=60m
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

# /lib/systemd/system/mintupdate-automation-upgrade.service
[Unit]
Description=Update Manager automatic upgrades
After=network-online.target

[Service]
Type=oneshot
CPUQuota=50%
CPUWeight=20
IOWeight=20
ExecStart=/usr/lib/linuxmint/mintUpdate/automatic_upgrades.py

[Install]
WantedBy=multi-user.target

The timer has the Persistent flag, but the service (is triggered by timer) has not

$ systemctl show mintupdate-automation-upgrade.timer --property=Persistent
Persistent=yes

Answer 1

Persistent= is only applicable to timers (see man systemd.timer). That's why you don't see it with

systemctl show mintupdate-automation-upgrade.service --property=Persistent

When we look at the timer it says:

[Timer]
OnCalendar=daily
OnStartupSec=60m
RandomizedDelaySec=60m
Persistent=true

• OnCalendar=daily: Defines realtime (i.e. wallclock) timers with calendar event expression daily. which effectively means that it will run at midnight every night.
• Persistent=true: means "The time when the service unit was last triggered is stored on disk. When the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was powered down."
• OnStartupSec=60m defines a timer relative to when the service manager was first started.
• RandomizedDelaySec=60m Delays the timer by a randomly selected, evenly distributed amount of time between 0 and 60m. Each timer unit will determine this delay randomly before each iteration, and the delay will simply be added on top of the next determined elapsing time. This setting is useful to stretch dispatching of similarly configured timer events over a certain time interval, to prevent them from firing all at the same time, possibly resulting in resource congestion.

So if you leave your machine on overnight, it'll fire between midnight and 01h00. If you reboot your machine it'll fire between 1 and 2 hours after boot. If you shutdown your machine overnight, then in the morning it'll trigger within 1 hour of booting (for the OnCalendar/Persistent trigger) then again between 1 and 2 hours of booting (for the OnStartupSec trigger).

I suspect you're most worried about the case when you boot your machine at 23h00. In that case, it'll fire twice between 00h00 and 01h00. When a trigger fires, it will be ignored if the oneshot service is still in the activating state which is the case while the ExecStart= is still running. Therefore you will not have two concurrent triggers of the same service and hence it is safe.

But then you ask about a minimum time between two executions. We generally try not to keep things safe with "time". This is akin to adding sleep in a bash script which is usually a work-around when we are too lazy to listen for the appropriate signals (./foo & sleep 5 && ./bar). It's also volatile-work around as there is no guarantee that things will actually be ready when the time expires. If this script is written well, it should execute several times in a row without needing any time separating it (./foo && ./foo && ./foo), and so you shouldn't have to worry about the safety.

Answer 2

It seems not to work, when the computer is turned off or no internet connection, it will not catch up as the message still says:

$ systemctl --no-pager status mintupdate-automation-upgrade.timer
...
Trigger: Tue 2021-02-02 00:32:40 CET; 13h left

So the update process (at least using this method, maybe in contrast to unattended-upgrades) is not really safe.

If there is the possibility to fix it, let me know?