disabled/static systemd service unit is always started at reboot (systemd timers as cron replacement)

Question

Summary

A systemd service unit that is disabled ("static") and is only supposed to be triggered by a timer is run on every reboot.

Background

This was a service unit (in /etc/systemd/system/) that previously had an [Install] section and was systemctl enabled.

The [Install] section was removed and the service disabled. The service is now triggered by a timer. The timer is set to run once per month and is persistent so it tracks last runs and won't trigger on reboot unless a run was missed while shutdown.

I ran systemctl --system daemon-reload after making the changes.

The timer works fine and triggers the service on schedule as expected.

The problem

On reboot the service unit always runs regardless of last run of the timer and the fact the timer is persistent. I've verified (via systemctl list-timers) that it is not the timer unit that is triggering the service unit (unless the list-timers output about last triggered time is wrong).

systemctl is-enabled <service-unit> shows static (aka disabled, no [Install] section in unit).

find /etc/systemd/system/*.wants -name <service-unit> does not show any installed symlinks left-over from when this service was previously installed/enabled.

I suspect there is something "left over" from when this service-unit was previously installed that is causing this service to start on reboot but don't know where to look.

This is on ubuntu 20.04 (in case there happens to be a known bug/issue).

Is there a way to debug why systemd started a unit? (e.g. unit X started because wants Y in file Z).

Is there a way to double-check this service really wasn't started by the timer (instead of just going by the list-timers output)?

Service Unit

# cat /etc/systemd/system/mysql_tzinfo.service
[Unit]
Description=mysql_tzinfo
Wants=mysql_tzinfo.timer

[Service]
Type=oneshot
Environment=
WorkingDirectory=/tmp
ExecStart=/bin/sh -c "/usr/bin/mysql_tzinfo_to_sql /usr/share/zoneinfo | /usr/bin/mysql --user=root mysql"

User=root
Group=root

Timer Unit

# cat /etc/systemd/system/mysql_tzinfo.timer
[Unit]
Description=Timer for mysql_tzinfo.service
Requires=mysql_tzinfo.service

[Timer]
Unit=mysql_tzinfo.service
OnCalendar=*-*-05 04:00:00
AccuracySec=30s
Persistent=true

[Install]
WantedBy=timers.target

Answer 1

Okay, in the process of writing this up I noticed a Requires=mysql_tzinfo.service in the [Unit] section of my timer unit.

It occurred to me that the starting of the timer at boot (dependency resolution by systemd) might be starting the service due to this Requires. Sure enough, removing this line from the timer and rebooting....the service no longer starts. That is what I get for not closely checking every config option when following "technical blogs".

Summary of important points for people that are trying to move from cron to systemd timers:

• Do not add an [Install] section to your service unit and do not systemctl enable your SERVICE unit.
• Do not add a Requires=<SERVICE_NAME>.service to your TIMER unit.
• If you want your timer to remember when it ran and run missed runs on boot, add Persistent=true to your TIMER.
• The timers run fairly early at boot and so you should add the necessary After= requirements to the SERVICE unit (that your TIMER unit runs) to make sure the service unit doesn't run until all required services are online (network, database, etc.). Otherwise it will likely fail.