systemd service on shutdown with disk access [backup]

Question

I'd like to run a little backup script (btrfs snapshot + transfer to usb) on system shutdown.

The following service executes perfectly fine on shutdown, but not on reboot as intended:

[Unit]
Description=Backup on poweroff to external encrypted USB disk.
DefaultDependencies=no
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
RemainAfterExit=yes

[Install]
WantedBy=poweroff.target

Where /usr/bin/sleep 30 will be replaced by the actual backup-script once this service works as intended.

---

But for the backup-script to work properly I need to decrypt an external USB disk with cryptsetup before it is started. This is all handled in the background by a combination of /etc/crypttab (noauto) and systemd (via systemd-cryptsetup-generator), which auto-generates a systemd unit-file systemd-cryptsetup@<name>.service that I can reference in the above unit.

But as soon as I add the following requirement to the [unit] section:

[email protected]
[email protected]

The service is not started anymore during shutdown/poweroff of the system.

Starting the service manually via systemctl start poweroff-backup.service works without a problem, so there is no issue with the service-file itself... :-/

---

I think the issue is that the poweroff.target via systemd-poweroff.service requires the umount.target which in turn unmounts all devices (including cryptsetup 'mounts') via a combination of ConflictedBy= and After= statements for the systemd mount und cryptsetup units. But adding another requirement towards the umount.target does not change anything, eg:

Before=umount.target

Without any of the additional requirements I can clearly see that the service is always started right after the umount.target has been reached.

Answer 1

Okay I found the issue and from that I was able to build a correctly working solution that stills follows the systemd workflow (for the most part).

Issue

The following service, while on first sight looking correct will never be executed:

# poweroff-backup.service
[Unit]
Description=Backup on poweroff to external encrypted USB disk.
DefaultDependencies=no
Before=shutdown.target

# This causes the service to be discarded/skipped due to
# a dependency cycle conflict. See below... 
[email protected]
[email protected]
Before=umount.target

[Service]
Type=oneshot
# Note:  The script executed here needs access to '/', '/tmp' 
# (or Systemd's PrivateTemp) and '/dev/mapper/ext' which is the 
# decrypted USB partition managed by /etc/crypttab and 
# systemd-cryptsetup-generator (as [email protected])
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
# Note: This does not make any difference
RemainAfterExit=yes

[Install]
WantedBy=poweroff.target

The reason why this service will never be started is because there is a dependency cycle issue/conflict between poweroff.target -[require]-> shutdown.target -[conflicts]-> cryptsetup.target <-[require]- [email protected] <-[require]- poweroff-backup.service <-[wants]- poweroff.target, where shutdown.target conflicts cryptsetup.target to ensure all encrypted disks are detached before shutdown (poweroff/reboot) and poweroff-backup.service which needs it to be activated by poweroff.target.

Solution

Knowing this, the obvious problem lies in my wrong assumption that systemd could somehow avoid this dependency cycle conflict by knowing that poweroff-backup.service is a "one-shot" service that can be resolved first by systemd while continuing with the normal shutdown transition once poweroff-backup.service has been activated and transitioned into inactive/done.

So here is a service that uses ExecStop= instead of ExecStart= which allows us to avoid any dependency-cycles, but has some minor caveats:

# poweroff-backup.service
[Unit]
Description=Backup external encrypted USB disk on poweroff.
[email protected]
After=multi-user.target
[email protected]

[Service]
Type=oneshot
ExecStart=/usr/bin/echo "Waiting for poweroff..."
# Note: We need this, since there is no other way to detect poweroff vs reboot now.
ExecStop=/usr/bin/systemctl list-jobs | /usr/bin/egrep -q 'poweroff.target.*start'
# Note: This should be replaced with your script
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

This works because:

• It prevents the dependency cycle issue caused by shutdown.target, cryptsetup.target and poweroff-backup.service
• The correct working order is still preserved, since systemd ensures that shutdown order is the reverse of the startup order (as given by After= and Before=)

There are only two minor gripes I have with this solution:

• Since the service is started on login, it will also decrypt the external USB disk on login. I would prefer if this would only happen once the actual backup-script is executed while still using the normal systemd After= and Require= order and dependency mechanics. This could be handled by adding the following, but this would mean I'm not using the cryptsetup service for decrypting/detaching the USB disk:

  ExecStop=/usr/lib/systemd/systemd-cryptsetup attach ext
  [...]
  ExecStop=/usr/lib/systemd/systemd-cryptsetup detach ext
  

• I don't have any way to conditionally execute ExecStop= only on shutdown, by referencing the shutdown.target and have to rely on a little helper command using systemctl list-jobs.

Question: Has somebody an idea how to improve on those points?

Full Service File

This is my current service file, which only starts the service once the USB disk is mounted, by attaching (WantedBy=) to the USB device (by its UUID) instead of the multi-user.target:

# poweroff-backup.service
[Unit]
Description=Backup external encrypted USB disk on poweroff.
[email protected]
Requires=-.mount
Requires=tmp.mount
After=dev-disk-by\<uuid here>.device
[email protected]
After=-.mount
After=tmp.mount

[Service]
Type=oneshot
ExecStart=/usr/bin/echo "Waiting for poweroff to trigger snapshot and archive transfer..."
ExecStop=/usr/bin/systemctl list-jobs | /usr/bin/egrep -q 'poweroff.target.*start'
ExecStop=-/usr/local/bin/btrfs-snapshots.sh --device='UUID=<uuid>' @ @boot @home
ExecStop=/usr/local/bin/btrfs-archive.sh --source='UUID=<uuid>' --target=/dev/mapper/ext
TimeoutSec=infinity
RemainAfterExit=true

[Install]
WantedBy=dev-disk-by\<uuid here>.device

Answer 2

I have a slightly different use-case, where I want to ensure all other services are stopped and perform a backup with Rsync during reboot. The following worked for me on Ubuntu 22.04. I am leaving this answer here for people who are looking to run scripts at shutdown in the future.

Source: https://documentation.suse.com/smart/systems-management/html/reference-managing-systemd-targets-systemctl/index.html

The service file looks like this:

[Unit]
Description=Run my custom task at shutdown
DefaultDependencies=no
Before=systemd-reboot.service
After=final.target

[Service]
Type=oneshot
ExecStart=/usr/local/scripts/custom_script.sh
TimeoutStartSec=0

[Install]
WantedBy=systemd-reboot.service

How my custom_script.sh looks like:

#!/bin/sh
ps aux > /usr/local/scripts/processes.txt
df -Th > /usr/local/scripts/df.txt
mount > /usr/local/scripts/mount.txt
sleep 180

As this is just for testing, I ran sudo chmod 777 -R /usr/local/scripts/ to prevent getting permission issues.

Below are the steps I tested it:

1. To make sure the script is run at shutdown and reboot, I have added sleep 180 in the custom_script.sh. Next, I shutdown the OS. It the OS completely powered off only after 3 minutes. And while the script was running (waiting), there was an Ubuntu loading logo shown. I booted up the system and rebooted it. The same Ubuntu loading logo is shown, and the system rebooted after 3 minutes. This proves that the script is being ran at shutdown/or reboot.
2. To make sure all other services are stopped; I logged all the processes that were running when the custom_script.sh is running with ps aux > processes.txt. The output of processes.txt does not contain other services. This proves that all other services have been stopped.
3. To make sure that the file system is still mounted, I logged the mounts with df -Th > df.txt and mount > mount.txt. The output of the files shows all my partitions are mounted and are in rw mode.