systemd twice mounts entry in fstab during boot, first attempt fails with bad option, second one works fine

Question

Running Debian 5.10 within QEMU, I'm using the 9pfs file system to access files stored on my host system, as Debian comes with built-in support and QEMU supports that file system natively.

When booted, I can simply mount the system using

sudo mount -t 9p -o trans=virtio,msize=131072 hostfs /mnt/host

Works as expected. For those not familiar with how this works: You need to tell QEMU what hostfs actually means, e.g. by adding an option like

-virtfs local,path="$HOME/Documents",mount_tag=hostfs,security_model=mapped-file

That way my Documents folder is exposed as hostfs and once mounted, fully available in the Debian guest system.

As I always want to have access to the host system, I modified /etc/fstab and added the following line:

hostfs /mnt/host 9p defaults,trans=virtio,msize=131072 0 0

And when I then run sudo mount -a, this also works as expected.

However, when I reboot the Debian guest system, the boot fails when systemd tries to mount that entry. It dynamically generates a mnt-host.mount and when trying to mount it, I get

mount: /mnt/host: bad option
mnt-host.mount: Mount process existed, code=exited, status=32/n/a

and this brings the boot process to a halt. I have to hit CTRL+D to contiune booting.

The crazy thing is, once the boot has finished, /mnt/host is mounted!

That's because apparently systemd is trying to mount it again. Later on in the log, there is another mnt-host.mount job executed and this time is just succeeds as expected.

I'm a bit lost here as to why systemd tries to mount that entry twice and why it never succeeds on the first attempt, despite the same options being used twice.

Answer 1

First of all, kudos to Tom Yan, who suggested in a comment to the question:

Maybe there's some kind of race. As a workaround, try adding x-systemd.automount (as an option) to the fstab entry.

While this doesn't explain the origin of the problem, it does indeed work around the problem. Using that option, the system won't try to mount the share immediately at boot (so the boot succeeds), however it will mount it immediately the moment any process tries to access the mount target directory (so you won't ever have to mount it yourself, it will just be there once you try to use it).

Despite this solution pretty good, I usually cannot give up on problems until I fully understood what's going on, so I kept digging and digging and found the real cause of the problem.

When saying bad option, this actually refers to trans=virtio in the mount request. To use this kind of transport, a kernel module is required: 9pnet_virtio.ko. The moment systemd is processing /etc/fstab, it doesn't have access to /lib/modules yet, so it cannot load that module and that's why the mount fails. Apparently at some later time, some other boot script triggers another mount -a (either directly or indirectly by restarting systemd services) and when that happens, /lib/modules is available and thus the mount now succeeds.

So the real solution to this issue is as follows:

1. Edit /etc/initramfs-tools/modules and add a line with the content 9pnet_virtio

2. Exectute sudo update-initramfs -u

The system will rebuild initramfs and make sure this particular module is included as well which fixes the boot error.

There are other ways to fix it, e.g. one can ensure that the share isn't mounted before /lib/modules is fully available, however, all of these solutions will fail if the device containing /lib/modules cannot be mounted for whatever reason. By placing the module onto the initramfs, I now can get access to this share even from a recovery console (I just need to mount it elsewhere) and this can help me to repair a broken system because that way I can access files I may need for a repair by placing them on my host system which is pretty neat IMHO.