thunderbolt SSD won't mount at boot

Question

My system won't boot when I have my Thunderbolt external SSD in my fstab.

I think maybe systemd is trying to mount drives before boltd is running, but I'm not sure how to verify that this is the problem, or rectify if it that is indeed the problem.

I've got a 2018 Mac Mini (an Intel one) and I've installed Ubuntu 22.04 on it.

The box only has 128 GB internal, so I've acquired a 1TB external SSD, a SanDisk PRO-G40.

It's a thunderbolt device, so I did a "boltctl" command which showed the device's UUID and then a "boltctl authorize ". This didn't persist across reboots, so I did a "boltctl enroll " which took care of that problem.

$ boltctl
 ● SanDisk Professional PRO-G40
   ├─ type:          peripheral
   ├─ name:          PRO-G40
   ├─ vendor:        SanDisk Professional
   ├─ uuid:          cc030000-0062-6c0e-8342-bf90cef33103
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     c7010000-0062-640e-03fc-9a92ccf13003
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Sun 20 Aug 2023 02:02:26 AM UTC
   ├─ connected:     Sun 20 Aug 2023 02:02:23 AM UTC
   └─ stored:        Fri 18 Aug 2023 09:14:53 PM UTC
      ├─ policy:     auto
      └─ key:        no

The SSD came with two partitions, a 200M EFI partition and a 930G APFS partition. I left the EFI partition and deleted the APFS partition and added an ext4 partition in its place. Now my lsblk looks like this:

$ lsblk -o NAME,LABEL,FSTYPE,SIZE,UUID,MOUNTPOINTS
NAME        LABEL FSTYPE     SIZE UUID                                 MOUNTPOINTS
loop0             squashfs     4K                                      /snap/bare/5
loop1             squashfs  63.4M                                      /snap/core20/1974
loop2             squashfs  63.5M                                      /snap/core20/2015
loop3             squashfs  73.9M                                      /snap/core22/858
loop4             squashfs 237.2M                                      /snap/firefox/2987
loop5             squashfs 237.2M                                      /snap/firefox/3026
loop6             squashfs 349.7M                                      /snap/gnome-3-38-2004/143
loop7             squashfs 485.5M                                      /snap/gnome-42-2204/120
loop8             squashfs 485.5M                                      /snap/gnome-42-2204/126
loop9             squashfs  91.7M                                      /snap/gtk-common-themes/1535
loop10            squashfs  12.3M                                      /snap/snap-store/959
loop11            squashfs  53.3M                                      /snap/snapd/19457
loop12            squashfs   452K                                      /snap/snapd-desktop-integration/83
nvme0n1                      113G                                      
├─nvme0n1p1 EFI   vfat       300M 5F66-17ED                            /boot/efi
├─nvme0n1p2       apfs      33.5G 416419fe-742b-4653-b709-42e0ff78f4ec 
├─nvme0n1p3       swap       4.7G f7bac42d-0730-40da-be0d-f6de7f4788f2 [SWAP]
├─nvme0n1p4       vfat       954M 32B6-990F                            
└─nvme0n1p5       ext4      73.6G 2532f568-b33b-4583-abd6-2f6e41bb8e28 /
nvme1n1                    931.5G                                      
├─nvme1n1p1 EFI   vfat       200M 67E3-17ED                            
└─nvme1n1p2       ext4     931.3G cb9940d0-5b0b-4273-97d7-b4dd29768926 

Lastly, I added my drive to /etc/fstab (the last line):

# / was on /dev/nvme0n1p5 during installation
UUID=2532f568-b33b-4583-abd6-2f6e41bb8e28 /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=5F66-17ED  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/nvme0n1p3 during installation
UUID=f7bac42d-0730-40da-be0d-f6de7f4788f2 none            swap    sw              0       0
# external SSD mounted to /opt
UUID=cb9940d0-5b0b-4273-97d7-b4dd29768926   /opt    ext4    rw,relatime 0   2

Then I did a "mount -a" and boom, victory!

$ mount | grep opt
/dev/nvme1n1p2 on /opt type ext4 (rw,relatime)
$ df -h /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p2  916G   28K  870G   1% /opt

Except... when reboot, the system wouldn't boot!

[ TIME ] Timed out waiting for device /dev/disk/by-uuid/cb9940d0-5b0b-4273-97d7-b4dd29768926.
[DEPEND] Dependency failed for /opt.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for File System Check on /dev/disk/by-uuid/cb9940d0-5b0b-4273-97d7-b4dd29768926.

and then later:

[FAILED] Failed to start Set Up Additional Binary Formats.
See 'systemctl status systemd-binfmt.service' for details.

I get an option to enter maintenance mode. In there, I do an lsblk but I don't see the external drive (just the internal).

I tried running the boltctl command, but the system just hung; I had to reboot.

As I stated at the outset, I think maybe boltd isn't running at the time systemd is trying to mount drives. I don't have much evidence for this. Any help is appreciated.

-- Mike

EDIT 1: I've found an article about a similar problem with VirtualBox shared folders.

EDIT 2: I've learned that boltd is started by systemd/udev, so I tried modifying my /etc/fstab entry to look like this:

UUID=cb9940d0-5b0b-4273-97d7-b4dd29768926   /opt    ext4    x-systemd.requires=systemd-udevd.service,rw,relatime    0   2

My hope was that this would make this mount unit depend on udev (and therefore boltd) before starting. Unfortunately, this did not change anything that I could see. I tried x-systemd.after as well, but no joy.

EDIT 3: For clarity, my question is "how do I get my external thunderbolt SSD to mount at boot?"

Answer 1

Victory! I have a solution and a workaround.

The workaround:

This article inspired me to remove the mount from my fstab and instead add a mount command to my /etc/rc.local.

mount -t ext4 -o rw,noatime -U cb9940d0-5b0b-4273-97d7-b4dd29768926 /opt

This does the job, but it's not a good solution because the drive's content can't be used by anything during startup.

For instance, if I place my docker images on the external partition, I'm assuming that docker will launch before rc.local, try to find autostart images on my external drive and fail.

The solution:

I created a unit mount file /etc/systemd/system/opt.mount:

# External thunderbolt SSD.  Thunderbolt drives require the boltd daemon which is launched by udev.

[Unit]
After=systemd-udevd.service
Requires=systemd-fsck@dev-disk-by\x2duuid-cb9940d0\x2d5b0b\x2d4273\x2d97d7\x2db4dd29768926.service
Requires=systemd-udevd.service
After=systemd-fsck@dev-disk-by\x2duuid-cb9940d0\x2d5b0b\x2d4273\x2d97d7\x2db4dd29768926.service
After=blockdev@dev-disk-by\x2duuid-cb9940d0\x2d5b0b\x2d4273\x2d97d7\x2db4dd29768926.target

[Mount]
Where=/opt
What=/dev/disk/by-uuid/cb9940d0-5b0b-4273-97d7-b4dd29768926
Type=ext4
Options=rw,relatime,nofail

[Install]
WantedBy=multi-user.target

I believe the Requires=systemd-udevd.service line is the trick. I don't know why the x-systemd.requires option in the fstab didn't work.

This unit file needs to be enabled using this command:

sudo systemctl enable opt.mount

Now when I reboot, my thunderbolt SSD is mounted!