Autofs fails to mount nfs-shared autofs bind mounted directory, but manual mount succeeds

Question

Apologies for the long read below. Hopefully this will give enough context and traces so someone can help me.

Context

I store user home directories on some zfs dataset (one per user), where each user has one different home directory per Linux distribution and version (e.g. Debian Buster), but where XDG directories (e.g., Downloads or Desktop) of all these homes actually points to the same directory so that the user has access to all its files regardless of the Linux distro run by the terminal the user logged on. I have the same kind of structure for binaries stored in home. See the example below (see below for meaning of numbers in square brackets):

/media/zfs/home/user1/XDG-DIRS/[1]Downloads
                     |        /[2]Videos
                     |        /<some more directories and files>
                     |
                     /homes/[3]debian_buster/<some files(1)>
                     |     /[4]ubuntu_groovy/<some files(2)>
                     |
                     /[5]usr/share/<some files(3)>
                     |   /include/<some files(4)>
                     |
                     /local/linux-x86_64/[6]bin
                           |            /[7]lib
                           |
                           /linux-armhf/[8]bin
                                       /[9]lib

Using specific mounts to correct points in the structure above, one can reconstitute a home directory with all user setting files matching the desktop environment served by the distro and its version, but still access all personal files and even home-installed binaries compatible with the OS and architecture the user logged on. For example, a user logging on a Linux Debian Buster running on a x86_64 architecture, should have its home directory structure as below. A number in square bracket at the end of a directory names means the content of that directory should actually be content in the directory with the same number at the beginning of its name in the structure above. I let you guess what the home directory of a user logged on a raspberry pie running ubuntu groovy looks like.

/home/AD.EXAMPLE.COM/user1[3]/Downloads[1]
                             /Videos[2]
                             /.local[5]/bin[6]
                                       /lib[7]

One can achieve this by running autofs on the terminal computer and using an executable map file that results in as many nfs mounts as necessary (5 per user in the example above, typically more because of more XDG directories). The problem I anticipate with this is that if the user wants to move a files from "Downloads" to "Videos", then because these are two different nfs mount points, the files is actually downloaded to the terminal, then uploaded back to the server. I actually did not test if that results in performance penalties. If you have any insight about this point, please let me know.

In order to limit the performance problem described above, I actually reconstitute home directories for each distro/version (on one side) and for each os/architecture (on the other side) on the server using autofs and then exporting the results through NFS. That means I build the following structure on the server using autofs bind mounts

/media/user_data/unix/user1/home/[10]debian_buster/Downloads[1]
                           |    |                 /Videos[2]
                           |    |                 /.local/<empty>
                           |    /ubuntu_groovy/Downloads[1]
                           |                  /Videos[2]
                           |                  /.local/<empty>
                           /[11]local/linux-x86_64[5]/bin[6]
                           |                         /lib[7]
                           /local/linux-armhf[5]/bin[8]
                                                /lib[9]

/etc/auto.master.d/user_data.autofs (server side)

/media/user_data/unix/  /etc/auto.AD.EXAMPLE.COM.unix       --ghost --timeout=120

/etc/auto.AD.EXAMPLE.COM.unix (server side, executable and read bits set for ugo)

#!/bin/bash

key=$1

echo '- /home   -fstype=bind            :/media/zfs/home/'$key'/unix/ browse \'
for i in $(ls /media/zfs/home/$key/unix) 
do
    for j in $(ls /media/zfs/home/$key/XDG_DIRS)
    do
        echo '  /home/'$i'/'$j' -fstype=bind            :/media/zfs/home/'$key'/XDG_DIRS/'$j' browse \'
    done
done

for i in $(ls /media/zfs/home/$key/usr) 
do
    echo '  /local/'$i' -fstype=bind            :/media/zfs/home/'$key'/local/ browse \'
    for j in $(ls /media/zfs/home/$key/usr/$i)
    do
        echo '  /local/'$i'/'$j'    -fstype=bind            :/media/zfs/home/'$key'/usr/'$i'/'$j' browse \'
    done
done
echo ''

Here is a sample output of the script shown above

root@server:~# /etc/auto.AD.EXAMPLE.COM.unix user1
-   /home   -fstype=bind            :/media/zfs/home/user1/unix/ browse \
    /home/debian_buster/Downloads   -fstype=bind            :/media/zfs/home/user1/XDG_DIRS/Downloads browse \
    /home/debian_buster/Videos  -fstype=bind            :/media/zfs/home/user1/XDG_DIRS/Videos browse \
    /local/linux-armhf  -fstype=bind            :/media/zfs/home/user1/local/ browse \
    /local/linux-armhf/bin  -fstype=bind            :/media/zfs/home/user1/usr/linux-armhf/bin browse \
    /local/linux-armhf/lib  -fstype=bind            :/media/zfs/home/user1/usr/linux-armhf/lib browse \
    /local/linux-armhf/sbin -fstype=bind            :/media/zfs/home/user1/usr/linux-armhf/sbin browse \
    /local/linux-x86_64 -fstype=bind            :/media/zfs/home/user1/local/ browse \
    /local/linux-x86_64/bin -fstype=bind            :/media/zfs/home/user1/usr/linux-x86_64/bin browse \
    /local/linux-x86_64/lib -fstype=bind            :/media/zfs/home/user1/usr/linux-x86_64/lib browse \

root@server:~# 

This works flawlessly so far, although a tad slow to my taste. I export /media/user_data/unix through NFS, using the export file below:

# <other exports of unrelated directories>
/media/user_data    *(sec=krb5p,rw,crossmnt)
# <other exports of unrelated directories>

At this point, it is worth mentioning that one reason why the "user_data" step exists in this file hierarchy is that because if I export /media/user_data/unix in /etc/exportfs (or equivalently, /media/unix although not demonstrated below), then I get the warnings below. This is not encouraging but I try anyway with this extra user data layer in the hierarchy, hoping that the crossmnt will manage to also export what is mounted inside the hierarchy being exported. The system does not seem to complain about this attempt.

root@server:~# cat /etc/exports
# Other exports of unrelated directories
/media/user_data    *(sec=krb5p,rw,crossmnt)
/media/user_data/unix   *(sec=krb5p,rw,crossmnt)
# More unrelated exports

root@server:~# exportfs -ra; zfs share -a
exportfs: /etc/exports [5]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/media/user_data".
  Assuming default behaviour ('no_subtree_check').
  NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: /media/user_data/unix does not support NFS export
root@server:~# showmount -e
Export list for server:
/media/user_data      *
/media/user_data/unix *
# <nfs exports of unrelated zfs datasets>
root@server:~# 

In the following text, I removed export of /media/user_data/unix in /etc/exports and ran exportfs -ra and zfs share -a so that the NFS server does not complain about any directory that does not support NFS exports. Finally, autofs on the terminal computer only need to mount the home directory corresponding to the distro it runs and its version, as well as the local subdirectory corresponding to the OS and architecture, resulting in the following hierarchy for user1 on a Linux Debian Buster x86_64.

/home/AD.EXAMPLE.COM/user1[10]/Downloads
                              /Videos
                              /.local[11]

Which I attempt to achieve with the autofs configuration below

/etc/auto.master.d/home.autofs (terminal side, Linux Debian Buster, x86_64)

/media/AD.EXAMPLE.COM  /etc/auto.AD.EXAMPLE.COM.home    --timeout=120

/etc/auto.AD.EXAMPLE.COM.home (terminal side, Linux Debian Buster, x86_64, executable and read bits set for ugo)

#!/bin/bash

key=$1

distributor()
{
    lsb_release -i | cut -f 2 -d : | xargs echo | tr '[:upper:]' '[:lower:]'
}

codename()
{
    lsb_release -c | cut -f 2 -d : | xargs echo | tr '[:upper:]' '[:lower:]'
}

architecture()
{
    uname -m | tr '[:upper:]' '[:lower:]'
}

os()
{
    uname -s | tr '[:upper:]' '[:lower:]'
}

echo '- /   -fstype=nfs,vers=4.2,sec=krb5p,fsc  server.example.com:/media/user_data/unix/'$key'/home/'$(distributor)'_'$(codename)' \'
echo '  /.local -fstype=nfs,vers=4.2,sec=krb5p,fsc  server.example.com:/media/user_data/local/'$key'/local/'$(os)'-'$(architecture)' \'
echo ''

Here is a sample output of the script above

root@terminal:~$ /etc/auto.AD.EXAMPLE.COM.exp user1
-   /   -fstype=nfs,vers=4.2,sec=krb5p,fsc  server.example.com:/media/user_data/unix/user1/home/debian_buster \
    /.local -fstype=nfs,vers=4.2,sec=krb5p,fsc  server.example.com:/media/user_data/local/user1/local/linux-x86_64 \

root@terminal:~$ 

The problem

Autofs on the terminal computer fails to mount the reconstituted home directory exported from the server. This is the trace I get from automount when trying to list the content of /home/AD.EXAMPLE.COM/user1:

root@terminal:~# automount -d -f -v 
... <lots of output>
get_nfs_info: called with host server.example.com(192.168.80.101) proto 17 version 0x30
get_nfs_info: nfs v3 rpc ping time: 0.000000
get_nfs_info: host server.example.com cost 0 weight 0
prune_host_list: selected subset of hosts that support NFS3 over TCP
mount_mount: mount(nfs): calling mkdir_path /media/AD.EXAMPLE.COM/user1
mount_mount: mount(nfs): calling mount -t nfs -s -o vers=4.2,sec=krb5p,fsc server.example.com:/media/user_data/unix/user1/home/debian_buster /media/AD.EXAMPLE.COM/user1
>> mount.nfs: mounting server.example.com:/media/user_data/unix/user1/home/debian_buster failed, reason given by server: No such file or directory
mount(nfs): nfs: mount failure server.example.com:/media/user_data/unix/user1/home/debian_buster on /media/AD.EXAMPLE.COM/user1
do_mount_autofs_offset: mount offset /media/AD.EXAMPLE.COM/user1/.local at /media/AD.EXAMPLE.COM/user1
mount_autofs_offset: calling mount -t autofs -s  -o fd=16,pgrp=20379,minproto=5,maxproto=5,offset automount /media/AD.EXAMPLE.COM/user1/.local
mounted offset on /media/AD.EXAMPLE.COM/user1/.local with timeout 120, freq 30 seconds
mount_autofs_offset: mounted trigger /media/AD.EXAMPLE.COM/user1/.local at /media/AD.EXAMPLE.COM/user1/.local
dev_ioctl_send_ready: token = 114
mounted /media/AD.EXAMPLE.COM/user1

And the listed content of /home/AD.EXAMPLE.COM/user1 is nothing:

root@terminal:~$ ls /home/AD.EXAMPLE.COM/user1
root@terminal:~$ 

Although the supposedly mounted directory from the server is full of files:

root@server:~# ls /media/user_data/unix/user1/home/debian_buster
 file1   file2   file3
root@server:~# 

The automount trace above hints that the directory attempted to mount does not exist on the server, but this is strange, first because trying to list that exact directory from the server does show that is exists (see above), and I can mount this directory manually from the terminal anyway, as shows the trace below:

root@terminal:~$ mount -vvvv -t nfs server.example.com:/media/user_data/unix/user1/home/debian_buster /mnt
mount.nfs: timeout set for Sat Feb 13 22:37:06 2021
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.80.101,clientaddr=192.168.104.1'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'addr=192.168.80.101'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.80.101 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.80.101 prog 100005 vers 3 prot UDP port 39874
root@terminal:~$ ls /mnt
 file1   file2   file3
root@terminal:~$ 

My attempt for a solution

I thought that autofs on the terminal may not find the directory to mount because it is not mounted (yet) on the server, therefore I attempted to use --ghost and browser options (you can see them in the server's /etc/auto.master.d/media.autofs and /etc/auto.AD.EXAMPLE.COM.unix files shown above), but to no avail. I am running out of ideas to explore to find a permanent solution.

Temporary workaround

The temporary workaround I use at the moment is not not use autofs on the server side, but instead bind mount manually all directories to obtain the correct file hierarchy to export. I am not too satisfied with this solution as it requires a lot of mounts to be permanently active and it seems to leave the server in a somewhat unstable state, though I don't know why exactly.

Remarks

• Both the server and the terminal run Debian Buster (Linux x86_64) in my tests and that produced the traces above.
• NFS complaining that the autofs-reconstituted directory does not support NFS export hints that I should not try to export it at all by sneakingly exporting its parent directory instead. I could not find any reference stating that it is not possible to NFS-export a directory having a autofs-mounted subdirectory, so it is still worth a try. Furthermore, it work fine when I mount --bind these subdirectories manually on the server side instead of using autofs so there should be some hope.
• This is a rather complex (and fragile?) setup; if you have a simpler (and more robust) suggestion to achieve the same functionalities, I am also interested :)

Answer 1

I seems your client is trying NFSv4 first by default. When mounting manually it falls back to NFSv3. Autofs most propably gives up on the first failure.

You might be able to tune the default client-behaviour to use NFSv3 first.