container mounting fails after moving Docker files (images) to AWS EFS NFS mounted volume

Question

Recently switched my AWS EC2 ubuntu 14.04 instance from using a ebs volume to the newish EFS. That all works great and I moved for example all my owncloud data there.

Now I wanted to move my docker files from var/lib/docker to my efs volume but when I do that and change the docker file in etc/defaults docker can't mount any container without this error.

root@aws-kebler:/mnt/efs/docker# docker restart $(docker ps -a -q)
Error response from daemon: Cannot restart container 3e5a2335837a: oci runtime error: exec: "/ui-for-docker": stat /ui-for-docker: no such file or directory

which kinda sounded like the image never got mounted so I just tried from scratch and got...

root@aws-kebler:/mnt/efs/docker# docker run -d -p 9001:9000 --name docker-ui --privileged -v /var/run/docker.sock:/var/run/docker.sock uifd/ui-for-docker
docker: Error response from daemon: error creating aufs mount to /mnt/efs/docker/aufs/mnt/11bd65bb4cf850a5267ba635c2a015d6f89b7ac5af2662d75aa4ca51d4401726-init: file too large.

This makes me think there is some "weirdness" with EFS volume that it isn't "expanding" when an image is created or attempts to load.

Just for grins I scrubbed the docker folder on the EFS volume and did the same run and got the same error about "file too large"

Just to be sure it's the efs volume I did the same but used my EBS volume and no problems at all....so yea it's something to do with the EFS volume

here is my mount command

mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).xxxxxxxx.efs.us-west-2.amazonaws.com:/ efs

Any ideas why this is happening or how to fix it.

Answer 1

Maybe similar, but in my case docker daemon (1.12) was started prior to the nfs mount being performed. Building a new machine at aws, it happens... anyway, no container new or otherwise could see any files in that folder, though they were clearly present on the docker host after mounting the nfs share. To test we tried a dumb container, same image, same bummer, no files.

Restarting docker daemon cleared the problem, because only at docker daemon startup does docker daemon itself enumerate the available drives (** and their sub folders ** ) on the host. Container start does not do that, it just asks docker daemon wuzzup.

hths.