Mounting docker into container shows executable but errors with: /usr/bin/docker: No such file or directory

Question

Enviornment details

$ docker --version

Docker version 19.03.4, build 9013bf583a

$ hostnamectl

   Static hostname: ohpc.novalocal
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 93f219319dd5bdb42d9f1c8f2e23d329
           Boot ID: c957802410104779a1e1e4c6ff52800c
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-957.12.2.el7.x86_64
      Architecture: x86-64

$ cat /etc/centos-release

CentOS Linux release 7.7.1908 (Core)

Problem

I have things successfully running on Ubuntu and appear to have boiled down the difference, but I'm uncertain how to resolve as it almost seems like a red herring. The following is from the host machine (Centos) that I'm having issues with:

$ ls -Z /usr/bin/docker
-rwxr-xr-x. root root system_u:object_r:container_runtime_exec_t:s0 /usr/bin/docker

I've been mounting the docker executable (and .sock) into a Jenkins container on my Ubuntu box so that it can execute docker commands. Snippet from my compose yml, note I've tried the equivalent docker run command, docker-compose up and docker stack deploy, all without success.

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
    privileged: true

Even though it's visible, and executable in the container, I'm unable to execute it:

$ docker exec -it tmp_jenkins_1 bash
bash-4.4$ which docker
/usr/bin/docker
bash-4.4$ ls -al /usr/bin/docker
-rwxr-xr-x 1 root root 88960560 Oct 18 15:53 /usr/bin/docker
bash-4.4$ docker --version
bash: /usr/bin/docker: No such file or directory
bash-4.4$ /usr/bin/docker --version
bash: /usr/bin/docker: No such file or directory

The only difference I've seen between the working environment (Ubuntu) environment and the failing environment (CentOS) is the . at the end of the file permissions (selinux acl policy). However, I've attempted setenforce Permissive and this did not change any observations, so I suspect it's not SELinux.

The docker daemon.json is non existent on the working box. There is an mtu:1200 and a special dns in the daemon.json on the centos box.

Logs in /var/log/messages show nothing useful.

Edit - 11 - 6 - 2019

Some additional things I've come across and attempted:

I attempted disabling SELinux altogether which still resulted in the no file found error above (enabled again afterwards).

Played around with the daemon.json, adding selinux-enabled": false and selinux-enabled": true inspired by this post on getting SELinux to place nice with the /etc and /usr host directories

When enabled I get errors from SELinux at runtime stating /usr/bin/docker labels cannot be rewritten for various enumerations of z and Z attributes tacked on at the end of the volume statements (note, attempted both docker run syntax and docker-compose.yml syntax changes -- I'd expect z as we do not want to lock the docker.sock or docker executable to only be accessible within the target container.

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:z
      - /usr/bin/docker:/usr/bin/docker:z
    privileged: true

I've also attempted using docker run --security-opt label:disable ... both with as well as an alternative to privileged to get around SELinux policies. I noticed RedHat documentation had some information on topic, but interestingly there was no docker_connect_any flag.

Important Booleans and Other Restrictions - "privileged" under docker is not really privileged. Even privileged docker processes cannot access arbitrary socket files. An SElinux Boolean, docker_connect_any, makes it possible for privileged docker processes to access arbitrary socket files. Even if run privileged, docker is restricted by the Booleans that are in effect.

Last thoughts, I've come across some helpful articles and documentation on the topic of everything, but still falling short in a solution.

docker inspect -f '{{ .ProcessLabel }}' tmp_jenkins_1 throughout all attempts has not given any labels, I'd expect after enabling selinux-enabled in the daemon.json that I'd see some SELinux syntax in the ProcessLabel, but have not.

Answer 1

It turns out SELinux was in fact a red herring... I changed the image up from jenkinsci/blueocean to jenkinsci/jenkins and things worked as expected (with SELinux enabled)

Now for the reasoning:

jenkinsci/blueocean (at least at the time of writing) apparently installs docker in the image.

That is, you cannot mount an executable over an executable and expect things to work as volumes should not overwrite if content already exists. I'm guessing this is a combination of Host OS binaries + Docker Image on why it worked on one host OS and not another.

In either route, it's because the executable already existed in the image and thus container.

Was going to delete, but generalized my answer a bit as I could see the reasoning for the weird case being something others might experience in a more broad sense.