Docker container can't connect other containers on same host with published ports using outer address

Question

I have Linux Mint host with docker(20.10.12).

On this host there are two containers, created with simple docker run, no configuration of networking was provided.:

• mysql, with published port 3306,
• grafana, with published port 3000.

Both containers accessible from outside (including from another host in the same network). But I cannot connect to msql from grafana's container.

For connection I use host's network IP (192.168.1.151).

nc -zv 192.168.1.151 3306 from inside of grafana's container "hangs up" and doesn't provide any output. Strangely nc -zv 192.168.1.151 3000 also "hangs up".

Checked with busybox, same result - nc also "hangs up".

Checked general networking issues: successfully connected to web-interface of router and google.com from within the container.

Also checked that from within container I successfully connected to open port of host itself.

I know that I can link both containers to same docker network, and skip publish port altogether. But I'm more interested in explanation why this happens.

Answer 1

When you launch Docker containers without a custom bridge network setup, Docker launches them in the default bridge network. The default bridge network has a different IP subnet and there's no way for it to route via the host IP address. Docker disables DNS resolution via the container name on the default bridge network as well. This is mentioned in their docs

Containers on the default bridge network can only access each other by IP addresses, unless you use the --link option, which is considered legacy. On a user-defined bridge network, containers can resolve each other by name or alias.

User-defined bridges provide automatic DNS resolution between containers.

The port publishing only makes it possible for applications on the host to connect the container.

The "IP address" here refers to the IP address of the container within the Docker default bridge network, not the host IP address. You can get the IP address from within the container via the ip a comnmand, if the packages are installed in the container, or from the Docker host by inspecting the container

docker inspect grafana | jq '.[].NetworkSettings.IPAddress'
"172.17.0.2"

docker inspect mysql | jq '.[].NetworkSettings.IPAddress'
"172.17.0.3"

Using these IPs, you can connect/verify connectivity:

docker exec -it grafana sh
/usr/share/grafana $ nc -vz 172.17.0.3 3306
172.17.0.3 (172.17.0.3:3306) open

If you need to connect the containers, you can create a custom network and join the containers.

docker network create monitoring
docker network connect monitoring mysql
docker network connect monitoring grafana

Now when you inspect the network you will see that both containers are on the network.

docker network inspect monitoring | jq '.[].Containers'
{
  "9b1cd51b1350769b0842ce901c732482024e6f3f51e0c1b30ece8d6f19793079": {
    "Name": "grafana",
    "EndpointID": "8f881091c298bc18d92a463bcca10dce7f0813c7a5d9232b4c693644443644ac",
    "MacAddress": "02:42:ac:12:00:03",
    "IPv4Address": "172.18.0.3/16",
    "IPv6Address": ""
  },
  "d7b7d4bc66f617f824509e013b7e5bfb283652d85eb041ca04ced3ae8f8fe44c": {
    "Name": "mysql",
    "EndpointID": "99b514fa927601d98a2da10f820329add905a6d47a02e81322607a445617a320",
    "MacAddress": "02:42:ac:12:00:02",
    "IPv4Address": "172.18.0.2/16",
    "IPv6Address": ""
  }
}

And you can connect to them via their container name.

docker exec -it grafana sh
/usr/share/grafana $ nc -vz mysql 3306
mysql (172.18.0.2:3306) open