Docker containers cannot connect to internet in Fedora 32?

Question

The Problem

I have recently created a docker container that has to pull a public github repository, however, it isn't able to resolve the host github.com. In fact, it isn't able to even execute ping command.

NOTE: all network operations here are inside the container after building, when the container runs

ping www.google.com gives cannot resolve host

It cannot connect to the internet at all.

After looking on the internet and Docker Documentation

Solutions Tried

1) Enabling IP Forwarding as given in Docker Documentation

I tried enabling IPForwarding but to no avail.

Contents of /usr/lib/systemd/network/80-container-host0.network after enabling IPForwarding

...
[Network]
DHCP=yes
LinkLocalAddressing=yes
LLDP=yes
EmitLLDP=customer-bridge
IPForward=true // this line was changed
[DHCP]
UseTimezone=yes

2) Recreating Docker Bridge as given in Stack Overflow

pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
sudo service restart docker

This was originally effective in this issue

There is a solution for ubuntu, but my network configuration doesn't have that problematic line

my sudo pico /etc/NetworkManager/NetworkManager.conf is just filled with comments

[main]
#plugins=ifcfg-rh


[logging]
#level=TRACE
#domains=ALL

3) Installing IP-Tables service StackOverflow

sudo yum install iptables-services
sudo service docker restart

4) Overriding DNS configuration in docker-compose-StackOverflow

version: "3.3"
services:
    airflow:
        build: 
            context: ./airflow
            dockerfile: Dockerfile
        ports: 
            - 8080:8080
        environment: 
            GITHUB_DAG_REPO: https://github.com/siddharths067/HelloAirflow.git
        dns:
            - 8.8.8.8
            - 8.8.4.4

The Docker Network Inspect of the Image's Network

I don't know if this would be of any use

docker network inspect airflowsetup_default 
[
    {
        "Name": "airflowsetup_default",
        "Id": "141a518c1440e603f75774c54f42de33e9173e3f062a0a0bc772db13a7f1ef5d",
        "Created": "2020-08-30T14:42:30.951975699+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "airflowsetup",
            "com.docker.compose.version": "1.25.4"
        }
    }
]

Output

airflow_1  | GITHUB DAG REPO IS
airflow_1  | https://github.com/siddharths067/HelloAirflow.git
airflow_1  | Cloning into 'HelloAirflow'...
airflow_1  | fatal: unable to access 'https://github.com/siddharths067/HelloAirflow.git/': Could not resolve host: github.com

Answer 1

Event though the OP found a workaround for his problem and closed this issue, he did not find the root issue. The fact that Docker's default bridged network bridge is connected to the internet and airflowsetup_default is not suggests that something is wrong with Docker networking setup.

I did some research and it turns Fedora 32 decided it doesn't really care if Docker works on it or not.

It's not even possible to install Docker in a way described in the documentation and if you install the package provided by Fedora it still doesn't work correctly - more information on that issue can be found here, here and here.

The main issue is that there is no internet connectivity inside the containers if they are connected to any custom bridged network - whether it was created using docker network create or by docker-compose.

The reason why is simple - Docker assumes that the firewall used by the OS is iptables, but Fedora 32 uses firewalld by default. That means Docker cannot configure the firewall manually - it has to be configured by hand.

For reference purposes I will first describe how to set up Docker on a clean Fedora 32 installation.

First run the following commands:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
sudo groupadd docker
sudo usermod -aG docker $USER

This will configure cgroups to be compatible with Docker daemon and allow your user to use Docker CLI without sudo.

Next restart your system to apply changes and run:

sudo dnf install -y moby-engine docker-compose nano
sudo systemctl enable docker
sudo systemctl start docker

to install and enable Docker.

Now go to the directory that contains your docker-compose.yml and run docker-compose up -d. Yuo should see docker-compose creating a network for you and then creating the container. If your container requires internet connection on startup (like the OP's) it will fail to start.

Now run sudo iptables-save | grep DOCKER and you should see something like:

:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o br-b56fa303f315 -j DOCKER
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-b56fa303f315 ! -o br-b56fa303f315 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-b56fa303f315 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-b56fa303f315 -j RETURN

docker0 is Docker's default bridged network, while br-b56fa303f315 is the new network created by docker-compose (yours may have a different name). If your OS were using iptables everything would work as expected, but it does not, so we need to check the configuration of docker0 in firewalld.

Run firewall-cmd --get-active-zones and you'll get something similar to:

docker
  interfaces: docker0
public
  interfaces: eth0 eth1

You can see that bridge network is in docker zone, but the new network is not. In fact it's not listed at all, which means it's in the default zone. You can check what that is by running firewall-cmd --get-default-zone. On a clean install of Fedora 32 it's public.

So run (remember to replace br-b56fa303f315 with your interface name):

sudo firewall-cmd --zone=docker --add-interface=br-b56fa303f315

run docker-compose up -d if your service failed to start previously and voila - your container has network connectivity.

Unfortunately if your restart your system it'll lose that connectivity again.

You can prevent that by using:

sudo firewall-cmd --permanent --zone=docker --add-interface=br-b56fa303f315
sudo firewall-cmd --reload

However if you create any new networks or recreate existing ones (for example by running docker-compose down and then docker-compose up -d again) you'll have to repeat the process.

So what is the solution to this problem?

First, write down all network interfaces that are currently attached to the default zone - in this example eth0 and eth1.

Then run the following (replacing public with your default zone's name)

sudo firewall-cmd --set-default-zone=docker
sudo firewall-cmd --permanent --zone=public --add-interface=eth0
sudo firewall-cmd --permanent --zone=public --add-interface=eth1
sudo firewall-cmd --reload

Now the interfaces that were previously in your default zone should be there again, but all the new interfaces (and therefore all the new Docker networks) will be automatically added to docker zone, which will give them full network connectivity.

Answer 2

I followed an alternative solution from the Fedora Magazine:

Whitelist docker in firewall

To allow Docker to have network access, two commands are needed.

sudo firewall-cmd --permanent --zone=trusted --add-interface=docker0
sudo firewall-cmd --permanent --zone=FedoraWorkstation
--add-masquerade

The first command will add the Docker-interface to the trusted environment which allows Docker to make remote connections. The second command will allow docker to make local connections. This is particularly useful when multiple Docker containers are in as a development environment.

Then you should run this command to validate the changes

sudo firewall-cmd --reload

Normally, you don't have to reboot your computer and your next containers are going to be connected.

In case, it does not work, you could try to:

1. First, restart a new container
2. Second, restart docker: sudo systemctl restart docker
3. Finally, reboot your computer

Answer 3

Adding network-mode: bridge to my docker-compose file seems to work. But I feel this might not be the proper way to do this. I shouldn't have to over-ride anything to allow my containers to connect.

version: "3.3"
services:
    airflow:
        build: 
            context: ./airflow
            dockerfile: Dockerfile
        ports: 
            - 8080:8080
        environment: 
            GITHUB_DAG_REPO: https://github.com/siddharths067/HelloAirflow.git
        # dns:
        #     - 8.8.8.8
        #     - 8.8.4.4
        network_mode: bridge