docker: bridge network: sporadically loosing packets

Question

Hope you're doing well.

I'm struggling since 2 days with an connection problem between host system and container. It appears I randomly loose packets in 'transit' to the container (47 out of 100).

I can see the packets 'leaving' the host interface docker0 but sometimes they never arrive in the container. This is reproduceable regardless of the used image (different software not version).

I'm really thankful for any pointers since I'm clueless right now. Thanks for your time!

Container

tcpdump host 172.17.0.6 and icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:34:22.406364 IP 172.17.0.1 > files.local: ICMP echo request, id 19794, seq 1, length 64
13:34:22.406397 IP files.local > 172.17.0.1: ICMP echo reply, id 19794, seq 1, length 64
>> missing seq# 2 <<
13:34:24.451683 IP 172.17.0.1 > files.local: ICMP echo request, id 19794, seq 3, length 64
13:34:24.451721 IP files.local > 172.17.0.1: ICMP echo reply, id 19794, seq 3, length 64
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

Host

Debian GNU/Linux 10 (buster)
Kernel 4.19.0-10-amd64 (Debian 4.19.132-1)
Docker version 19.03.12, build 48a66213fe

ping 172.17.0.6 -c3

PING 172.17.0.6 (172.17.0.6) 56(84) bytes of data.
64 bytes from 172.17.0.6: icmp_seq=1 ttl=64 time=0.111 ms
>> missing seq# 2 <<
64 bytes from 172.17.0.6: icmp_seq=3 ttl=64 time=0.097 ms

--- 172.17.0.6 ping statistics ---
3 packets transmitted, 2 received, 33.3333% packet loss, time 47ms
rtt min/avg/max/mdev = 0.097/0.104/0.111/0.007 ms

tcpdump -i docker0 host 172.17.0.6 and icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:34:22.406332 IP 172.17.0.1 > 172.17.0.6: ICMP echo request, id 19794, seq 1, length 64
15:34:22.406401 IP 172.17.0.6 > 172.17.0.1: ICMP echo reply, id 19794, seq 1, length 64
15:34:23.427549 IP 172.17.0.1 > 172.17.0.6: ICMP echo request, id 19794, seq 2, length 64
15:34:24.451657 IP 172.17.0.1 > 172.17.0.6: ICMP echo request, id 19794, seq 3, length 64
15:34:24.451725 IP 172.17.0.6 > 172.17.0.1: ICMP echo reply, id 19794, seq 3, length 64
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel

docker network inspect bridge

[
    {
        "Name": "bridge",
        "Id": "fbd2aea6a1c634c95ea3e0ac628daf0c266f77cdda63edc573d978d142c57ed8",
        "Created": "2020-08-21T22:13:41.905418474+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "90f16aa06342932ae2bb258c0f1b67a833db3de8142336dbec1cda4468ddf76e": {
                "Name": "container-test",
                "EndpointID": "6e3b17d55b1836d6a1a12219011ceb31950c0fc421fb17923a983eea3c2d559a",
                "MacAddress": "02:42:ac:11:00:07",
                "IPv4Address": "172.17.0.6/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Update 1

I created a new bridge network with minimal options and the problem seems to be solved for containers running in the new network.

docker network create --subnet=172.20.0.0/24 --gateway=172.20.0.1 docker20

The only difference seem to be the associated options. While the new one has none the old one has:

        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },

Will do some more investigating.

Answer 1

TL;DR Do not use the default bridge network, always create a user defined one.

After completely redoing the network over night everything works as expected. I can only assume that it went sideways after some time due to the differences between user-defined bridges and the default bridge.

Here are my resolving steps:

1. Stop all containers docker stop <container>
2. Remove all containers from bridge: docker network disconnect bridge <container>
3. Stop the daemon: systemctl stop docker
4. Create daemon.json: vim /etc/docker/daemon.json

{
        "bridge": "none"
}

5. Start docker systemctl start docker
6. Create new network: docker network create --subnet=172.17.0.0/24 --gateway=172.17.0.1 -o "com.docker.network.bridge.name=docker1" docker1
7. I was unsuccessful in using docker network connect --ip=<ip> docker1 <container> to reconnect my containers, therefore I redeployed them. --ip optionally sets the IP to use by the container.

An alternative to step 4 is starting the docker daemon with --bridge=none.

There is one minor downside: When containers are deployed they need to be put to the network specifically (--net <network>). Theoretically this can be avoided by changing the daemon.json to { "bridge": "docker1" } after step 6 and restarting the daemon, but I encountered issues with "overlapping IP's" although I do not have other hosts in 172.17.0.0/16. For reference the manual for the Daemon configuration file which also explains the CLI arguments.

If you are left with the old docker0 bridge and want to remove it you can issue ip link delete docker0.