Routing container to wireguard container for a certain subnet not working as expected

Question

! See updated post at the end !

I am trying to run Wireguard in a container using docker compose on a remote host in a site-to-site configuration with my intranet at home which works flawlessly on its own.

Wireguard configuration

wg0.conf

[Interface]
PrivateKey = ***
Address = 10.6.0.4/24
ListenPort = 12345
DNS = 208.67.222.222, 208.67.220.220
PostUp = iptables -t nat -A POSTROUTING -o wg+ -j MASQUERADE
PreDown = iptables -t nat -D POSTROUTING -o wg+ -j MASQUERADE

[Peer]
PublicKey = LXSxUv5lp9A2WOz5mV33GQa5jpJYJ04j4Rl6FWlnczA=
PresharedKey = ***
Endpoint = vpn.example.com:12345
AllowedIPs = 192.168.178.0/24, 10.6.0.0/24

Docker compose file

docker-compose.yml

version: '3.*'

networks:
  outside:
    external: true

  wireguard_ghf68:
    internal: true
    driver: "bridge"
    ipam:
      config:
        - subnet: 10.7.3.0/24

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard

    cap_add:
      - NET_ADMIN
      - SYS_MODULE

    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Berlin

    volumes:
      - './wireguard:/config'
      - '/lib/modules:/lib/modules:ro'

    networks:
      wireguard_ghf68:
       ipv4_address: 10.7.3.3
      outside: {}

    ports:
      - target: 51820
        published: 51820
        protocol: "udp"
        mode: "host"

    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv4.ip_forward=1

    restart: unless-stopped

After a docker compose up I can attach a shell to the container and ping all the endpoints in my intranet (192.168.178.0/24).

root@wireguard:/# ping -c1 192.168.178.25
PING 192.168.178.25 (192.168.178.25) 56(84) bytes of data.
64 bytes from 192.168.178.25: icmp_seq=1 ttl=63 time=47.6 ms

--- 192.168.178.25 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 47.640/47.640/47.640/0.000 ms

Docker compose file with homeassistant

docker-compose.yml:

services:
  homeassistant:
    container_name: "homeassistant"
    image: "ghcr.io/home-assistant/home-assistant:stable"

    depends_on:
     - "wireguard"

    cap_add:
      - NET_ADMIN

    volumes:
      - ./homeassistant:/config
      - /etc/localtime:/etc/localtime:ro

    environment:
      - PUID=1000
      - PGID=1000
      - "TZ=Europe/Berlin"

    restart: unless-stopped
    networks:
      wireguard_ghf68:
        ipv4_address: 10.7.3.2

After docker compose up I can attach a shell to "homeassistant" and ping the wireguard container. But I also want to be able to ping my intranet (192.168.178.0/24) from there. So I create a new route inside the container:

ip -4 route add 192.168.178.0/24 via 10.7.3.3

which results in these routes:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.7.3.1        0.0.0.0         UG    0      0        0 eth0
10.7.3.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.178.0   10.7.3.3        255.255.255.0   UG    0      0        0 eth0

Now I theoretically should be able to ping my intranet but it does not work.

root@homeassistant:/# ping -c1 10.7.3.3
PING 10.7.3.3 (10.7.3.3): 56 data bytes
64 bytes from 10.7.3.3: seq=0 ttl=64 time=0.147 ms

--- 10.7.3.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.147/0.147/0.147 ms

root@homeassistant:/# ping -c1 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes

--- 192.168.178.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Docker Host

However if I create the same route on the docker host I am able to reach my intranet from that host but still not from within the "homeassistant" container.

root@dockerhost:~$ ip -4 route add 192.168.178.0/24 via 10.7.3.3
root@dockerhost:~$ ping -c1 192.168.178.25
PING 192.168.178.25 (192.168.178.25) 56(84) bytes of data.
64 bytes from 192.168.178.25: icmp_seq=1 ttl=62 time=43.8 ms

--- 192.168.178.25 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 43.826/43.826/43.826/0.000 ms

Questions

What am I doing wrong? Am I missing some routes or sysctl configurations or something in cap_add? Why is it working from the docker host but not from the within the container?

If you need more information please ask.

Update

I finally found the culprit. When docker creates a network it also creates these iptables rules:

-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
...
-A DOCKER-ISOLATION-STAGE-1 ! -s 10.7.3.0/24 -o br-a575507e42d8 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 10.7.3.0/24 -i br-a575507e42d8 -j DROP
...
-A DOCKER-USER -j RETURN

After removing the two DROP rules the routing works as expected.

I think the better way would be to add a few rules to the DOCKER-USER chain which accept the packets before they can be dropped.

If I find a elegant way to add these rules dynamically I will answer my question myself.

Answer 1

I will now answer my own question because I found a way which seems to be the best for me at the moment.

First I give the docker network a static bridge name, in this case d-wg (short for docker-wireguard). Keep in mind that there is a maximum length of 15 characters or 16 bytes including the null character (see if.h)

networks:
  wireguard:
    driver: "bridge"
    internal: true
    name: "wireguard"
    ipam:
      driver: "default"
      config:
        - subnet: 10.7.3.0/24
    driver_opts:
      com.docker.network.bridge.name: d-wg

Now a docker compose up will create a new docker network with the name wireguard and the interface name on kernel level will be d-wg:

$ docker network ls -f name=wireguard
NETWORK ID     NAME        DRIVER    SCOPE
648c3d39638b   wireguard   bridge    local

$ ip link show d-wg
289: d-wg: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:43:xx:yy:zz brd ff:ff:ff:ff:ff:ff

Now I can easily create two iptables rules in the DOCKER-USER chain which allow traffic for my two subnets 10.7.3.0/24 on the local side and 192.168.178.0/24 on the remote side. Then make sure the already existing rule -A DOCKER-USER -j RETURN goes after these again:

$ iptables -A DOCKER-USER -d 192.168.178.0/24 -i d-wg -o d-wg -j ACCEPT
$ iptables -A DOCKER-USER -d 10.7.3.0/24 -i d-wg -o d-wg -j ACCEPT
# Add a new return rule
$ iptables -A DOCKER-USER -j RETURN
# Delete the old one (deletes always the first if there are multiple rules)
$ iptables -D DOCKER-USER -j RETURN

Now I also have to create a new route inside my homeassistant container when it is starting up. Its original entrypoint is /init and I want to create the new route to the wireguard container before that. Therefore I create a small script init-script.sh which will be the new entrypoint and mount it into the container: init-script.sh

#!/bin/bash

ip route add 192.168.178.0/24 via 10.7.3.3
# 'exec' is necessary to not spawn a new process with an other PID than 1.
exec /init

And this is my new docker-compose.yml for homeassistant.

services:
  homeassistant:
    container_name: "homeassistant"
    image: "ghcr.io/home-assistant/home-assistant:stable"

    cap_add:
      - NET_ADMIN

    volumes:
      - ./homeassistant:/config
      - ./init-script.sh:/init-script.sh:ro
      - /etc/localtime:/etc/localtime:ro

    environment:
      - PUID=1000
      - PGID=1000
      - "TZ=Europe/Berlin"

    entrypoint:
      - /init-script.sh

    restart: unless-stopped

    networks:
      traefik:
      wireguard:
        ipv4_address: 10.7.3.2

An additional idea would be to create a new systemd service which creates the iptables rules automatically after the docker service starts. Maybe I will edit this answer as soon as I did this.