I'm not even sure that's the right wording, I'll try to explain.
I want to setup a small VPN are to host some private projects as well as access internet. I also want to have my own DNS server to have cool top level domain names.
How does my setup look
I am using two Docker images:
- ubuntu/bind9 for DNS
- donaldzou/wgdashboard (https://github.com/donaldzou/WGDashboard) for Wireguard and GUI dashboard
My wireguard interface has name main
and runs on port 20000
. Just a heads up because these aren't "default" settings.
Now, it almost works meaning I can connect several peers to this "server" peer. I can ping any peer from another peer. So the "wireguard" part of the setup works. I believe it is a routing issue that's outside wireguard scope.
Here's relevant parts of my docker-compose.yml
file:
services:
dns:
image: ubuntu/bind9
restart: unless-stopped
networks:
main:
ipv4_address: 172.16.0.53
wgdashboard:
image: donaldzou/wgdashboard
cap_add:
- NET_ADMIN
- SYS_MODULE
networks:
main:
ipv4_address: 172.16.0.2
ports:
- 10086:10086
- 20000:20000/udp
networks:
main:
ipam:
driver: default
config:
- subnet: 172.16.0.0/16
ip_range: 172.16.0.0/24
gateway: 172.16.0.1
Network is created and both containers have their correct IPs
So as my setup runs on UDP port 20000, and not on the host
network, I have some rules in iptables. This part I'm super unsure about.
# iptables -L | grep 20000
ACCEPT udp -- anywhere 172.16.0.2 udp dpt:20000
ACCEPT udp -- anywhere anywhere udp dpt:20000
# iptables -t nat -L | grep 20000
MASQUERADE udp -- 172.16.0.2 172.16.0.2 udp dpt:20000
DNAT udp -- anywhere anywhere udp dpt:20000 to:172.16.0.2:20000
I've also seen some people requested this network info, so I'm posting it here:
$ ip -br link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
ens3 UP 52:54:00:0c:f3:c5 <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0 DOWN 02:42:61:07:a5:d4 <NO-CARRIER,BROADCAST,MULTICAST,UP>
br-e12a65bf943c UP 02:42:5b:d4:1a:a6 <BROADCAST,MULTICAST,UP,LOWER_UP>
$ ip -br address
lo UNKNOWN 127.0.0.1/8 ::1/128
ens3 UP XX.YY.ZZ.24/24
docker0 DOWN 172.17.0.1/16
br-e12a65bf943c UP 172.16.0.1/16
$ ip route
default via XX.YY.ZZ.1 dev ens3 onlink
XX.YY.ZZ.0/24 dev ens3 proto kernel scope link src XX.YY.ZZ.24
172.16.0.0/16 dev br-e12a65bf943c proto kernel scope link src 172.16.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
$ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
$
What's wrong
I can ping and query
172.16.0.53
(the "dns" container) from server that runs dockerI can ping and query
172.16.0.53
from "wgdashboard" containerI can ping and query
1.1.1.1
from server and both containersI can ping and open
http://172.16.0.2:10086/
("wgdashboard") container in browser from all connected peersI can NOT ping outside world from either peers
I can NOT ping or issue a DNS query to
172.16.0.53
from either peers
What I'm trying to do to troubleshoot this problem:
port mapping
first, I added port mapping (53:53/udp
) to dns
service in docker-compose.yml:
$ docker-compose up
[+] Running 3/3
⠿ Network apps_main Created 0.1s
⠿ Container apps-dns-1 Created 0.1s
⠿ Container apps-wgdashboard-1 Created 0.1s
Attaching to apps-dns-1, apps-wgdashboard-1
Error response from daemon: driver failed programming external connectivity on endpoint apps-dns-1 (4d28f6ed5232a08b266aed9fd232bc9894b7ae0990d25129b27d4f4112ef289f): Error starting userland proxy: listen udp4 0.0.0.0:53: bind: address already in use
uh what, does it try to bind to port 53 of host machine? Anyway I shouldn't need this anyway, because I only plan to query this server from inside of wireguard (and therefore, docker) network and don't need it from outside?
tcpdump
I ran tcpdump from four locations:
- host machine, interface
br-e12a65bf943c
(docker-compose network),tcpdump -i br-e12a65bf943c icmp
- host machine, interface
ens3
(actual internet),tcpdump -i ens3 icmp
- wireguard container, interface
main
(like wg0 but i gave it the different name),tcpdump -i main icmp
- bind9 container, interface
eth0
(it's the only one there)
Pinging dns
container from connected peer in chronological order (I don't know what's up with these timezones, ignore hours):
[wireguard,main] 16:18:22.021439 IP 10.0.0.2 > 172.16.0.53: ICMP echo request
[host,br-e12a65bf943c]17:18:22.021500 IP 10.0.0.2 > 172.16.0.53: ICMP echo request
[dns,eth0] 14:18:22.021508 IP 10.0.0.2 > 172.16.0.53: ICMP echo request
[dns,eth0] 14:18:22.021567 IP 172.16.0.53 > 10.0.0.2: ICMP echo reply
[host,br-e12a65bf943c]17:18:22.021575 IP 172.16.0.53 > 10.0.0.2: ICMP echo reply
[host,ens3] 17:18:22.021584 IP 172.16.0.53 > 10.0.0.2: ICMP echo reply
takeaway: three requests,three replies, but one reply decided to go through ens3
instead of returning to wireguard
Pinging 1.1.1.1
[wireguard,main] 16:30:16.689460 IP 10.0.0.2 > 1.1.1.1: ICMP echo request
[host,br-e12a65bf943c]17:30:16.689570 IP 10.0.0.2 > 1.1.1.1: ICMP echo request
host machine, ens3
has nothing (!!!)
takeaway: I can't ping outside world from 10.0.0.0/16 network
So, I have three questions
- what happened with ICMP reply at
br-e12a65bf943c
? How to make it go back to wireguard interface? - what happened with ICMP request on its way to 1.1.1.1?
- is it iptables
If you've read so far, thank you for your time
Solution, I guess?
Thanks for answering! You're rignt, I forgot to mention the network mask, I selected /16 in this case.
Indeed, adding this route on dns
container fixed pinging
ip route add 10.0.0.0/16 via 172.16.0.2
Now my task is to make it "permanent", I guess I need to add this to dockerfile or something.
Second issue is external traffic.
I added this rule to wireguard
container:
iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -o eth0 -j MASQUERADE
And yeah it looks like it's working