1

I was recommended to move this to superuser from servervault, hopefully it's appropriate for here.

I'm a hobbyist / tinkerer using bhyve (in pfSense running on a physical machine) to start up a Linux VM (trying with both NixOS and Ubuntu Server). I followed instructions in this thread and think I might be running into the same issue as the linked comment -- specifically, the VM boots and runs, but my local DNS resolution (unbound, running on the pfSense host) is not working. Disabling pfblockerng doesn't seem to change anything. Firewall is disabled and rules flushed.

As per that thread, I have set up a TAP interface, a bridge between it and the LAN, and firewall rules on each to allow all traffic (for now).

From the VM, I can:

  • get an IP address via DHCP (in the LAN subnet)
  • ping both internal and external hosts by IP address, including the host pfsense machine at 192.168.0.2
  • send and receive TCP and UDP data with netcat to both internal and external hosts
  • resolve DNS using an external resolver (e.g. host n8henrie.com 8.8.8.8 or 1.1.1.1 or what have you)

I have unbound running on pfsense and it's working great for over a year, is set to work on all interfaces. Using tcpdump, I can see that unbound is responding to DNS requests from the VM with a correct response, but for some reason they don't seem to be arriving. I've recorded a pcap file (from the ubuntu guest) to inspect these DNS responses to see if I can figure out why the packets are arriving but not working.

I don't see any relevant blocked traffic in /var/log/filter.log, so I think the firewall rules are working.

I have watch host -4 -W1 n8henrie.com 192.168.0.2 running in the VM, which just shows ;; connection timed out; no servers could be reached . Simultaneously, I have tcpdump -i tap0 -v host 192.168.0.202 and udp running on pfsense and I see appropriate resolution of the domain (repeating over time as expected):

[root@pfSense ~]# tcpdump -i tap0 -v host 192.168.0.202 and udp
tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:25:34.981350 IP (tos 0x0, ttl 64, id 1081, offset 0, flags [none], proto UDP (17), length 58)
    192.168.0.202.45334 > pfSense.home.arpa.domain: 14697+ A? n8henrie.com. (30)
11:25:34.994974 IP (tos 0x0, ttl 64, id 27251, offset 0, flags [none], proto UDP (17), length 90)
    pfSense.home.arpa.domain > 192.168.0.202.45334: 14697 2/0/0 n8henrie.com. A 104.21.37.209, n8henrie.com. A 172.67.213.115 (62)
11:25:35.814435 IP (tos 0x0, ttl 64, id 50026, offset 0, flags [none], proto UDP (17), length 58)

I can even see that the Running tcpdump in the guest shows the responses arriving...

root@ubuntu-server:/# tcpdump -vv -i enp0s2 host 192.168.0.2 and udp
11:46:49.569313 IP (tos 0x0, ttl 64, id 34767, offset 0, flags [none], proto UDP (17), length 58)
    192.168.0.202.44462 > 192.168.0.2.domain: [udp sum ok] 59526+ A? n8henrie.com. (30)
11:46:49.576273 IP (tos 0x0, ttl 64, id 30262, offset 0, flags [none], proto UDP (17), length 90)
    192.168.0.2.domain > 192.168.0.202.44462: [bad udp cksum 0x8274 -> 0x871c!] 59526 q: A? n8henrie.com. 2/0/0 n8henrie.com. A 104.21.37.209, n8henrie.com. A 172.67.213.115 (62)

So if the DNS responses can be seen in the client, why aren't they working?

Here's the resolv.conf from the NixOS guest, I can also post ubuntu if that helps:

# cat /etc/resolv.conf
domain home.arpa
nameserver 192.168.0.2
options edns0

From the host:

[root@pfSense /]# ifconfig tap0
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT1
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:88:32
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 76444
[root@pfSense /]# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:33:2d:65:15:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000000
        groups: bridge
        nd6 options=1<PERFORMNUD>

Other notes / things I've tried:

  • As noted, firewall is disabled
  • Setting rp_filter to 0 doesn't change anything
  • Same behavior on both ubuntu-22.04.1-live-server-amd64.iso and nixos-minimal-22.11.1705.b83e7f5a04a-x86_64-linux.iso
  • At the end of tcpdump from client I see e.g. X packets captured, X packets received by filter, 0 packets dropped by kernel, so I guess they're not being dropped.

Thanks in advance for any suggestions!

Improve this question

1 Answer 1

Reset to default
1

The key to solving the issue was noticing the bad udp cksum in the tcpdump output from the VM guest. Searching for this led me to a few posts that pointed out the issue, which was having hardware checksum offloading enabled:

Disabling this setting in the pfSense web interface (System -> Advanced -> Networking, check to disable Hardware Checksum Offloading), then a reboot, and afterwards DNS from the pfSense host was working fine.

I wrote up the process in greater detail at https://n8henrie.com/2023/03/running-nixos-and-ubuntu-vms-on-pfsense-via-bhyve/

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .