I can't see what's preventing me pinging local machines from the remote site when using WireGuard

Question

I have a flaw in my WireGuard setup, but I can't see it. Any suggestions to help debug would be appreciated.

Setup: A local pi (192.168.1.2/24) running WireGuard that sits behind an ISP gateway. A remote pi (192.168.2.1/24) that is a gateway/wifi ap also running WireGuard. The remote Pi iptables filters are temperalily flushed. iptables is not installed on the local pi.

Desired State: A LAN to LAN L3 bridge.

Problem: I can't ping machines on the local network from any machine on the remote network. I can ping/ssh to any machine on the remote network from a local machine. I can ping to the local pi running Wireguard (ping 192.168.1.2 works) from a remote machine, but not beyond that to other local machines (ping 192.168.1.180 does not work).

CORRECTION: I could always ping/ssh from a regular client on the remote network to any machine on the local network. It was only a ping/ssh from 192.168.2.1 on the remote network to any machine on the local network other than the WireGuard server 192.168.1.2 where the issue was.

Seems I'm have a routing issue on the local pi. Any direction re debug would be appreciated.

dhcpcd.conf and interfaces on the local pi have not been touched.

timemachine@pibackup4:/etc/wireguard $ ip route list
default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.2 metric 202 
10.8.0.0/24 dev wglink2 proto kernel scope link src 10.8.0.1 
192.168.1.0/24 dev eth0 proto dhcp scope link src 192.168.1.2 metric 202 
192.168.2.0/24 via 10.8.0.2 dev wglink2

timemachine@pibackup4:/etc/wireguard $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:03:fe:6f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 74276sec preferred_lft 63476sec
    inet6 fe80::f2a8:7662:6de:5b4e/64 scope link 
       valid_lft forever preferred_lft forever
10: wglink2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.8.0.1/24 scope global wglink2
       valid_lft forever preferred_lft forever

timemachine@pibackup4:/etc/wireguard $ cat wglink2.conf
[Interface]
# LAN Address: 192.168.1.2/24
# Wireguard Address: 10.8.0.1
PrivateKey = xxxxxx
Address = 10.8.0.1/24
ListenPort = 51820

Table = off
PostUp = ip route add 192.168.2.0/24 via 10.8.0.2 dev wglink2
PreDown = ip route del 192.168.2.0/24 via 10.8.0.2 dev wglink2

[Peer]
# LAN Address: 192.168.2.1/24
# Wireguard Address: 10.8.0.2
PublicKey = xxxxxx
AllowedIPs = 10.8.0.2/32, 192.168.2.0/24
PersistentKeepalive = 25

timemachine@pibackup4:/etc/network $ sudo wg
interface: wglink2
  public key: xxxxxx
  private key: (hidden)
  listening port: 51820

peer: xxxxxx
  endpoint: 192.168.1.1:39915
  allowed ips: 10.8.0.2/32, 192.168.2.0/24
  latest handshake: 8 seconds ago
  transfer: 37.14 KiB received, 42.57 KiB sent
  persistent keepalive: every 25 seconds

Answer 1

If you find this question and wondered what was going on, read on.

The routing table on the remote pi (Gateway, WiFi AP, WireGuard Client) is setup to pass anything destined for the local network 192.168.1.0/24 to the interface wglink2 which has an IP address of 10.8.0.2. Consequently the ICMP ping packet has a source IP address of 10.8.0.2. The ICMP packet reaches the destination machine on the local network and sends off a replay, back to 10.8.0.2. However the local network gateway has no entry for 10.8.0.2, so off to the ISP it goes never to be heard from again.

tcpdump from the destination machine 192.168.1.180 on the local network looking at a ping from the remote pi (Gateway, Wi-Fi AP, WireGuard Client):

20:32:01.375555 IP (tos 0x0, ttl 63, id 64374, offset 0, flags [DF], proto ICMP (1), length 84)
    10.8.0.2 > 192.168.1.180: ICMP echo request, id 11, seq 1, length 64
20:32:01.375738 IP (tos 0x0, ttl 64, id 34197, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.180 > 10.8.0.2: ICMP echo reply, id 11, seq 1, length 64

tcpdump from the destination machine 192.168.1.180 on the local network looking at a ping from a regular client on the remote network:

20:31:21.059165 IP (tos 0x0, ttl 62, id 11367, offset 0, flags [none], proto ICMP (1), length 44)
    192.168.2.163 > 192.168.1.180: ICMP echo request, id 56286, seq 38228, length 24
20:31:21.059294 IP (tos 0x0, ttl 64, id 41939, offset 0, flags [none], proto ICMP (1), length 44)
    192.168.1.180 > 192.168.2.163: ICMP echo reply, id 56286, seq 38228, length 24

Everything working as it's supposed to, just not how I expected.