Outbound connections blocked by the ghost of a recently departed IP address

Question

I have a raspberry Pi running Raspberry OS 12 (Debian Bookworm). It had an IP address of 172.16.1.17 and it appeared to be working fine. I moved it to a new subnet with a new IP of 192.168.0.117 and it lost internet connectivity and exhibited strange behaviour.

Specifically; I can ping the RPi on it's new address and ssh into it. If I run 'ifconfig', 'ip addr', route etc the output looks fine. Nevertheless, I cannot make an outbound connection and all attempts end with timeouts. If I run a packet capture from my router (OPNsense) I see that some packets exit the RPI with the old/original IP address and others with the new address.

Return packets from my ssh connection have the correct address, obviously or my ssh wouldn't work. Broadcast packets soliciting the Mac address of the gateway also have the correct address but outgoing connection requests and ping packets have the old address and are consequently blocked by the firewall as the RPi is now on a different network segment.

The IP address on the RPi has been set using the interface file and NetworkManger has been disabled and masked. There is no DHCP from the router, although there is dnsmasq on the RPi itself, serving the wifi connections on the wlan. I have checked this and it is not serving the RPi eth0.

When I look at network connections on the RPi using tools like 'netstat' and 'ss' I can see no sign of the old IP address. Although without 'established connections' I don't have much to look at.

Additionally I can find no trace of the old IP address in any of the configuration files. I have run grep against every file in /etc/ and all its sub directories. I've also run it against a number of application related directories. The only hit I got was within a binary file: /etc/pihole/pihole-FTL.db.

I posted a query on (https://discourse.pi-hole.net/t/cannot-change-ip-subnet/70414/5) and they informed me the pihole-FTL.db file simply holds a record of previous dns queries. They also suggested a number of checks. But they turned up nothing suspicious.

I've checked the network settings within the run time environments of the running processes using the /proc file system and found nothing. I've queried the Network setting within all the listed namespaces and nothing relating to the old IP shows up. I removed avahi/mDNS from the RPi in case the IP address was being passed in from outside machines. I've also run a packet trace on the network segment during the full period of a RPi reboot and I can see nothing being passed into the RPi via the likes of DHCP, mDNS or any other broadcasts or packet transfer. I've flushed the Arp cache on the router and completely changed the mac address on the RPi in case Arp caching was an issue. I've also checked for static entries on my router even though the DCHP service is turned off. All to no avail.

Consequently it appears to me that this old IP address is nowhere to be found accept in the outgoing packets of the Pi box! That is clearly impossible as it must be getting the old IP address from somewhere but I am at a loss as to where to look next. I am no network expert and consequently don't know how to dig deeper into issues like; how or where the http and icmp packets of processes are getting tagged with the old IP address. And how/why the return ssh packets are tagged with the correct IP.

I started by posting this issue on (https://forums.raspberrypi.comviewtopic.php?p=2225401#p2225401). I had some helpful suggestions which seem to point at the Pihole app. This was and still is my main suspect, except for the fact that I have no knowledge of the pihole code base or how and what it 'plugs' into on the RPi. Consequently I have no hard evidence to support my suspicions. Additionally the posts over at 'discourse.pi-hole.net' deny that pihole would be making any changes to the RPi network settings in my case.

The way inbound and outbound connection traffic is being tagged differently suggests to me an issue with the IP stack itself or the code area that feeds packets into the stack, rather than a simple configuration setting. Hence, I suspect a network related bug or an application "poking it's fingers" where they don't belong! Namely; into the the network stack or related libraries.

Any help or suggestions would be gratefully appreciated.