Why do I get "Destination Host Unreachable" on the same subnet, pinging a secondary IP address?

Question

I have two AWS EC2 instances (RedHat 8), sa216p and sa216t, connected to the same subnet 172.31.80.0/20. Primary IP addresses are assigned to them by the EC2 DHCP, 172.31.84.105 and 172.31.92.210 respectively. On sa216p I assign a secondary address to the only interface, using ip addr add 172.31.92.123/20 dev eth0.

[ec2-user@sa216p ~]$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether 12:a5:24:f4:78:05 brd ff:ff:ff:ff:ff:ff
    inet 172.31.84.105/20 brd 172.31.95.255 scope global dynamic noprefixroute eth0
       valid_lft 3249sec preferred_lft 3249sec
    inet 172.31.92.123/20 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::10a5:24ff:fef4:7805/64 scope link
       valid_lft forever preferred_lft forever

The security group for the VPC allows SSH and ICMP traffic on the subnet. There's no firewall installed on either instance, so far as I can see. The AWS route table contains an entry for 172.31.0.0/16 local, covering the subnet in question.

From sa216t I can ping sa216p using both its host name and the IP address:

[ec2-user@sa216t ~]$ ping sa216p
PING sa216p (172.31.84.105) 56(84) bytes of data.
64 bytes from sa216p (172.31.84.105): icmp_seq=1 ttl=64 time=0.584 ms
64 bytes from sa216p (172.31.84.105): icmp_seq=2 ttl=64 time=0.674 ms
64 bytes from sa216p (172.31.84.105): icmp_seq=3 ttl=64 time=0.540 ms

However, pinging the secondary address fails anyway:

[ec2-user@sa216t ~]$ ping 172.31.92.123
PING 172.31.92.123 (172.31.92.123) 56(84) bytes of data.
From 172.31.92.210 icmp_seq=1 Destination Host Unreachable
From 172.31.92.210 icmp_seq=2 Destination Host Unreachable
From 172.31.92.210 icmp_seq=3 Destination Host Unreachable
^C

Neighbours table shows the address resolution was unsuccessful:

[ec2-user@sa216t ~]$ ip neigh
172.31.80.1 dev eth0 lladdr 12:c7:16:59:97:6f REACHABLE
172.31.91.6 dev eth0 lladdr 12:68:fb:11:f5:61 STALE
172.31.84.105 dev eth0 lladdr 12:a5:24:f4:78:05 REACHABLE
172.31.92.123 dev eth0  FAILED

On sa216p I can ping both IP addresses:

[ec2-user@sa216p ~]$ ping 172.31.84.105
PING 172.31.84.105 (172.31.84.105) 56(84) bytes of data.
64 bytes from 172.31.84.105: icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from 172.31.84.105: icmp_seq=2 ttl=64 time=0.035 ms
64 bytes from 172.31.84.105: icmp_seq=3 ttl=64 time=0.031 ms
64 bytes from 172.31.84.105: icmp_seq=4 ttl=64 time=0.020 ms
^C
--- 172.31.84.105 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3091ms
rtt min/avg/max/mdev = 0.020/0.026/0.035/0.009 ms
[ec2-user@sa216p ~]$ ping 172.31.92.123
PING 172.31.92.123 (172.31.92.123) 56(84) bytes of data.
64 bytes from 172.31.92.123: icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from 172.31.92.123: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 172.31.92.123: icmp_seq=3 ttl=64 time=0.033 ms
64 bytes from 172.31.92.123: icmp_seq=4 ttl=64 time=0.035 ms
^C
--- 172.31.92.123 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3071ms
rtt min/avg/max/mdev = 0.021/0.033/0.043/0.007 ms

though address resolution apparently fails:

[ec2-user@sa216p ~]$ ip neigh
172.31.92.123 dev eth0  FAILED
172.31.95.166 dev eth0 lladdr 12:e5:55:02:1c:8f REACHABLE
172.31.80.1 dev eth0 lladdr 12:c7:16:59:97:6f REACHABLE
172.31.92.210 dev eth0 lladdr 12:9e:db:f8:9b:7f STALE
172.31.91.6 dev eth0 lladdr 12:68:fb:11:f5:61 REACHABLE

What am I missing? Why can't I reach sa216p on the secondary IP address?

P.S. There's a similar question, but in my case the target host is clearly alive and connected.

P.P.S. This might be specific to the AWS, because I'm told the same setup works as expected when run on-premises, using virtual machines.

Answer 1

Well, it is indeed specific to AWS: one can only assign secondary private IP addresses using the AWS tooling, either from the management console or using the CLI, as described in the docs.