PXE boot fails with PXE-99: "Unexpected network error"

Question

I've configured a Kea DHCP server as follows (two different interfaces serving two different networks):

{
    "Dhcp4": {
        "valid-lifetime": 4000,
        "renew-timer": 1000,
        "rebind-timer": 2000,
        "interfaces-config": {
            "interfaces": [ "eth0", "eth1" ]
        },
        "lease-database": {
            "type": "memfile",
            "persist": true,
            "name": "/var/lib/kea/kea-leases4.csv"
        },
        "subnet4": [
            {
                "subnet": "192.168.8.0/24",
                "pools": [
                    {
                         "pool": "192.168.8.2 - 192.168.8.200"
                    }
                ]
            },
            {
                "subnet": "192.168.9.0/24",
                "pools": [
                    {
                         "pool": "192.168.9.2 - 192.168.9.200"
                    }
                ]
            }
        ],
        "option-data": [
            {
                "name": "domain-name-servers",
                "data": "1.1.1.1,8.8.8.8"
            },
            {
                "name": "routers",
                "data": "192.168.8.1,192.168.9.1"
            }
        ],
        "client-classes":
        [
            {
                "name": "UEFI-32",
                "test": "substring(option[60].hex,0,20) == 'PXEClient:Arch:00006'",
                "boot-file-name": "ipxe/i386.efi"
            },
            {
                "name": "UEFI-64",
                "test": "substring(option[60].hex,0,20) == 'PXEClient:Arch:00007'",
                "boot-file-name": "ipxe/x86_64.efi"
            },
      ]
   }
}

When I do tcpdump -i eth0 -nn -vvvv, I do see the DHCP request/response:

    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 74:56:3c:83:a0:85, length 359, xid 0xb9760545, Flags [Broadcast] (0x8000)
      Client-Ethernet-Address 74:56:3c:83:a0:85
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message (53), length 1: Request
        Server-ID (54), length 4: 192.168.8.1
        Requested-IP (50), length 4: 192.168.8.2
        MSZ (57), length 2: 65280
        Parameter-Request (55), length 35: 
          Subnet-Mask (1), Time-Zone (2), Default-Gateway (3), Time-Server (4)
          IEN-Name-Server (5), Domain-Name-Server (6), Hostname (12), BS (13)
          Domain-Name (15), RP (17), EP (18), RSZ (22)
          TTL (23), BR (28), YD (40), YS (41)
          NTP (42), Vendor-Option (43), Requested-IP (50), Lease-Time (51)
          Server-ID (54), RN (58), RB (59), Vendor-Class (60)
          TFTP (66), BF (67), GUID (97), Unknown (128)
          Unknown (129), Unknown (130), Unknown (131), Unknown (132)
          Unknown (133), Unknown (134), Unknown (135)
        GUID (97), length 17: 0.0.192.165.218.38.183.238.17.128.0.116.86.60.131.160.133
        NDI (94), length 3: 1.3.16
        ARCH (93), length 2: 7
        Vendor-Class (60), length 32: "PXEClient:Arch:00007:UNDI:003016"
        END (255), length 0
22:54:47.774286 IP (tos 0x10, ttl 128, id 0, offset 0, flags [DF], proto UDP (17), length 318)
    192.168.8.1.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 290, xid 0xb9760545, Flags [Broadcast] (0x8000)
      Your-IP 192.168.8.2
      Client-Ethernet-Address 74:56:3c:83:a0:85
      file "ipxe/x86_64.efi"
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message (53), length 1: ACK
        Subnet-Mask (1), length 4: 255.255.255.0
        Default-Gateway (3), length 4: 192.168.9.1
        Domain-Name-Server (6), length 8: 1.1.1.1,8.8.8.8
        Lease-Time (51), length 4: 4000
        Server-ID (54), length 4: 192.168.8.1
        RN (58), length 4: 1000
        RB (59), length 4: 2000
        END (255), length 0

but I don't see any tftp requests at all. On the screen I see (very briefly):

Station IP address is 192.168.8.2
Server IP address is 192.168.8.1
NBP filename is ipxe/x86_64.efi
NBP filesize 0 Bytes
PXE-E99: Unexpected network error

What am I doing wrong?

Answer 1

There's a few behaviors that are interacting here to cause the problem:

1. The Kea DHCP server does not use the assigned subnet to figure out what router to serve. As you can see in the tcpdump, it gave out the 192.168.9.1 router to the 192.168.8.0/24 subnet. To fix this, you can provide separate "option-data" specifications in each of your subnets:

   "subnet4": [
       {
           "subnet": "192.168.8.0/24",
           "pools": [
               {
                    "pool": "192.168.8.2 - 192.168.8.200"
               }
           ],
           "option-data": [
               {
                   "name": "routers",
                   "data": "192.168.8.1"
               }
           ]
       },
       {
           "subnet": "192.168.9.0/24",
           "pools": [
               {
                    "pool": "192.168.9.2 - 192.168.9.200"
               }
           ],
           "option-data": [
               {
                   "name": "routers",
                   "data": "192.168.9.1"
               }
           ]
       },
   ]
   

2. The reason you might not see a problem with this configuration outside the pxe context is that ordinarily, the router is not required to reach a host within the same subnet as the client will add a route for the /24 subnet based on the netmask. However, the PXE networking stack does not do this and always uses the default route, so adding it to the configuration is required.