DHCP failover with PXE

Question

I'm trying to set up DHCP failover with PXE booting allowed from one of the DHCP servers. As required by the DHCP specification, I've setup separate pools for "regular" DHCP and for PXE booting. My failover configuration works fine, but the DHCP configuration that's supposed to respond to PXE requests no longer works.

Background: I've recently upgraded to AlmaLinux 9 (from CentOS 7), which runs ISC DHCP 4.4. In the older configuration, I had no DHCP failover, and I allowed PXE booting from the entire pool. Due to a history of hardware failures at our site, I want to set up DHCP failover.

For the purposes of this configuration let's call the system that's supposed to respond to PXE requests the "primary" DHCP server. Here's a fragment of /etc/dhcp/dhcpd/conf from that server. Note that I set up a separate pool just to handle the PXE/BOOTP queries. (Please excuse the didactic tone of the comments. They're meant for me as I do my sysadmin stuff.)

authoritative; # Send out acknowledgements to DHCP client queries.

failover peer "dhcp-failover" {
  primary; # declare this to be the primary server
  address 10.4.7.9;
  port 647;
  peer address 10.4.7.210;
  peer port 647;
  # How many seconds to wait before we assume that the other has failed.
  max-response-delay 30;
  # How many BNDUPD messages to send before receiving BNDACK.
  max-unacked-updates 10;
  # How many seconds to wait before disabling load balancing.
  load balance max seconds 3;
  # Maximum Client Lead Time = How long a lease may be renewed
  # without contacting the other DHCP peer.
  mclt 1800;
  # The split between primary and secondary. 128 means a
  # 50% split between peers; 255 means the primary handles
  # everything until it fails. 
  split 128;
}

# This is the primary DHCP server. Respond to BOOTP requests.
allow booting;
allow bootp;

option domain-name "company.example.com";
option time-offset -18000; # Eastern Standard Time

# Is this a DHCP query (as opposed to a BOOTP query)?
class "dhcp" {
      match if exists dhcp-message-type;
}
class "pxe" {
      match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
}

subnet 10.4.0.0 netmask 255.255.0.0 {
    default-lease-time 86400; # one day (in seconds)
    option subnet-mask 255.255.0.0;
    option broadcast-address 10.4.255.255;
    option routers 10.4.0.1;
    option domain-name-servers 10.4.7.7, 10.4.7.29; 
    option domain-name "company.example.com";
    option time-offset -18000; # Eastern Standard Time
    option ntp-servers 10.4.7.105, 10.4.7.7, 10.4.7.29;

    pool {
         failover peer "dhcp-failover";
         deny dynamic bootp clients;
         deny members of "pxe";
         range 10.4.45.1 10.4.45.250; # DHCP pool on private network
    }
    # A separate pool for BOOTP services.
    pool {
         range dynamic-bootp 10.4.45.251 10.4.45.255; # DHCP pool on private network
         allow dynamic bootp clients;
         deny members of "dhcp";
         allow members of "pxe";
         next-server 10.4.7.9;    # On which system the bootp filename is located.

         if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {

            if substring(option vendor-class-identifier,15,5) = "00007" {
               log(info,"UEFI PXE Boot - private network");
               filename "pxelinux/grubx64.efi"; # The file to load for EFI systems.
               }
            else {
                log(info,"BIOS PXE Boot - private network");
                filename "pxelinux.0"; # The file to load via bootp for BIOS systems.
                }
        }
    }
}

This is from /etc/dhcp/dhcpd.conf on the failover / secondary / non-PXE server:

authoritative; # Send out acknowledgements to DHCP client queries. 

failover peer "dhcp-failover" {
  secondary; # declare this to be the secondary server
  address 10.4.7.210;
  port 647;
  peer address 10.4.7.9;
  peer port 647;
  # How many seconds to wait before we assume that the other has failed.
  max-response-delay 30;
  # How many BNDUPD messages to send before receiving BNDACK.
  max-unacked-updates 10;
  # How many seconds to wait before disabling load balancing.
  load balance max seconds 3;
}

# Make sure that this failover DHCP server does _not_
# respond to bootp.
deny bootp;

option domain-name "company.example.com";
option time-offset -18000; # Eastern Standard Time

# Is this a DHCP query (as opposed to a BOOTP query)?
class "dhcp" {
      match if exists dhcp-message-type;
}
class "pxe" {
      match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
}

subnet 10.4.0.0 netmask 255.255.0.0 {
    default-lease-time 86400; # one day (in seconds)
    option subnet-mask 255.255.0.0;
    option broadcast-address 10.4.255.255;
    option routers 10.4.0.1;
    option domain-name-servers 10.4.7.7, 10.4.7.29; 
    option domain-name "company.example.com";
    option time-offset -18000; # Eastern Standard Time
    option ntp-servers 10.4.7.105, 10.4.7.7, 10.4.7.29;

    # Note that there are a few IP addresses in the range of the primary
    # server that are not included here. This is for BOOTP, which is
    # not handled by the secondary server.
    pool {
         failover peer "dhcp-failover";
         deny dynamic bootp clients;     
         deny members of "pxe";
         range 10.4.45.1 10.4.45.250; # DHCP pool on private network
    }
}

I know that I'm over-doing it with the "dhcp" and "pxe" classes. I added them as I attempted to fix the problem. They had no effect, except to introduce the peer holds all free leases log messages below.

This is what I see in the logs of the "primary" server. Note that 52:54:00:31:f2:7f is the MAC address of a test system I set up to boot via PXE before it "gives up" and boots from disk instead.

Sep  8 14:20:46 dhcpd dhcpd[17922]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:20:49 dhcpd dhcpd[17922]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:20:57 dhcpd dhcpd[17922]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:21:13 dhcpd dhcpd[17922]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases

This is from the log on the "secondary" server. This is consistent with the roughly one-minute delay from when the client firsts boots, as it tries to locate a PXE server, to the point where it switches from booting from the OS and acquires a DHCP address in the usual way.

Sep  8 14:20:46 dhcpdsec dhcpd[67768]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:20:46 dhcpdsec dhcpd[67768]: bind update on 10.4.45.183 from dhcp-failover rejected: incoming update is less critical than outgoing update
Sep  8 14:20:49 dhcpdsec dhcpd[67768]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:20:57 dhcpdsec dhcpd[67768]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:21:13 dhcpdsec dhcpd[67768]: DHCPDISCOVER from 52:54:00:31:f2:7f via enp7s0: peer holds all free leases
Sep  8 14:22:03 dhcpdsec dhcpd[67768]: DHCPREQUEST for 10.4.45.183 from 52:54:00:31:f2:7f via enp7s0
Sep  8 14:22:04 dhcpdsec dhcpd[67768]: DHCPACK on 10.4.45.183 to 52:54:00:31:f2:7f via enp7s0

From flailing around in earlier tests, I confirmed that the value of substring (option vendor-class-identifier, 0, 9) is indeed PXEClient.

I've already tried stopping the dhcpd daemon on the both machines and manually editing out the entries for 52:54:00:31:f2:7f in /var/lib/dhcpd/dhcpd.leases. No change.

Any ideas?

Edit: It occurred to me that it may help to post my previous DHCP configuration, with no failover. PXE booting worked fine:

subnet 10.4.0.0 netmask 255.255.0.0 {
    range dynamic-bootp 10.4.45.1 10.4.45.254; # DCHP pool on private network
    default-lease-time 86400; # one day (in seconds)
    option subnet-mask 255.255.0.0;
    option broadcast-address 10.4.255.255;
    option routers 10.4.0.1;
    option domain-name-servers 10.4.7.7, 10.4.7.29; 
    option domain-name "nevis.columbia.edu";
    option time-offset -18000; # Eastern Standard Time
    option ntp-servers 10.4.7.105, 10.4.7.7, 10.4.7.29;
    next-server 10.4.7.9;    # On which system the bootp filename is located.

    if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {

        if substring(option vendor-class-identifier,15,5) = "00007" {
            log(info,"UEFI PXE Boot - private network");
            filename "pxelinux/grubx64.efi"; # The file to load for EFI systems.
            }
        else {
            log(info,"BIOS PXE Boot - private network");
            filename "pxelinux.0"; # The file to load via bootp for BIOS systems.
        }
    }
}

Answer 1

I found an answer after much experimenting: It turns out that the order of the access-control statements within a pool matters.

Here's a repeat of the class definitions in my original post:

class "dhcp" {
      match if exists dhcp-message-type;
}
class "pxe" {
      match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
}

Here's the subnet definition that works on my primary DHCP server. The main difference between this and the configuration in my original post is that order of the range statements compared to any allow or deny statements, and that I define the "pxe" pool first. The original failover lines are unchanged.

subnet 10.4.0.0 netmask 255.255.0.0 {
    default-lease-time 86400; # one day (in seconds)
    option subnet-mask 255.255.0.0;
    option broadcast-address 10.4.255.255;
    option routers 10.4.0.1;
    option domain-name-servers 10.4.7.7, 10.4.7.29; 
    option domain-name "company.example.com";
    option time-offset -18000; # Eastern Standard Time
    option ntp-servers 10.4.7.105, 10.4.7.7, 10.4.7.29;
    next-server 10.4.7.9;    # On which system the bootp filename is located.

    if substring (option vendor-class-identifier, 0, 9) = "PXEClient" {
        if option architecture-type =  00:07 {
           filename "uefi/grubx64.efi"; # The file to load for EFI systems.
        }
        else {
           filename "pxelinux/pxelinux.0"; # The file to load via bootp for BIOS systems.
        }
    }

    # A separate pool for PXE services.
    pool {
         range dynamic-bootp 10.4.45.251 10.4.45.255; # DHCP pool on private network
         allow dynamic bootp clients;
         allow members of "pxe";
    }

    # The "regular" DHCP pool.
    pool {
         failover peer "dhcp-failover";
         range 10.4.45.1 10.4.45.250; # DHCP pool on private network
         deny dynamic bootp clients;
         deny members of "pxe";
    }
}

Here are the revised subnet lines in my secondary DHCP server configuration, though these changes probably don't matter:

subnet 10.4.0.0 netmask 255.255.0.0 {
    default-lease-time 86400; # one day (in seconds)
    option subnet-mask 255.255.0.0;
    option broadcast-address 10.4.255.255;
    option routers 10.4.0.1;
    option domain-name-servers 10.4.7.7, 10.4.7.29; 
    option domain-name "company.example.com";
    option time-offset -18000; # Eastern Standard Time
    option ntp-servers 10.4.7.105, 10.4.7.7, 10.4.7.29;

    # Note that there are a few IP addresses in the range of the primary
    # server that are not included here. This is for PXE, which is
    # not handled by the secondary server.
    pool {
         failover peer "dhcp-failover";
         deny dynamic bootp clients;     
         range 10.4.45.1 10.4.45.250; # DCHP pool on private network
    }
}

I now have a set-up with both DHCP failover, and PXE booting for installing/repairing an OS, that accommodates both BIOS and EFI systems. I hope someone else finds the above lines useful!