iptables service fails with systemd because bind9 is not really ready

Question

I'm trying to invoke iptables-restore in a systemd.service script. I have several rules that require DNS lookup. I included After=bind9.service in the Unit section for the firewall. However, systemd thinks that bind9 is started when it forks, not when named is ready to serve requests. It kicks off the iptables-restore before it can resolve the hostname and therefore fails. Here is the pertinent section of the syslog:

Dec 13 21:29:02 fw systemd[1]: Started BIND Domain Name Server.
Dec 13 21:29:02 fw systemd[1]: Reached target Host and Network Name Lookups.
Dec 13 21:29:02 fw systemd[1]: Starting Load iptables Rules...
Dec 13 21:29:02 fw systemd[1]: Started Homebridge HomeKit bridge.
Dec 13 21:29:02 fw systemd[1]: Starting OpenBSD Secure Shell server...
Dec 13 21:29:02 fw sh[1841]: iptables-restore v1.6.0: host/network `example.com' not found
Dec 13 21:29:02 fw sh[1841]: Error occurred at line: 24
Dec 13 21:29:02 fw sh[1841]: Try `iptables-restore -h' or 'iptables-restore --help' for more information.
Dec 13 21:29:02 fw named[1838]: starting BIND 9.10.3-P4-Ubuntu <id:ebd72b3> -f -4 -u bind

Here's the service file:

[Unit]
Description=Load iptables Rules
After=bind9.service

[Install]
WantedBy=multi-user.target
WantedBy=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/sbin/iptables-restore < /etc/iptables.up.rules"

What is the best way to have the iptables-restore command execute on boot, but wait for named to be active?

Answer 1

However, systemd thinks that bind9 is started when it forks, not when named is ready to serve requests

Systemd has no other way of knowing when named is "ready to serve requests". The commonly accepted protocol, going back to SysVinit days, is that a daemon forks, sets everything up, and once ready tells its parent that it may now exit (usually through a pipe or signal). This is simple and it even works if your "init" is a mere shell-script: named; echo Ready!

(In other words, systemd doesn't care at all about the initial process forking its child, but about when it exits.)

There are other possible protocols as well – e.g. if Type=notify is used, then the daemon is expected to send a READY=1 message; with Type=dbus it has to claim a name on system bus; etc.

But if named doesn't implement this protocol correctly – that is, if its initial process exits too early after forking – then there is nothing init can do about that. It's not init's job to deal with daemons which are lying to it (through incompetence or otherwise).

Having said that, the first thing to do would be to check how bind9.service actually works. It could still be that it doesn't run named directly but through layers of init.d scripts, which tends to cause similar problems. (Even though the logs don't show the typical "LSB: …" prefix I would check anyway.)

It could also be that bind9.service was changed to Type=simple, which does not support any kind of 'ready' notification at all! This also frequently happens when a maintainer blindly picks up "systemd says forking is evil" from some website.

So make sure bind9.service actually Type=forking (not Type=simple) and that it runs named directly (and with the necessary "please daemonize" options).

If that doesn't work (i.e. named still daemonizes too early), your only other option is to have a second service which keeps checking for DNS availability.