SSH forwarding not terminating on server upon client hard reboot

Question

I have a server with SSHD to which I connect with remote SSH client using the command (on startup):

while true
do
    ssh user@server -N \
        -o "ServerAliveInterval=60" \
        -o "ExitOnForwardFailure=yes" \
        -L 1234:127.0.0.1:4321 \
        -R 1234:127.0.0.1:4321
done

Everything works fine and I am able to use both forwarded ports. If I reboot or power off&on the client or server I am eventually able to use the forwarding.

My problem is that if I turn off the client violently (via hypervisor or by echo b > /proc/sysrq-trigger) - the server keeps listening on port 1234 with the previous process and doesn't ever kill it so the new connection from the client doesn't start (because port is used and can't bind again) so the forwarding doesn't work.

Tried adding config of ClientAliveInterval 60 and ClientAliveCountMax 1 on the server but if the client recovers faster (say 30 sec) then the old process that listens on 1234 will remain forever, like the recovered client satisfies the ClientAlive check somehow despite being unable to use it.

How can I fix this and have the forwarding functional automatically (without having to kill the old listening process on the server)?