Why HAProxy tcp-request content do-resolve is not working

Question

i am using a combination of HAProxy (docker image v2.3.2) and Hashicorp Consul (v1.9.1) to access my VMs behind haproxy using SSH (i followed https://www.haproxy.com/fr/blog/route-ssh-connections-with-haproxy/)

1. My VMs are registered in consul
2. I use the dns service of consul which listens on 53/tcp and 53/udp port to map the node (vm) name to it's ip address

# the following command works just fine (it gives the ip address of registered node)
dig @<consul_server_ip> -p 53 <node_name>.node.<mydomain> +noall +answer

3. I started the haproxy container attached the host network with the following haproxy.cfg file

global
  log stdout format raw local0

defaults
  log global
  timeout client 60s
  timeout server 20s
  timeout connect 20s

resolvers consul
  accepted_payload_size 8192
  nameserver dns1 <consul_server_ip>:53
  resolve_retries      3
  timeout resolve      1s
  timeout retry        1s
  hold other           30s
  hold refused         30s
  hold nx              30s
  hold timeout         30s
  hold valid           10s
  hold obsolete        30s

frontend fe_ssh
  bind *:2222 ssl crt <path/to/ssl.pem>
  mode tcp
  log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq dstName:%[var(sess.dstName)] dstIP:%[var(sess.dstIP)]"
  tcp-request content do-resolve(sess.dstIP,consul,ipv4) ssl_fc_sni
  tcp-request content set-var(sess.dstName) ssl_fc_sni
  default_backend be_ssh-servers

backend be_ssh-servers
  mode tcp
  tcp-request content set-dst var(sess.dstIP)
  server ssh-srv 0.0.0.0:22

Now when i try to connect to one of my VMs behind the haproxy with the command

ssh -i ~/.ssh/<mynode-priv-key.pem> -o ProxyCommand="openssl s_client -quiet -connect haproxy.mydomain:2222 -servername <node_name>.node.<mydomain>" <user>@<node_name>.node.<mydomain>

the ssh command tries to connect to the VM where the haproxy is running (haproxy.mydomain) and not the specified one with

-servername <node_name>.node.<mydomain>

and haproxy log gives me this

source_ip:port [date/time] fe_ssh~ be_ssh-servers/ssh-srv 60/1/3546 3037 -- 1/1/0/0/0 0/0 dstName:<node_name>.node.<mydomain> dstIP:-

so as you see in the log haproxy printed the correct name of my node

dstName=<node_name>.node.<mydomain>

and not the of haproxy.mydomain (VM where haproxy is running) but,

THERE IS NO ip (dstIP) address associated to my vm (<node_name>.node. ) that i am trying to connect to using SSH due the resolver issue i think

any help and ideas how to debug this issue (as i am new to haproxy and consul) will be appreciated. thanks all in advance.

NB: i already used the consul-template to get what i want but i wanna make it work without using it

Answer 1

you need to add this to your configuration

tcp-request inspect-delay 5s

in the fronted part of your configuration.

Had the same issue and asked the same question: https://discourse.haproxy.org/t/do-resolve-not-working/6504