ssh connection issues from 5.3 client to 7.4 server

Question

We have a setup like this:

• client openssh 5.3 on centos 6
• nginxplus stream proxy on centos 7 (proxies custom ssh port to gitlab server port 22)
• gitlab server which runs openssh 7.4

We cannot ssh from the client to the server. We can connect to the ssh server from other places. We can connect to the custom tcp port and while running the ssh client command we see a tcp connect on the port. But no data is proxied to the gitlab server, probably because the client doesn't supply data.

We also tried to start an ssh session to the client host with a port forward to the proxy/gitlab server so we could connect from the client node IP, but with a newer client. This does work, so we don't think it's a firewall or denyhost-like issue. We renewed the ssh host keys on the client node, but this also has no effect.

While connecting we see in tcpdump it doesn't start to negotiate a ssh protocol. We think it might have to do with the client version, or maybe we are missing a configuration setting, but I can't find out what.

[root@clienthost ~]#  ssh -i ~/.ssh/id_rsa [email protected] -p $CUSTOMPORT -vv
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: Applying options for gitlab.test.fqdn
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to gitlab.test.fqdn [a.b.c.d] port $CUSTOMPORT.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1

As far as I understand, this has mostly to do with the old rsa 1 protocol and shouldn't really matter for rsa 2. Privileges for the id_rsa file are 0400 Btw it also does not work for other users than root.

ssh client config:

Host gitlab.test.fqdn
  HostName gitlab.test.fqdn
  User git
  IdentityFile /root/.ssh/id_rsa
  Port $CUSTOMPORT

Edit after comments on the id_rsa file properties

Also fixed that the file command reports the id_rsa as an ASCII text file and now it reports a pem rsa private key file

Anyone any ideas on what I'm missing or could be wrong?

Answer 1

If you run nc -v gitlab.test.fqdn $CUSTOMPORT or telnet gitlab.test.fqdn $CUSTOMPORT, you should immediately get a SSH server version string back if the proxy works in a way that's directly compatible with SSH.

The string should look like this:

SSH-2.0-<something...>

If the proxy has been accidentally misconfigured to expect HTTP protocol, the connection will succeed but there will be no initial response. In HTTP, the client is expected to speak first, as opposed to the SSH protocol which requires the server side to speak first.

(A HTTP-proxy-aware client could then use HTTP CONNECT request to get connected to a specified port at a specified destination, but that would allow the client to choose the destination of the proxied connection, which is apparently not what you want here.)

Answer 2

We had ssl_preread enabled in nginx, which apparently prevents the server-first protocol and version announcement. After turning this off the server does send the protocol and version first and everything works as desired.

telcoM's answer was a huge help in this, so thank you! But I wanted to post the specific configuration change as a solution.