SSH host key checking cannot disable when using proxy jump

Question

I am trying to SSH through a jumpbox, but SSH seems to be intent on checking host keys for the jumpbox, even though I'm telling it not to, using the normal -o StrictHostKeyChecking=no -o UserKnownHostsFile=no command line options.

If I SSH directly to the jumpbox, I can have SSH ignore the error as expected:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_jumpuser_rsa jumpuser@jumpbox

However, if I add the proxy jump option, I suddenly get the error. The error is NOT coming from the jumpbox there are no known_hosts files in any .ssh directory on the jumpbox, nor am I logging in as the jumpuser:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_jumpuser_rsa -J jumpuser@jumpbox [email protected]

The error message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
<redacted>.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:10
  remove with:
  ssh-keygen -f "/home/user/.ssh/known_hosts" -R jumpbox
ECDSA host key for jumpbox has changed and you have requested strict checking.
Host key verification failed.
ssh_exchange_identification: Connection closed by remote host

Where user is my regular user, not the user I am attempting to SSH as.

I have no clue what's going on here. Does SSH have a special override forcing hostkey checking for proxy jump situations? If so, it's supremely irritating, as it's going to make local VM provisioning a real pain.

Answer 1

The ProxyJump issues another ssh process, that does not inherit the command-line arguments that you specify on the command-line of the first ssh command. There are two possible ways out:

• Use these options in configuration file in ~/.ssh/config -- it can save you a lot of typing too!

  Host jumpbox
    User jumpuser
    StrictHostKeyChecking=no
    UserKnownHostsFile=/dev/null
    IdentityFile ~/.ssh/id_jumpuser_rsa
  

  and then you can connect just as ssh -J jumpbox [email protected].

• Use ProxyCommand option instead -- it does the same job, but more transparently so you can see what is actually going on there:

  ssh -o ProxyCommand="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_jumpuser_rsa -W %h:%p jumpuser@jumpbox" -i ~/.ssh/id_jumpuser_rsa [email protected]

Answer 2

From @Jakuje excellent answer, I made this script for generic file retrieval through a bastion host:

#!/usr/bin/env bash
set -e

ME=$(basename $0)
log_() { echo "[$ME] $@"; }

PRIVATE_IP=$1
BASTION=$2
SOURCE=$3
TARGET=$4
USER=${5:-ubuntu}
BASTION_USER=${6:-$USER}

log_ Copying ${USER}@${PRIVATE_IP}:${SOURCE} to ${TARGET} via ${BASTION_USER}@${BASTION}

scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -o ProxyCommand="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l ${BASTION_USER} -W ${PRIVATE_IP}:22 ${BASTION}" \
    ${USER}@${PRIVATE_IP}:${SOURCE} ${TARGET}

It is useful if you are provisioning a private host inside a data center.

Answer 3

Being an avid Google'er, and this my first result for stackoverflow.

To ssh through a jumpbox/bastion without all the hassle, and without system modifications, etc. Then you must use StrictHostKeyChecking=no both for the proxied session, the jumper. Also you must use StrictHostKeyChecking=no in your proxy_command.

Additionally, don't get too smart for yourself and try to use UserKnownHostsFile=/dev/null for either session or proxy_command, because the your shell will lose context when piping and just fail. I didn't troubleshoot any further.

This command was discovered while cleanly downloading a file from newly minted instance to my local workstation via Terraform local-exec provisioner

ssh -i ssh_server_key.pem -o StrictHostKeyChecking=no -o ProxyCommand="ssh -o StrictHostKeyChecking=no -i ssh_server_key.pem ec2-user@<BASTION_IP> nc <INTERNAL_IP> 22" ubuntu@<INTERNAL_IP>