SSH client seems to be checking '/dev/null' instead of 'known_hosts' file when checking remote server's ECDSA key

Question

I'd like to know why my known_hosts file doesn't seem to be checked correctly while using SSH. In short, my guess is that my SSH client is checking /dev/null instead of the known_hosts file. Details on how I guessed that are written below.

For my example, I'm logging in from a node named mars into a client node named saturn. I can get into saturn after setting up the public and private keys, but I get this warning:

root@mars# ssh saturn
Warning: Permanently added 'saturn,10.30.3.3' (ECDSA) to the list of known hosts.

When I logged out and logged in to the same saturn node, I got the same warning message. It doesn't matter how many times I log out and log back in, I get this message. I don't want to suppress the warning. I want to know why this warning keeps appearing. I checked if my known_hosts file in the mars node has the saturn's ECDSA key by doing the following, but I get an error:

# ssh-keygen -F saturn
do_known_hosts: hostkeys_foreach failed: No such file or directory

I wondered if the known_hosts file is not checked correctly while using the SSH client, so logged in with the verbosity flags to check where things went wrong. Below is a truncated output:

root@mars# ssh -vvv saturn
.
.(truncated)
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:nkvxyuLtlDdO8pAycafcfqSPE7OUWgN6Z++Aia/Cygg
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: hostkeys_foreach: reading file "/dev/null"
Warning: Permanently added 'saturn,10.33.3.3' (ECDSA) to the list of known hosts.
.
.(truncated)

So, it seems like my SSH client on mars is looking into /dev/null for the known hosts key, instead of /root/.ssh/known_hosts.

I wanted to see what a "good" behavior looks like, so I used SSH on a different pair of servers (here named earth and neptune) that I already know does not give me the Warning: Permanently added message. I've turned on verbosity and I'm only showing a portion of the log messages. Logging in from earth to neptune gives:

root@earth# ssh -vvv neptune
.
. (truncated)
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:qo7vcBwG53p/9MlaTIQJbMZ8Wgf6QxiCJLR1jUiblQ8
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from saturn
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.33.9.10
debug1: Host 'neptune' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:9
.
.(truncated)

From the above, I can see that earth correctly checks /root/.ssh/known_hosts. Another confirmation that the key is found in known_hosts in this "good" scenario:

root@earth# ssh-keygen -F neptune
# Host neptune found: line 7 

In summary, does anyone know why the Warning message keeps appearing, and if the SSH client is indeed checking /dev/null instead of known_hosts? If my guess is correct, how might the client be fixed so that the message doesn't reappear?

I'm using Ubuntu 18.04 and this SSH client version on all nodes:

root@mars:~# ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

Thanks in advance for any help.

Answer 1

The only explanation that comes to mind is that the SSH client configuration either on this machine or on this account has been modified to not remember known public keys in a known_hosts file. Check the settings for GlobalKnownHostsFile and UserKnownHostsFile in /etc/ssh/ssh_config, /etc/ssh/ssh_config.d/* and ~/.ssh/config.

This may have been done to increase privacy, so that other administrators on the machine (or people who manage to compromise the machine or its backups in the future) can't see where you've connected to unless they spy on the connection. For the known_hosts file, this is not a very good compromise between privacy and security: having to check the peer's public key manually each time is error-prone (in addition to being inconvenient). Turning on the HashKnownHosts option gives a decent amount of privacy with a decent amount of functionality: with this option turned on, it's impossible to directly list the entries in a known_hosts file, all you can do is guess what the entry might be and check your guess (and each check is somewhat costly, so you can't brute-force a very large number of potential server names).