1

The server in question is a runner and builds projects with a system user gitlab-runner. That user is used to SSH into other servers to deploy code, etc..

Always worked fine until we added a new server as destination. The command SSH now always fails with the error 'Host key verification failed.'. The error is also thrown when I try other servers that worked before. The known_hosts file is cleared but SSH doesn't ask to add the server to known_hosts anymore, it returns the error message directly.

I checked the permissions of the ~/.ssh folder and files. These are correct (.ssh: 700, known_hosts: 600, id_rsa: 600, id_rsa.pub: 644). Als rebooted the server but no success.

It feels like SSH is not working correctly. Here is a debug output of connecting to a server through SSH.

OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to megatron.domain.com [10.139.20.204] port 22.
debug1: Connection established.
debug1: identity file /home/gitlab-runner/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gitlab-runner/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to megatron.achillescm.nl:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/zgPQuuy6sG8UuLG9EHFSFAuY1QYNvQzKSyNYq//DJ0
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.

Anybody an idea?

Improve this question
2
  • ssh is indeed trying to ask you about accepting (and adding) the host key, but for some reason /dev/tty cannot be opened. I've never seen that, and it suggests something is quite messed up on your system, but with the info given I have no clue what. (The function name read_passphrase is somewhat misleading here; it is also called from confirm in ssh_connect.c) If you manually rebuild known_hosts e.g. with ssh-keyscan it should avoid this particular problem, although what else may go wrong I cannot say.
    – dave_thompson_085
    Commented Feb 20, 2018 at 5:19
  • @dave_thompson_085 Thx for the comment. The system is a very simple setup; ubuntu + gitlab runner, nothing more.
    – Sven van Zoelen
    Commented Feb 21, 2018 at 11:04

4 Answers 4

Reset to default
1

I found the problem. The problem was that the known_hosts file had the wrong permissions.

It was set to 600 and it should be 644.

Improve this answer
1
  • It gives me error: Permissions 0644 for 'SSH_PRIVATE_KEY' are too open. It is required that your private key files are NOT accessible by others.
    – Sky
    Commented Nov 8, 2019 at 13:08
0 
debug1: identity file /home/gitlab-runner/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory

Above lines suggest that .ssh/id_rsa exists (type 1), but its public key (.ssh/id_rsa.pub) for the user doesn't or got some problems (type -1).

So make sure the file ~/.ssh/id_rsa exists, has 600 permission, is owned by the user. And the ~/.ssh/id_rsa.pub matches/belongs to the same identity (as per this post).

Here is a simple test to run as a sudo access user (e.g. root):

sudo -u gitlab-runner sh -c 'cd; wc -l .ssh/id_rsa; stat .ssh/id_rsa; head -n1 .ssh/id_rsa'

See also: What does "key_load_public: no such file or directory" mean?

Improve this answer
2
  • Exactly the opposite; type 1 means id_rsa could be read and does contain an RSA key. (Other algorithms are other positive numbers; negative one is an error.) corrected: It isn't finding the subsequent file id_rsa-cert.
    – dave_thompson_085
    Commented Feb 20, 2018 at 5:23
  • The command you specified returns bash: line 0: cd: /root: Permission denied wc: .ssh/id_rsa: Permission denied stat: cannot stat '.ssh/id_rsa': Permission denied. When I execute the command after su gitlab-runner - then it returns: File: '.ssh/id_rsa' Size: 3243 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 283738 Links: 1 Access: (0600/-rw-------) Uid: ( 999/gitlab-runner) Gid: ( 998/gitlab-runner) Acces..
    – Sven van Zoelen
    Commented Feb 20, 2018 at 7:33
0

Sometimes, the same problem arises if /dev/tty permissions are too restrictive. Then you can only ssh to a server properly as user root, nobody else can. The fix is to chmod 777 /dev/tty so the others can do it.

Improve this answer
1
  • 666 is probably just as good (or maybe better). I don't know of any reason to give execute permission to a device.
    – Scott - Слава Україні
    Commented May 27, 2018 at 15:59
0

While debugging a similar thing, the awesome problem was a missing line-break (\n) at the private key file end.

You may check for the inconsistency using

cat ~/.ssh/identity_file

The public key file - i.e. ~/.ssh/identity_file.pub - isn't need to succesfully authenticate against the target host, assuming it has being authorized within target host's ~/.ssh/authorized_keys.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .