All SSH keys suddenly stopped Working

Question

All SSH keys to connect to my Debian 10.5 (buster) running ISPConfig 3 have suddenly stopped working with a Permission denied (publickey). error. I created a series of keys and directories on my machine and uploaded the public keys to the ISPConfig console as is common. But when I went to test them, suddenly all the keys stopped working.

I used ssh-keygen -b 4096 to generate my keys on my machine

Using $stat ~/.ssh/id_rsa and $stat ~/.ssh/id_rsa.pub are all 600 and 644 permissions respectively.

When I run this (all keys / users):

ssh -vvvv -i ~/.ssh/id_test_root_rsa [email protected]

I get the same error:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/username/.ssh/config
debug1: /home/username/.ssh/config line 39: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "test.example.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to test.example.com [192.168.1114.16] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_wayne_greysky_rsa type 0
debug1: identity file /home/username/.ssh/id_wayne_greysky_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to test.example.com:22 as 'root'
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from test.example.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/UbY27WLpQv3cKjD9DYVcBFO9PvWOGQedqZBiNMmDgQ
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from test.example.com
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:10
debug3: load_hostkeys: loaded 1 keys from 192.168.1114.16
debug1: Host 'test.example.com' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:9
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/username/.ssh/id_wayne_greysky_rsa RSA SHA256:MOlw6wxat1DTsaL41395RV7t78GJa6Cze2a91bSa6PY explicit agent
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner
Welcome to the Thunderdome
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/username/.ssh/id_wayne_greysky_rsa RSA SHA256:MOlw6wxat1DTsaL41395RV7t78GJa6Cze2a91bSa6PY explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

Also I noticed that ssh was checking ssh_config.. instead of sshd_config...

NOTE: The root key was still working but it amid testing I re-named and then re-generated an SSH key for root… so now I am completely locked out.

Any tips?

Answer 1

I fixed the problem by doing the following:

In /etc/ssh/sshd_config, I changed:

PermitRootLogin without-password

To:

PermitRootLogin yes

This in /erc/ssh/ssh_config I actually uncommented line 19 that says IdentityFile ~/.ssh/id_rsa

From there, I ran service sshd restart and service sshrestart.

Then in my local machine terminal I ran:

ssh-copy-id [email protected]

This gave me a message stating that multiple keys were uploaded.

So I tried ssh -vvvv -i ~/ssh/id_rsa [email protected]

And presto! It worked. I tested my other users / keys and they all work as well.

Finally, I commented out IdentityFile ~/.ssh/id_rsa in /etc/ssh/ssh_config and changed PermitRootLogin Yes back to PermitRootLogin without-password

Tested everything again, and everything this still works...

I don't think my edit of ssh_config was relevant, it may have been because it is referenced in the debug output in my original post above. I think the problem was that no new keys would be accepted without changing PermitRootLogin to Yes, then switching back to without-password.