SSH-Key authentication fails

Question

I'm trying to ssh into a CentOS server which I have no control over.. the admin has added my public key to the server and insists the fault lies with me but I can't figure out what is wrong.

Config in .ssh:

tim@tim-UX31A:~$ cat ~/.ssh/config
User root
PasswordAuthentication no
IdentityFile ~/.ssh/id_rsa

Permission on my key-files:

tim@tim-UX31A:~$ ls -l ~/.ssh/id_rsa*
-rw------- 1 tim tim 3326 Okt 20 17:28 /home/tim/.ssh/id_rsa
-rw-r--r-- 1 tim tim  746 Okt 20 17:28 /home/tim/.ssh/id_rsa.pub

Connection log which I can't make any sense of:

tim@tim-UX31A:~$ ssh -vvv [email protected]
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/tim/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "10.0.12.28" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.0.12.28 [10.0.12.28] port 22.
debug1: Connection established.
debug1: identity file /home/tim/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/tim/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.0.12.28:22 as 'root'
debug3: hostkeys_foreach: reading file "/home/tim/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/tim/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.0.12.28
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: 
debug3: hostkeys_foreach: reading file "/home/tim/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/tim/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.0.12.28
debug1: Host '10.0.12.28' is known and matches the ECDSA host key.
debug1: Found key in /home/tim/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/tim/.ssh/id_rsa (0x55ee619ab2b0), explicit, agent
debug2: key: /home/tim/.ssh/id_rsa (0x55ee619bcfa0), agent
debug2: key: tim@Tim-UX31A-Debian (0x55ee619b9370), agent
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/tim/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering RSA public key: /home/tim/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering RSA public key: tim@Tim-UX31A-Debian
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Answer 1

This will usually resolve most SSH authorized key permission issues on the server side, assuming someone didn't make additional changes to the permissions.

# paste these into an SSH session that server (probably from
# another user account or root)

# change this to YOUR username on the server.
YOURUSER=example

# paste these lines verbatim:
sudo chown $YOURUSER:$YOURUSER /home/$YOURUSER/{.,.ssh/,.ssh/authorized_keys}
sudo chmod u+rwX,go-rwX,-t /home/$YOURUSER/{.ssh/,.ssh/authorized_keys}
sudo chmod go-w /home/$YOURUSER/

(This is what Userify does automatically in its "shim" script to update and fix any permission issues based on changes in the team's web dashboard.)

If your admin created the .ssh/ directory or .ssh/authorized_keys file as root (which is most commonly how this becomes broken), then having the file owned by another user isn't allowed, EVEN if that user is root.

Answer 2

I had the exact same problem on two servers: a Linux running Debian stretch and on a NAS (Synology DS715)

it turned out that in both cases, the home directory permissions on the server were wrong

the auth.log on the server was very helpful

Authentication refused: bad ownership or modes for directory /home/cyril

on the Linux, it had the write/group bit on (drwxrwxr--x), so I had to remove at least the write on group (chmod g-w ~/) and then it worked

on the Synology, for whatever reason, there was a sticky bit

drwx--x--x+ 4 toto users 4096 Jan 6 12:11 /var/services/homes/toto

I had to change it with

chmod -t ~/

and I could then connect without a password

Answer 3

When using CentOS 7, and I'm confident applies to other Linux OS's using sshd as well. With root access, you can determine more about why authentication may be failing. To do this:

1. Enable logging for the sshd daemon: sudo vi /etc/ssh/sshd_config
2. Under logging uncomment:

SyslogFacility AUTH
LogLevel INFO

1. Change LogLevel from INFO to DEBUG
2. Save and exit
3. Restart the SSH daemon with sudo systemctl restart sshd
4. Watch the messages file tail -l /var/log/messages
5. Using another terminal, attempt to connect with ssh
6. Attempt to connect with ssh
7. Review the authentication log for the exact cause

For example, I was experiencing some of the same problems as mentioned above.

Authentication refused: bad ownership or modes for file /home/user/.ssh/authorized_keys

Using these steps I was able to confirm the problem was permissions on the authorized_keys file. By setting chmod 644 on the authorized_keys file of my user, the problem was fixed.

Answer 4

It looks like the permissions on your .ssh folder didn't copy+paste correctly. Could you please add it again?

If strict mode is enabled then we have to make sure .ssh has the correct permissions of:

• .ssh/ should have perms 0700/rwx------
• .ssh/*.pub files should be 644/rw-r--r--
• .ssh/* (other files in .ssh) 0600/rw-------

How do things look for you permission-wise?

Answer 5

I've had a similar problem, where the ssh connection tries key ~/.ssh/id_rsa before unexpectedly stopping on:

debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

In my case, it was due to an old public key file lying around in the .ssh directory:

[gitlab-runner@validation-k8s-1 ~]$ ll .ssh/id_rsa*
total 16
-rw------- 1 gitlab-runner gitlab-runner 1675 Sep 18 18:02 id_rsa      --> new private key
-rw-r--r--. 1 gitlab-runner gitlab-runner  423 Jun 12 13:51 id_rsa.pub --> old public key

Moving/deleting the id_rsa.pub from the .ssh directory solved the problem.

From what I understand: when there is a public key present on the client-side, SSH 1st validates the private key against it. If it fails, it won't try to use the private key to connect remotely.

I sent an e-mail to the openssh mailing list: https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-April/035048.html.

Answer 6

Also encountered this problem. setroubleshoot did not seem to work in my environment so there were no such log record in /var/log/messages. Disabling SELinux was not an option for me, so I did

restorecon -Rv ~/.ssh

After that login by rsa key worked fine.

Answer 7

Just in case this also saves someone. I was trying to copy a key from my Ubuntu 18.04 Machine to 2 CentOS 7 Servers. I used ssh-copy-id to transfer them. One worked, one didn't. So I went through all the permissions debugging and found nothing. So finally I pulled up the file /etc/ssh/sshd_config on both servers and stepped line by line through them. Finally I found it, probably something that someone modified long before I got on the job.

One read: AuthorizedKeysFile .ssh/authorized_keys

And another read: AuthorizedKeysFile ~/.ssh/authorized_keys, which was on the server that wasn't accepting my keys. Obviously looking between the two files and noting the comment that states the default search patterns do not include the leading ~/ I removed it and restarted sshd. Problem solved.

Additionally alternative path to key of user "Megatron" (which have home directory "/usr/sap/Earth") can be written in "/etc/ssh/sshd_config" like

`AuthorizedKeysFile /home/Megatron/.ssh/authorized_keys

And even if you make new keys in your home directory with correct permissions they wouldn't work, because you forget this custom path to keys.

Answer 8

Just in case someone stumbles upon this answer - none of the recommendations worked in my scenario. In the end, the problem was that I had created an account with no password set. Once I set the password using usermod -p "my password" username and then forcibly unlocked the account usermod -U username everything was peachy.

Answer 9

The reason in my case was a customly set option AuthorizedKeysFile in file /etc/ssh/sshd_config. It was set to another user's home dir (/home/webmaster/.ssh/authorized_keys), so the user I was trying to log in had no access to that file/directory.

After changing it and restarting ssh-server (service ssh restart) everything came back to normal. I can log in by my private key now.

Answer 10

In my case, ssh -vvv my_dev_box gives me this,

...
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/userxyz/.ssh/id_rsa RSA SHA256:FzMfrbORgYEtcIaWYg2iZOBctxYeNZ9bz/vFxLLtefw agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
...

the /var/log/auth.log on the remote machine has this,

Invalid user userxyz from 10.11.50.126 port 50310 

So solution is,

ssh correct_username@my_dev_box

Or you can specify the username in ssh config file.

Answer 11

We encountered this problem. Permissions and ownership on .ssh files were all correct. In /var/log/messages we found:

Mar 29 15:45:36 centos70 setroubleshoot: SELinux is preventing /usr/sbin/sshd from read access on the file authorized_keys. For complete SELinux messages run: sealert -l 05963b94-f318-4615-806c-b6c3a9066c82

SO, solution for developer vm where we do not care about security is disable selinux. Edit /etc/sysconfig/selinux and change SELINUX=disabled and reboot.

Answer 12

In my case the issues was with the incorrect shell exec.

journalctl -f
....
Feb 25 11:45:54 59a02b89e0f6 sshd[]: User user not allowed because shell /usr/bin/env /bin/bash does not exist
....

Changed /etc/passwd file for that user

vi /etc/passwd 
....
user:x:1000:1000::/home/user:/bin/bash
....

Answer 13

In our case the issues was related to the fact that our firewall and NATing rules were not setup correctly.

port 22, was being directed to the incorrect server where our keys and user was not being recognised.

If some one gets to this point. tcpdump and telnet can be your friend

[aaron@aaron-pc ~]$ telnet someserver 22
Trying 1.1.1.1...
Connected to someserver.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.7p1
^]
telnet> 

[aaron@aaron-pc ~]$ telnet someotherserver 22
Trying 1.1.1.2...
Connected to someotherserver.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
^]

You will notice that these two servers have different openssh versions. This helped me spot the problem pretty quickly. If your hosts are using the same ssh versions you will have to try doing a packed trace on the destination to see if traffic is actual arriving at the destination.

Ssh can generate a lot of traffic which makes tcpdump out put difficult to find what you are looking for.

This helped me

 tcpdump -i any  "not host [mylocalip] and not localhost and not ip and not arp"

Try to telnet from a 3 different server not your current computer @ [mylocalip]. You want to see what traffic actually reaches your server.

Answer 14

A client-side error log ending like this:

Enter passphrase for key '/root/.ssh/id_rsa':
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password:

can be caused by a server-side (remote) restriction on root login when the sshd configuration file contains:

PermitRootLogin: no

JonCav's suggestion to enable logging was helpful in debugging such an issue. While client-side debug spew was remarkably unhelpful, placing the following in the sshd server's sshd_config file:

SyslogFacility AUTH
LogLevel DEBUG

ended up producing helpful log messages:

Jul 19 19:16:38 500265-web1 sshd[21188]: Found matching RSA key: ...
Jul 19 19:16:38 500265-web1 sshd[21188]: ROOT LOGIN REFUSED FROM ...
Jul 19 19:16:38 500265-web1 sshd[21188]: Failed publickey for root from ... port ... ssh2
Jul 19 19:16:38 500265-web1 sshd[21189]: ROOT LOGIN REFUSED FROM ...

In the case where only root login fails, and providing that using only key-based authentication for root login is permitted by your security policy, a change to the sshd_config file can help:

 PermitRootLogin without-password

Your mileage may vary, though this does often help, some other configuration may still interfere according to a comment found in some sshd_config files:

# Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".

Even if you cannot easily change the remote server configuration to debug in this manner, one may proof the client-side configuration to some extent by using the same identity files to ssh to a non-root account on the same remote server.

Answer 15

I can see why security can bother people. I just had the ssh won't use my key problem again. I solved it by logging into the remote server and running

/usr/sbin/sshd -sDp 23456

and then from my desktop, (trying to ssh to server)

ssh -vvvv server -p 23456

On the server I got Authentication refused: bad ownership or modes for directory /

Some new sysadmin had messed up the permission and ownership, which I fixed with:

chmod 0755 / ; chown root:root /

(I'm used to needing chmod 0600 ~/.ssh/* ; chmod 0644 ~/.ssh/*.pub but sshd checking/finding the root permissions is a new one for me.) Now I'm going to check for a rootkit and then wipe and re-install anyway.

Answer 16

I had this problem on CentOS 7. I am a regular Debian-based Linux user so I was a fish out of the water. I noticed that in some of the servers it worked and in just one it didn't. The audit.log said nothing useful and the secure.log did not give anything either. I found that the only real difference was some security context differences on files and directories between those that worked and those that didn't. Get the security with

sudo ls -laZ <user-home>/.ssh

of the directory (I'm assuming a lot of defaults on sshd_config).

You should see some ssh_home_t and user_home_t attributes. If you don't, use the chcon command to add the missing attributes.

For example

home="$(getent passwd <user> | cut -d: -f6)"
sudo chcon -R unconfined_u:object_r:ssh_home_t:s0 "$home".ssh
sudo chcon unconfined_u:object_r:user_home_t:s0 "$home"

In my case, my suspicion is that the user was created in a non standard way. His home was a directory in /var/lib.

More info in: https://www.linuxquestions.org/questions/linux-security-4/selinux-preventing-ssh-login-with-~-ssh-authorized_keys-4175469538/

Answer 17

Another reason this can fail is if the authorized_keys or administrators_authorized_keys was edited by notepad.exe or similar and the encoding was set to something stupid like UTF-16.

In that case, use notepad to Save As UTF-8 (not UTF-8 with BOM) and save as type All Files.