Nutshell: Looking for a bit of an ELI5 explaining the protocol and/or implementation (openssh) of [post-KEX] user key verification (who does what) without being in code I don't understand or a too-simple website for noob setting up key auth. We've got Workday and Red Hat looking into it, but I'm trying to be an informed consumer when dealing with them.
How does this verification work right around mm_answer_keyverify? How do they verify the user keys (after authorized_keys is checked and allowed)?
- Does each side sign their keys and signatures are matched?
- Do they encrypt something using their local signing algorithm, and then compare?
More detail:
We're seeing some weird problems when making connections from another server to our inhouse EL9 system. Everything works fine and the same with an EL7 system (being replaced) - using same keys (RSA), same users, same files (NFS home), etc. Host keys and KEX and even authorized_keys checks are successful, and it seems to fail on user keys: We get fails from Workday, and from an AIX system running curl+sftp, but NOT AIX using sftp alone):
debug1: /home/USER/.ssh/authorized_keys:12: matching key found: RSA SHA256:aGrK...
Accepted key RSA SHA256:aGrK... found at /home/USER/.ssh/authorized_keys:12
debug3: mm_answer_keyallowed: publickey authentication: RSA key is allowed
debug3: mm_answer_keyverify: publickey RSA signature unverified: error in libcrypto
We've resorted to running the LEGACY crypto policy just to attempt to diagnose (no joy). Personally, I think we're running into library differences where one is still using a ssh-rsa algorithm, and the other side is using a compatibility algorithm ("you asked for ssh-rsa, but i'll use rsa-sha256-512"), and thus the issue and my questions, but that's just a guess and I'm a layman.
Just for clarity, here's what I see in server logs for a success:
debug3: userauth_pubkey: have rsa-sha2-512 signature for RSA SHA256:aGrK.....
and same server, a fail (keeping in mind we are temporarily allowing SHA1, etc, so that's not the issue; our user keys are 2k RSA sshv2 keys, so that's ok too):
debug3: userauth_pubkey: have ssh-rsa signature for RSA SHA256:aGrK.....
The most obvious bit is the signature difference, but I don't know why the server would use different ones unless that's really the CLIENT saying that to the server...
Thanks for any pointers!