1

I have two users on desktop - root and user. I have a bastion and a protected host. When run ssh protected as root on desktop, I connect fine. When I run ssh protected as user on desktop, I get no output - just a blank line, like it's waiting for something. However, user can log in directly to the bastion host and from there to the protected host.

Both root and user have the same contents in their .ssh directories (#cp -r ~/.ssh /home/user; chown -R user:user /home/user/.ssh).

The bastion host appears to be forwarding properly - running $(which sshd) -Ddp 10222 (per https://unix.stackexchange.com/a/128910/9583) shows the same debug1: channel 0: connected to protected port 22 line on both.

Running the same on protected shows the same output until:

debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]

The second line does not display when connecting from user on desktop.

ssh -vvv protected as user on desktop shows:

OpenSSH_7.9p1, OpenSSL 1.1.1  11 Sep 2018                                                                                                                       
debug1: Reading configuration data /home/user/.ssh/config                                                                                                     
debug1: /home/user/.ssh/config line 1: Applying options for *                                                                                                
debug1: /home/user/.ssh/config line 10: Applying options for protected                                                                                            
debug1: Reading configuration data /etc/ssh/ssh_config                                                                                                          
debug1: Executing proxy command: exec ssh bastion -W protected:22                                                                                                
debug1: identity file /home/user/.ssh/id_protected type 0                                                                                                         
debug1: identity file /home/user/.ssh/id_protected-cert type -1                                                                                                   
debug1: Local version string SSH-2.0-OpenSSH_7.9                                                                                                                
debug1: ssh_exchange_identification: \033[3g
\033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H     \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H    \033H
SSH-2.0-Op
debug1: ssh_exchange_identification: enSSH_7.9


debug1: ssh_exchange_identification:
debug1: ssh_exchange_identification: 6,diffie-hellman-group14-sha1
debug1: ssh_exchange_identification: aes192-ctr,aes256-ctr,[email protected],[email protected]                                                       
debug1: ssh_exchange_identification: 2-256,hmac-sha2-512,hmac-sha1

As root on desktop everything is the same up until the first ssh_exchange_identification line.

My ssh config is:

Host *
ServerAliveInterval 60
IdentitiesOnly yes

Host           bastion
HostName       bastion.host
IdentityFile   ~/.ssh/id_protected
User           user

Host          protected
IdentityFile   ~/.ssh/id_protected
Hostname       protected
User           user
ProxyCommand   ssh bastion -W %h:%p

I have also tried https://askubuntu.com/a/976226/427339, but I believe this doesn't apply for two reasons - 1. emptying my ~/.config/fish/fish.config made no difference, and 2. I can log in to the same user on protected from root on desktop.

All three systems are running Arch Linux. protected and desktop are both using the fish shell.

Edit:

user@bastion's ~/.ssh/config:

Host *
ServerAliveInterval 60

Host protected
User user
IdentityFile ~/.ssh/id_protected

This, as mentioned above, works fine to log into protected. /etc/hosts has an entry for protected pointing to the net-local IP - 10.x.x.x.

Edit 2:

My issue appears very similar to these:

I have not yet tried the MTU workaround, and am not familiar enough with protocol analyzers to have one set up and handy right now.

Edit 3:

Adding -v to the ProxyCommand (is now ProxyCommand ssh -v bastion -W %h:%p), full output of user@desktop$ ssh protected:

OpenSSH_7.9p1, OpenSSL 1.1.1  11 Sep 2018
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 1: Applying options for *
debug1: /home/user/.ssh/config line 5: Applying options for bastion
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to bastion [x.x.x.x] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_protected type 0
debug1: identity file /home/user/.ssh/id_protected-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8
debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000
debug1: Authenticating to bastion:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none                                                             
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none                                                             
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY                                                                                                                       
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HfDmNOGgLrPLMsnCbyZuEuJapj+T6wrSTTiFSd+37ag                                                                 
debug1: Host 'bastion' is known and matches the ECDSA host key.                                                                                           
debug1: Found key in /home/users/.ssh/known_hosts:3                                                                                                           
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                   
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                
debug1: Server accepts key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                 
debug1: Authentication succeeded (publickey).
Authenticated to bastion ([x.x.x.x]:22).
debug1: channel_connect_stdio_fwd protected:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Remote: /home/user/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/user/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
--- very long delay; from `root` everything is the same till here, but the next line is `Last login: ...`, etc - a successful connection ---
debug1: channel 0: FORCE input drain
ssh_exchange_identification: Connection closed by remote host
debug1: channel 0: free: direct-tcpip: listening port 0 for protected port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Killed by signal 1.
Improve this question
11
  • Just in case, to verify the obvious: The permissions on both .ssh and their contents are the same (You used cp -r, not cp -a)? You double-checked that the owner change worked correctly?
    – dirkt
    Commented Nov 21, 2018 at 7:06
  • @dirkt Yep, I entered the commands as pasted, except on two lines instead of with a ;. I would have gotten a different error had the directory had the wrong permissions - I don't recall it, but I've seen it before. Visual inspection just now confirms - the files are owned by user:user, config and id_protected are 0600 and id_protected.pub and known_hosts are 0644. The /home/user/.ssh directory is 700 and also owned by user:user.
    – Iiridayn
    Commented Nov 21, 2018 at 19:02
  • probably need a ForwardAgent yes in each Host section at least, plus I use nc as the proxy.
    – strobelight
    Commented Nov 24, 2018 at 3:26
  • @strobelight tried that - neither made any difference, separate or together. root@desktop can still log in via bastion, user@desktop still cannot. I typically dislike using nc as the proxy as it tends to leave running nc processes on bastion. I'll add user@bastion's .ssh/config to the post - it's set up so it can log in direct, so I wouldn't need agent forwarding (either way didn't make a difference though).
    – Iiridayn
    Commented Nov 27, 2018 at 20:19
  • is the private key the same for both root and user? they don't have to be, but the corresponding public key of each must be in the .ssh/authorized_keys files.
    – strobelight
    Commented Nov 28, 2018 at 1:35

2 Answers 2

Reset to default
0

Since your protected host is behind a bastion host, you still need to go through the bastion host in order to reach it whether from a root user or normal user account.

Simply duplicate the root ssh config to the user ssh config and you should be good to go, but add a ForwardAgent yes in each Host section for both the root user and your normal user on your desktop.

Host *
ServerAliveInterval 60
IdentitiesOnly yes

Host           bastion
HostName       bastion.host
IdentityFile   ~/.ssh/id_protected
User           user
ForwardAgent   yes

Host          protected
IdentityFile   ~/.ssh/id_protected
Hostname       protected
User           user
ForwardAgent   yes
ProxyCommand   ssh bastion -W %h:%p

The sshd_config file on the bastion as well as the protected host, usually in /etc/ssh/sshd_config, will need to have AllowAgentForwarding yes which is usually the default but worth checking. If you do have to change, then restart sshd after saving.

Hopefully the private keys are the same between root and normal user, but if not, ensure the corresponding public keys are in the .ssh/authorized_keys file on the bastion host as well as the protected host. In other words, the bastion users' .ssh/authorized_keys file has an entry (single line, 2 or 3 space-separated fields) containing the public keys of the corresponding private ones used from your desktop and so does the protected host.

If you don't have the public keys any longer, you can get them via this command:

ssh-keygen -l -f /path/to/private_key
Improve this answer
4
  • That is exactly what I stated I did in my question in the second paragraph.
    – Iiridayn
    Commented Nov 28, 2018 at 3:03
  • My understanding is you have two account on desktop, root and user and you're trying to go directly to the protected host user account from your desktop whether logged in as root or as user. ` sudo -i sum .ssh/config sum ~user/.ssh/config # above two numbers should match sum .ssh/id_protected sum ~user/.ssh/id_protected # above two numbers should match ` an ssh config on the bastion is only useful if actually logging into the bastion first.
    – strobelight
    Commented Nov 28, 2018 at 3:14
  • 1) I have two accounts on desktop, correct. 2) I am trying to go directly to protected from desktop whether logged in as user or root, correct. 3) sudo -i sum /root/.ssh/config ~user/.ssh/config are both the same, correct. 4) sudo -i sum /root/.ssh/id_protected ~user/.ssh/id_protected are both the same, correct. This was demonstrated however in paragraph 2 of my question, however, I have repeated these steps for you and verified that I can still log in from root@desktop and cannot from user@desktop. I will leave the bastion config for now, as it might help someone else.
    – Iiridayn
    Commented Nov 28, 2018 at 4:24
  • AllowAgentFowarding is at the default yes on both protected and bastion in /etc/ssh/sshd_config. I have shown 3 times now that the private keys are the same between root and user @ desktop. I have again put ForwardAgent yes in both entries of /root/.ssh/config and then copied/chowned /root/.ssh to /home/user/.ssh and the protected host still shows the same result - root@desktop logs in fine, user@desktop stops at debug1: SSH2_MSG_KEXINIT sent [preauth]. I am frustrated that you have now asked me to take the same steps 2 and 3 times respectively.
    – Iiridayn
    Commented Nov 28, 2018 at 19:22
0

I ran into a very similar problem shelling from laptop (not mentioned in question - I also have pi and several other computers not relevant to the issue) to desktop - which is set up very similarly to protected. The symptom:

user@laptop$ ssh desktop "echo foo" | xxd
00000000: 1b5b 3367 0d1b 4820 2020 201b 4820 2020  .[3g..H    .H
00000010: 201b 4820 2020 201b 4820 2020 201b 4820   .H    .H    .H
00000020: 2020 201b 4820 2020 201b 4820 2020 201b     .H    .H    . 
00000030: 4820 2020 201b 4820 2020 201b 4820 2020  H    .H    .H
00000040: 201b 4820 2020 201b 4820 2020 201b 4820   .H    .H    .H
00000050: 2020 201b 4820 2020 201b 4820 2020 201b     .H    .H    .
00000060: 4820 2020 201b 4820 2020 201b 4820 2020  H    .H    .H
00000070: 201b 4820 2020 201b 4820 2020 0d66 6f6f   .H    .H   .foo
00000080: 0a

This was easily fixed by changing the line tabs -4 to status --is-interactive; and tabs -4 in /home/user/.config/fish/config.fish on desktop. This change I have already made some time ago on protected.

However - changing the /home/user/.config/fish/config.fish on desktop appears to have fixed the login from user@desktop to user@protected. I am posting this as an answer as I feel the resolution is somewhat novel - https://askubuntu.com/a/976226/427339 which I linked in the question states that I need to check for commands producing output on protected, and I resolved this by changing a command which produces output on desktop.

That said, I would love to mark as accepted an answer which explains how and why my shell configuration on the local machine interfered with the ssh pipe, but not direct ssh connections.

Until then, I have come to the tentative conclusion that tabs is a risky program, and should be treated with the utmost of caution if placing it in a shell startup file.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .