Teleport compatibility with OpenSSH v6

Hello!
When I try to connect to the server with Ubuntu 14 I get the message:

* open failed: connect failed: ssh: handshake failed: remote host presented a public key, expected a host certificate

Server data:

* vagrant@trusty:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"
vagrant@trusty:~$ ssh -V
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f 6 Jan 2014

Do I understand correctly that Teleport is not compatible with the OpenSSH version? There is no such problem on the server with the Bionic.

OpenSSH certificates were added in OpenSSH 5.4 which is older, so they should work fine on 6.6.1.

This error message actually looks like you’re connecting to a regular sshd which hasn’t been configured with a HostCertificate entry in its configuration file. Can you share the command you’re using and maybe the output of ssh -vv?

In sshd_config added parameters:

TrustedUserCAKeys /etc/ssh/teleport_user_ca.pub
HostKey /etc/ssh/ssh_known_hosts
HostCertificate /etc/ssh/ssh_known_hosts-cert.pub

Deleting the TrustedUserCAKeys parameter on Bionic also leads to an error when connecting

open failed: connect failed: ssh: handshake failed: remote host presented a public key, expected a host certificate

result of ssh connect to Trusty server

$ ssh -vvv vagrant@trusty.mysite.com
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/jimdell/.ssh/config
debug1: /home/jimdell/.ssh/config line 6: Applying options for *.mysite.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Executing proxy command: exec ssh -o 'ForwardAgent yes' -p 3023 vagrant@tp.mysite.com -s proxy:trusty.mysite.com:22
debug1: permanently_drop_suid: 1000
debug1: identity file /home/jimdell/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimdell/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to trusty.mysite.com:22 as 'vagrant'
debug3: hostkeys_foreach: reading file "/home/jimdell/.ssh/known_hosts"
debug3: record_hostkey: found ca key type RSA in file /home/jimdell/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from trusty.mysite.com
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal                                                                                                                                       
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:BURkziWpd3xhdkim+RiQG9AqCHiNYb9Nop6rUZKO+pk, serial 0 ID "" CA ssh-rsa SHA256:RezINUgeZc7ixTVjx0eL5IAoirnSaMcnzgt43kHeL2I valid after 2020-10-09T08:07:31
debug2: Server host certificate hostname: trusty.mysite.com.main
debug2: Server host certificate hostname: trusty.mysite.com
debug2: Server host certificate hostname: localhost
debug2: Server host certificate hostname: 127.0.0.1
debug2: Server host certificate hostname: ::1
debug3: hostkeys_foreach: reading file "/home/jimdell/.ssh/known_hosts"
debug3: record_hostkey: found ca key type RSA in file /home/jimdell/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from trusty.mysite.com
debug1: Host 'trusty.mysite.com' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in /home/jimdell/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: teleport:vagrant (0x56112daeccd0), agent
debug2: key: teleport:vagrant (0x56112dae9050), agent
debug2: key: /home/jimdell/.ssh/id_rsa (0x56112dae6920)
debug2: key: /home/jimdell/.ssh/id_dsa ((nil))
debug2: key: /home/jimdell/.ssh/id_ecdsa ((nil))
debug2: key: /home/jimdell/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA-CERT SHA256:1FG+HKmg2TEvXMr47mBXipzQe6tpsV+makMgxcWYF24 teleport:vagrant
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa-cert-v01@openssh.com blen 1209
debug2: input_userauth_pk_ok: fp SHA256:1FG+HKmg2TEvXMr47mBXipzQe6tpsV+makMgxcWYF24
debug3: sign_and_send_pubkey: RSA-CERT SHA256:1FG+HKmg2TEvXMr47mBXipzQe6tpsV+makMgxcWYF24
debug2: sign_and_send_pubkey: using private key "teleport:vagrant" from agent for certificate
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to trusty.mysite.com (via proxy).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: proc
debug3: receive packet: type 92
channel 0: open failed: connect failed: ssh: handshake failed: remote host presented a public key, expected a host certificate
debug2: channel_input_open_failure: channel 0: callback start
debug2: channel_input_open_failure: channel 0: callback done
debug2: channel 0: zombie
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: zombie
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:

debug3: send packet: type 1
Connection to trusty.mysite.com closed.
Transferred: sent 4464, received 3348 bytes, in 0.0 seconds
Bytes per second: sent 515710.2, received 386782.6
debug1: Exit status -1

Interesting - it connects fine to the Teleport proxy (which presents a host certificate) but then the error comes when the proxy tries to forward on the connection to the host running trusty.

I think this error is coming from here: https://github.com/gravitational/teleport/blob/5ed0df368bf88e8fc93951403ab9d04452a6a830/lib/srv/authhandlers.go#L271

What was the command you ran to generate the key (/etc/ssh/ssh_known_hosts) and certificate (/etc/ssh/ssh_known_hosts-cert.pub)?

I use auth/proxy roles on one server - u100.mysite.com with IP 192.168.33.33 (proxy’s address - tp.mysite.com:3080).
Command for generating a key and certificate:

sudo tctl auth sign --host=u100,u100.mysite.com,192.168.33.33,tp.mysite.com,tp,trusty,trusty.mysite.com,192.168.33.84 --format=openssh --out ssh_known_hosts

I also generated a separate key/certificate for each server, but the result was the same.

I think the problem is related to TrustedUserCAKeys
I deleted the TrustedUserCAKeys line from the sshd_config and checked the work with signed by Teleport HostKey and HostCertificate.
Then I enabled TrustedUserCAKeys in sshd_config and viewed the connection information output with the -vvv flag.

debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA-CERT SHA256:kKiBzIdnFNpDb6UhIByO0wRtoHStBbjgPxD+PfPDG2Y teleport:vagrant
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: RSA SHA256:kKiBzIdnFNpDb6UhIByO0wRtoHStBbjgPxD+PfPDG2Y teleport:vagrant
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: RSA SHA256:HGf01hlV2jGIsRnUOlvA37BDXlae4oFjW6DgCRxhG+I /home/jimdell/.ssh/id_rsa

It can’t use the teleport keys for some reason
here’s what I found in the server /var/log/auth.log log:

Oct 12 14:20:20 vagrant-ubuntu-trusty-64 sshd[4976]: Address 192.168.33.1 maps to workbook, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Oct 12 14:20:20 vagrant-ubuntu-trusty-64 sshd[4976]: error: Certificate critical option "teleport-roles" is not supported
Oct 12 14:20:22 vagrant-ubuntu-trusty-64 sshd[4976]: Connection closed by 192.168.33.1 [preauth]

I guess it doesn’t work with openssh version lower than 6.8:

In my ~/.tsh/keys/tp.example.com/vagrant-cert.pub :

        Extensions: 
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                teleport-roles UNKNOWN OPTION (len 38)
                teleport-route-to-cluster UNKNOWN OPTION (len 8)
                teleport-traits UNKNOWN OPTION (len 82)

Interesting - good research! It does indeed look like the certificate extensions weren’t treated as optional until version 6.9 (when the patch to fix this was merged). I will add this information to our OpenSSH docs - thank you!

As I understand it, the only solution is to update the OpenSSH version, right?

Yes, you’d need to update OpenSSH to version 6.9 or newer.

1 Like