Client sent unsupported cluster name ... expected suffix teleport.cluster.local

Introduction

I first installed Teleport on a dedicated node, and joined all the my nodes to Teleport and got the SSH proxying working. Perfect.
Those nodes are used in order to run a Kubernetes cluster, so when I tried to add the kubernetes integration, I had the issue that Teleport wasn’t able to reach Kubernetes in this configuration.

Where I am now

I’ve then decided to deploy Teleport on a Kubernetes master node so that it can access Kubernetes, and it does, and then deploy teleport to the other nodes for the SSH tunneling.

But the other nodes can’t join the Teleport cluster and I see the following error in the node where Teleport daemon (all roles) runs:

Client sent unsupported cluster name "dev-k8s-master-1.domain.co", what resulted in error unrecognized name, expected suffix teleport.cluster.local, got "dev-k8s-master-1.domain.co".

And looking at the logs of a node trying to join the Teleport daemon, I see the following :

Oct 23 08:27:51 dev-k8s-worker-1 teleport[26858]: INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:349
Oct 23 08:27:51 dev-k8s-worker-1 teleport[26858]: ERRO [PROC:1]    Node failed to establish connection to cluster: remote error: tls: internal error. service/connect.go:65

When I try to curl the Teleport daemon I got this:

curl https://dev-k8s-master-1.domain.co:3025
curl: (35) error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error

Tried to reset Teleport

I tried already to stop Teleport on all the nodes, remove the /var/lib/teleport/ folder and restart Teleport, so that it regenerates the certificates, but I have the same errors.

Without Kubernetes

I’ve also tried disabling the Kubernetes integration:

  1. remove the entire kubernetes block from the /etc/teleport.yaml file
  2. remove the /var/lib/teleport folder
  3. restart teleport

but it didn’t fixed the issue.

The Teleprot bootstrap

Here is the entire Teleport logs when bootstraping the daemon from scratch :

root@dev-k8s-master-1:~# /usr/local/bin/teleport start --roles=auth,node,proxy --config=/etc/teleport.yaml
INFO [PROC]      Generating new host UUID: 6e1294a4-f928-4777-b702-a27e04e100f0. service/service.go:544
INFO [AUTH]      Updating cluster configuration: StaticTokens([ProvisionToken(Roles=[Node], Expires=never)]). auth/init.go:270
INFO [AUTH]      Updating cluster configuration: AuthPreference(Type="local",SecondFactor="otp"). auth/init.go:277
INFO [AUTH]      Created namespace: "default". auth/init.go:284
INFO [AUTH]      Created default admin role: "admin". auth/init.go:290
INFO [AUTH]      First start: generating user certificate authority. auth/init.go:301
INFO [AUTH]      First start: generating host certificate authority. auth/init.go:356
INFO [AUTH]      Auth server is running periodic operations. auth/init.go:424
INFO [CA]        Generating TLS certificate {0x36257a0 0xc000598110 CN=6e1294a4-f928-4777-b702-a27e04e100f0.staging,O=Admin,POSTALCODE=null,STREET= 2029-10-20 07:02:50.934313742 +0000 UTC [dev-k8s-master-1.domain.co dev-k8s-master-1.domain.co *.teleport.cluster.local teleport.cluster.local]}. common_name:6e1294a4-f928-4777-b702-a27e04e100f0.staging dns_names:[dev-k8s-master-1.domain.co dev-k8s-master-1.domain.co *.teleport.cluster.local teleport.cluster.local] locality:[] not_after:2029-10-20 07:02:50.934313742 +0000 UTC org:[Admin] org_unit:[] tlsca/ca.go:203
INFO [PROC]      Admin has obtained credentials to connect to cluster. service/connect.go:377
INFO [PROC:1]    The process has successfully wrote credentials and state of Admin to disk. service/connect.go:417
INFO [PROC:1]    Service auth is creating new listener on 0.0.0.0:3025. service/signals.go:213
INFO [AUTH:1]    Starting Auth service with PROXY protocol support. service/service.go:1044
WARN [AUTH:1]    Configuration setting auth_service/advertise_ip is not set. guessing 10.244.0.1:3025. service/service.go:1119
INFO [AUTH]      Auth service is starting on 0.0.0.0:3025. utils/cli.go:177
[AUTH]    Auth service is starting on 0.0.0.0:3025.
INFO [CA]        Generating TLS certificate {0x36257a0 0xc0005c8130 CN=6e1294a4-f928-4777-b702-a27e04e100f0.staging,O=Proxy,POSTALCODE=null,STREET= 2029-10-20 07:02:51.315203483 +0000 UTC [dev-k8s-master-1.domain.co remote.kube.proxy.teleport.cluster.local]}. common_name:6e1294a4-f928-4777-b702-a27e04e100f0.staging dns_names:[dev-k8s-master-1.domain.co remote.kube.proxy.teleport.cluster.local] locality:[] not_after:2029-10-20 07:02:51.315203483 +0000 UTC org:[Proxy] org_unit:[] tlsca/ca.go:203
INFO [PROC]      Proxy has obtained credentials to connect to cluster. service/connect.go:377
INFO [PROC:1]    The process has successfully wrote credentials and state of Proxy to disk. service/connect.go:417
INFO [PROC:1]    Service proxy:web is creating new listener on 0.0.0.0:3080. service/signals.go:213
INFO [PROC:1]    Service proxy:tunnel is creating new listener on 0.0.0.0:3024. service/signals.go:213
INFO [PROXY]     Reverse tunnel service is starting on 0.0.0.0:3024. utils/cli.go:177
[PROXY]   Reverse tunnel service is starting on 0.0.0.0:3024.
INFO [PROXY:SER] Starting on 0.0.0.0:3024 using cache that will expire after connection to database is lost after 20h0m0s, will cache frequently accessed items for 2s service/service.go:1992
INFO [PROXY:SER] Using TLS cert /etc/ssl/dev-k8s-master-1.domain.co.crt, key /etc/ssl/dev-k8s-master-1.domain.co.key service/service.go:2048
INFO [PROC:1]    Service proxy:ssh is creating new listener on 0.0.0.0:3023. service/signals.go:213
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1570
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1570
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1570
INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1570
INFO [PROXY]     Web proxy service is starting on 0.0.0.0:3080. utils/cli.go:177
[PROXY]   Web proxy service is starting on 0.0.0.0:3080.
INFO [PROXY:SER] Web proxy service is starting on 0.0.0.0:3080. service/service.go:2061
INFO [PROXY]     SSH proxy service is starting on 0.0.0.0:3023. utils/cli.go:177
[PROXY]   SSH proxy service is starting on 0.0.0.0:3023.
INFO [PROXY:SER] SSH proxy service is starting on 0.0.0.0:3023 service/service.go:2103
INFO [PROXY:AGE] Starting reverse tunnel agent pool. service/service.go:2129
INFO [CA]        Generating TLS certificate {0x36257a0 0xc0004ba220 CN=6e1294a4-f928-4777-b702-a27e04e100f0.staging,O=Node,POSTALCODE=null,STREET= 2029-10-20 07:02:51.598358434 +0000 UTC [dev-k8s-master-1.domain.co 127.0.0.1]}. common_name:6e1294a4-f928-4777-b702-a27e04e100f0.staging dns_names:[dev-k8s-master-1.domain.co 127.0.0.1] locality:[] not_after:2029-10-20 07:02:51.598358434 +0000 UTC org:[Node] org_unit:[] tlsca/ca.go:203
INFO [PROC]      Node has obtained credentials to connect to cluster. service/connect.go:377
INFO [PROC:1]    The process has successfully wrote credentials and state of Node to disk. service/connect.go:417
INFO [PROC:1]    Service node is creating new listener on 0.0.0.0:3022. service/signals.go:213
INFO [NODE:1]    Service is starting on 0.0.0.0:3022 cache that will expire after connection to database is lost after 20h0m0s, will cache frequently accessed items for 2s. service/service.go:1462
INFO [NODE]      Service is starting on 0.0.0.0:3022. utils/cli.go:177
[NODE]    Service is starting on 0.0.0.0:3022.
INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:433

Now the entire Teleport logs from a joining node:

root@dev-k8s-worker-1:~# /usr/local/bin/teleport start --roles=node --config=/etc/teleport.yaml
INFO [PROC]      Generating new host UUID: 72a2adf2-e766-4b2b-ac85-ca181c73328b. service/service.go:544
INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:349
ERRO [PROC:1]    Node failed to establish connection to cluster: remote error: tls: internal error. service/connect.go:65

Which write the following logs on the daemon node:

WARN [AUTH:1]    Client sent unsupported cluster name "dev-k8s-master-1.domain.co", what resulted in error unrecognized name, expected suffix teleport.cluster.local, got "dev-k8s-master-1.domain.co". logrus/entry.go:188
2019-10-23 09:02:58.637517 I | http: TLS handshake error from 167.86.118.36:45600: access is denied

Can you please help me to understand what is happening and how to solve it?

teleport.cluster.local looks like a Kubernetes internal DNS name while I’m deploying teleport outside Kubernetes.

Could it be that there is a bug when deploying teleport outside of Kubernetes?

Actually the error is misleading, there is no issue with certificate or what so ever, it is an issue with the CA pin.

So the on the teleport server, I’m setting a ca_pin in the /etc/teleport.yaml, removing the /var/lib/teleport folder and start teleprot.

When doing tctl status it shows a complete different CA pin than the one I configured, so obviously the node can’t join the cluster with a wrong CA pin.

Either I misunderstood the doc and the ca_pin attribute is only for the nodes to join the server, either there is a bug where the ca_pin is ignored by the teleport server.

Okay so reading again the doc carefully, it looks like the CA pin is a hash of the server’s certificate so can’t be forced obviously.

I’m updating my provisioning cookbook so that it retrieves the CA pin instead of generating it.

I confirm updating my cookbook in order to retrieve the CA pin instead of generating it fixed my issue.

1 Like

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.