Teleport-auth inside k8s cluster (and behind traefik)

Hello there!

I’m trying to deploy teleport infrastructure in my orgranisation. Current vision is that both teleport-auth-node and teleport-proxy node need to be deployed in our infrastructure k8s-cluster. I’ve successfully deployed these instances of teleport. I was able to connect proxy-instance to the auth-instance successfully inside cluster network (direct connection, essentially). But I’m struggling to connect teleport-ssh-nodes to that cluster.

The node I’m trying to add to teleport-cluster is outside of k8s-cluster. Teleport-auth and teleport-proxy instances inside k8s are behind Traefik (which is behind load-balancer and serves the purpose as single entry point to all of the deployments inside k8s cluster). More to the point, Traefik serves all it’s routes with configured TLS-certificate of my organisation.

I’ve tried two ways of configuring teleport-auth-node:

1

Teleport-auth-node is listening on 0.0.0.0:443, advertise_ip and public_addr are pointing to FQDN (teleport-auth-test.mindbox.ru:443). When I’m trying to authenticate remote ssh-node, I get the following error in teleport-auth-node logs:

DEBU [AUTH] ClientCertPool -> cert(teleport-vassyutovich issued by teleport-vassyutovich:151509504299134967317085424432469306525) auth/middleware.go:359
DEBU [AUTH] ClientCertPool -> cert(teleport-vassyutovich issued by teleport-vassyutovich:146673723890650197902012578441866342957) auth/middleware.go:359
DEBU [AUTH:1] Server certificate cert(999cf689-1631-44b6-9797-7ec135a2ad25.teleport-vassyutovich issued by teleport-vassyutovich:151509504299134967317085424432469306525). auth/middleware.go:164
2020-06-09 18:04:27.401502 I | http: TLS handshake error from 10.156.132.58:34602: remote error: tls: bad certificate

2

Teleport-auth-node is configured to listen on “standard” 0.0.0.0:3025, advertise_ip and public_addr are pointing to FQDN (teleport-auth-test.mindbox.ru:443). When I’m trying to authenticate remote ssh-node, I get the following error in teleport-auth-node logs:

DEBU [MX:1] Detected an HTTP request. If this is for a health check, use an HTTPS request instead. multiplexer/multiplexer.go:240

Please, help me (:

Source code for both configurations is available here (watch for tags):

I’m unable to update initial post due to “new user” limitations

I think your biggest issue is likely to be that with Teleport, only port 3080 expects to receive HTTPS traffic. Port 3025 (auth server) uses GRPC (wrapped in HTTP/2) and port 3023/3024 are actually SSH traffic. 3023 is for client connections to the cluster, and 3024 is for nodes to create reverse tunnels back into the cluster.

If I understand correctly, Traefik is just an HTTPS proxy, right? It’s basically a replacement for a standard Kubernetes nginx-based Ingress? From an initial glance, It looks like Traefik is providing a layer 7 (HTTP) load balancer - for Teleport to work properly, you would need to be using a layer 4 (TCP) load balancer for ports 3023, 3024 and 3025.

Yeah, thanks friend! I realized that yesterday in the middle of the night sleep. And spent entire day reworking everything and haven’t got any time to update my topic. You’re absolutely correct. In fact, traefik can proxy TCP, but I found this too much of hussle now. And I ended up using just two separate load-balancers for 3023, 3024, 3025 ports. And only 3080 I’ve put behind traefik.

I will post an update to this topic and, probably, update source code in linked repo for the future generations.

Morale of the story — be more attentive to the documentation.

1 Like

Great news! Thanks for the update. I didn’t realise Traefik could be a TCP proxy too - as you say, it’s probably easier to just use a regular LoadBalancer instead.