Unable to connect to auth server

Hello! I have 3 servers:

  1. teleportmaster (uses teleport auth,node roles) with IP 192.168.33.30
  2. teleportproxy (uses teleport auth,node roles) with IP 192.168.33.34
  3. haproxy server with IP 192.168.33.32

Haproxy configuration:
I use several frontends and backends with the same configuration (for 3025 I use 192.168.33.30 IP on backend, on other ports use 192.168.33.34).
Example of frontend:

    mode tcp
    timeout client 500ss
    bind 192.168.33.32:3024
    use_backend teleport_tunnel_backend

Example of backend:

    mode tcp
    timeout server 500ss
    server teleportproxy 192.168.33.34:3024

For 3080 port:

    bind 192.168.33.32:80
    bind 192.168.33.32:443 ssl crt /etc/haproxy/ssl/

    acl is-url-client hdr_dom(host) -i my-site.example
    use_backend teleport_web if is-url-client

backend teleport_web
    http-request redirect scheme https if !{ ssl_fc }
    http-response set-header Strict-Transport-Security "max-age=31536000"
    server teleportproxy 192.168.33.34:3080 check ssl verify none

Teleport is started by the command:

/usr/local/bin/teleport start --config=/etc/teleport.yaml --pid-file=/var/run/teleport.pid

Configuration /etc/teleport/yaml on auth server:

teleport:
  auth_token: my-token
  auth_servers: [ "localhost:3025" ]
  log:
      output: /var/lib/teleport/teleport.log
      severity: INFO
auth_service:
  enabled: "yes"
  tokens:
  - proxy,node:my-token
  authentication:
      second_factor: otp
ssh_service:
  enabled: "yes"
proxy_service:
    enabled: "no"

Configuration /etc/teleport/yaml on proxy server:

teleport:
  auth_token: my-token
  auth_servers: [ "my-site.example:3025" ]
  log:
      output: /var/lib/teleport/teleport.log
      severity: INFO
ssh_service:
  enabled: "yes"
auth_service:
  enabled: "no"
proxy_service:
    enabled: "yes"
    listen_addr: 0.0.0.0:3023
    tunnel_listen_addr: 0.0.0.0:3024
    web_listen_addr: 0.0.0.0:3080
    public_addr: my-site.example:443
    ssh_public_addr: my-site.example:3023
    https_key_file: /var/lib/teleport/my-site.example.key
    https_cert_file: /var/lib/teleport/my-site.example.crt

both servers are added to the same cluster:

vagrant@teleportmaster:~$ sudo tctl nodes ls
Nodename       UUID                                 Address            Labels
-------------- ------------------------------------ ------------------ ------
teleportmaster 55107645-331f-447c-8b83-014e93617d29 127.0.0.1:3022            
teleportproxy  632b2b01-4ed0-4d6c-9095-706cf41ae2d1 192.168.33.32:3022

One user registered:

User    Allowed logins
------- --------------
vagrant vagrant

When try to log in, the user receives a message:

Enter password for Teleport user vagrant:
Enter your OTP token:
******
error: Get https://teleport.cluster.local/v2/authorities/host?load_keys=false: failed connecting to node . unable to connect to auth server

In proxy’s log file:

WARN [PROXY]     Subsystem request proxySubsys(cluster=default/teleportmaster, host=, port=) failed: unable to connect to auth server. id:1 local:192.168.33.34:3023 login:vagrant remote:192.168.33.32:33168 teleportUser:vagrant regular/sshserver.go:1256
ERRO [NODE]      unable to connect to auth server regular/sshserver.go:1434

clearing /var/lib/teleport doesn’t help. This configuration works when using auth,proxy roles on a single server. What am I doing wrong?

Thanks for the detailed post. I’ll be the first to admit that I’m not an expert in the use of HAProxy so it’s possible that there’s something obvious that I’m missing, but I think we have enough to go on for now.

  1. Can you please add --debug to your regular tsh login command and post the output here? There is likely to be some extra output there which may assist.

  2. Can you please turn up the severity in your teleport.yaml files to DEBUG, restart your Teleport processes and try the process again, posting the logs from the proxy and auth servers here?

  3. What IP is in the DNS for my-site.example? Presumably the IP address of the HAProxy server? Can you see logs in HAProxy showing an incoming connection from the proxy being routed to on the auth server backend?

Yes, for my-site.example was used haproxy’s IP 192.168.33.32 on all servers. I excluded the haproxy server from this chain and checked work again (cleared /var/lib/teleport) and change IP in DNS on 192.168.33.34 (teleport proxy’s).

Slightly changed the configuration /etc/teleport.yaml:

  auth_servers: [ "teleportmaster:3025" ]
[...]
proxy_service:
[...]
    public_addr: my-site.example:3080
[...]

The problem is still relevant.

Proxy's logs with DEBUG in severity:

DEBU [SSH:PROXY] Incoming connection 192.168.33.1:49014 -> 192.168.33.34:3023 vesion: SSH-2.0-Go. sshutils/server.go:430
DEBU [PROXY]     Handling request subsystem, want reply true. id:1 local:192.168.33.34:3023 login:vagrant remote:192.168.33.1:49014 teleportUser:vagrant regular/sshserver.go:1140
DEBU [NODE]      parse_proxy_subsys("proxy:@teleportmaster") regular/proxy.go:70
DEBU [NODE]      newProxySubsys({default   teleportmaster}). regular/proxy.go:172
DEBU [PROXY]     Subsystem request: proxySubsys(cluster=default/teleportmaster, host=, port=). id:1 local:192.168.33.34:3023 login:vagrant remote:192.168.33.1:49014 teleportUser:vagrant regular/sshserver.go:1254
DEBU [SUBSYSTEM] Starting subsystem trace.fields:map[dst:192.168.33.34:3023 src:192.168.33.1:49014] regular/proxy.go:201
DEBU [KEEPALIVE] Starting keep-alive loop with with interval 5m0s and max count 3. srv/keepalive.go:67
WARN [PROXY]     Subsystem request proxySubsys(cluster=default/teleportmaster, host=, port=) failed: unable to connect to auth server. id:1 local:192.168.33.34:3023 login:vagrant remote:192.168.33.1:49014 teleportUser:vagrant regular/sshserver.go:1256
ERRO [NODE]      unable to connect to auth server regular/sshserver.go:1434
DEBU [SSH:PROXY] Closed connection 192.168.33.1:49014. sshutils/server.go:432

tsh with --debug

$ tsh login --debug --login=vagrant --user=vagrant --proxy=my-site.example:3080
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/tmp/ssh-BkKSQFkEe5wI/agent.1039" client/api.go:2201
DEBU [CLIENT]    not using loopback pool for remote proxy addr: my-site.example:3080 client/api.go:2162
DEBU [CLIENT]    HTTPS client init(proxyAddr=my-site.example:3080, insecure=false) client/weblogin.go:295
Enter password for Teleport user vagrant:
Enter your OTP token:
897779
DEBU [CLIENT]    not using loopback pool for remote proxy addr: my-site.example:3080 client/api.go:2162
DEBU [CLIENT]    HTTPS client init(proxyAddr=my-site.example:3080, insecure=false) client/weblogin.go:295
DEBU [KEYAGENT]  Adding CA key for teleportmaster client/keyagent.go:243
DEBU [KEYSTORE]  Adding known host teleportmaster with key: SHA256:klnQgOp9l8//VXNZmUTs+dGUBZBR2utubjiowUqMy4k client/keystore.go:381
INFO [CLIENT]    Connecting proxy=my-site.example:3023 login='vagrant' method=0 client/api.go:1633
DEBU [KEYAGENT]  Validated host my-site.example:3023. client/keyagent.go:285
INFO [CLIENT]    Successful auth with proxy my-site.example:3023 client/api.go:1623
DEBU [KEYSTORE]  Returning SSH certificate "/home/jimdell/.tsh/keys/my-site.example/vagrant-cert.pub" valid until "2020-09-01 21:54:46 +0400 +04", TLS certificate "/home/jimdell/.tsh/keys/my-site.example/vagrant-x509.pem" valid until "2020-09-01 17:54:46 +0000 UTC". client/keystore.go:277
DEBU [CLIENT]    Client  is connecting to auth server on cluster "teleportmaster". client/client.go:435
ERROR REPORT:
Original Error: *trace.ConnectionProblemError failed connecting to node . unable to connect to auth server
Stack Trace:
        /gopath/src/github.com/gravitational/teleport/lib/httplib/httplib.go:110 github.com/gravitational/teleport/lib/httplib.ConvertResponse
        /gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:349 github.com/gravitational/teleport/lib/auth.(*Client).Get
        /gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:539 github.com/gravitational/teleport/lib/auth.(*Client).GetCertAuthorities
        /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1888 github.com/gravitational/teleport/lib/client.(*TeleportClient).GetTrustedCA
        /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1898 github.com/gravitational/teleport/lib/client.(*TeleportClient).UpdateTrustedCA
        /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1778 github.com/gravitational/teleport/lib/client.(*TeleportClient).Login
        /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:481 main.onLogin
        /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:351 main.Run
        /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:188 main.main
        /opt/go/src/runtime/proc.go:212 runtime.main
        /opt/go/src/runtime/asm_amd64.s:1358 runtime.goexit
User Message: Get https://teleport.cluster.local/v2/authorities/host?load_keys=false: failed connecting to node . unable to connect to auth server

Hi - could you possibly share your Vagrantfile so I can try and reproduce this locally?

I didn’t use a specialized Vagrantfile.
I spent a little time creating it now :slight_smile:
Please download the my repository
I use my own domain and an ssl certificate for it. For security reasons, I changed the domain and didn’t post the certificate.
Please change the settings for /etc/teleport.yaml (you also need to add certificates to the /var/lib/teleport directory)

1 Like

Thanks - I can reproduce this locally. I’ll try and figure out what’s causing it.

1 Like

Hi - thanks for your patience. I figured out what was causing the issue with the aid of tcpdump.

As you’re running Teleport inside Vagrant with a custom network, your machines are multi-homed (i.e. they have multiple network interfaces with different IPs). Teleport picks the first available IP by default, so it looks like it was trying to communicate using the host interface’s IP (10.0.2.15) rather the custom Vagrant IP that you assigned (192.168.33.35).

You can resolve this by adding a line to your /etc/teleport.yaml on the auth server and restarting Teleport:

teleport:
  advertise_ip: 192.168.33.35

Please let me know if this works for you!

1 Like

It works now. Thank you very much! :slight_smile:

1 Like

Great news, you’re welcome!

1 Like