Tsh not using custom proxy port

Before open an issue on github, I would like to see whats going on. I have deployed two EC2 instances and they was configured to act as auth and proxy. The proxy machine is behind and AWS ALB, that is responsible to terminate SSL traffic, forwarding all traffic on 443 HTTPS to 3080 HTTP plus an AWS NLB, that is responsible to forward TCP ports 3023 and 3024 to proxy host.

The auth server doesn’t have any load balancer in front of it. Booth proxy and auth are in a private subnet and they don’t have any public IP addresses associated with them.

I’ve added a third EC2 instance to act as a node (for testing SSH connection). The SSH connection works pretty well using proxy’s web interface. But, when I try it trough tsh, no. It seems that after I provide valid credentials and OTP token, tsh is trying to connect to proxy’s address through TCP 3080 instead of TCP 443.

This is what is happening:

$ tsh -d login --proxy=auth.eu-west-1.my.domain.com:443,3023 --user=root root
INFO [CLIENT]    no host login given. defaulting to galindro client/api.go:770
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/tmp/ssh-XEKpkYzi47Br/agent.2715" client/api.go:2000
DEBU [CLIENT]    not using loopback pool for remote proxy addr: auth.eu-west-1.my.domain.com:443 client/api.go:1961
DEBU [CLIENT]    HTTPS client init(proxyAddr=auth.eu-west-1.my.domain.com:443, insecure=false) client/weblogin.go:252
Enter password for Teleport user root:
Enter your OTP token:
777777
DEBU [CLIENT]    not using loopback pool for remote proxy addr: auth.eu-west-1.my.domain.com:3080 client/api.go:1961
DEBU [CLIENT]    HTTPS client init(proxyAddr=auth.eu-west-1.my.domain.com:3080, insecure=false) client/weblogin.go:252

ERROR REPORT:
Original Error: *trace.ConnectionProblemError dial tcp 11.111.11.111:3080: i/o timeout
Stack Trace:
    /gopath/src/github.com/gravitational/teleport/lib/httplib/httplib.go:110 github.com/gravitational/teleport/lib/httplib.ConvertResponse
    /gopath/src/github.com/gravitational/teleport/lib/client/https_client.go:76 github.com/gravitational/teleport/lib/client.(*WebClient).PostJSON
    /gopath/src/github.com/gravitational/teleport/lib/client/weblogin.go:404 github.com/gravitational/teleport/lib/client.SSHAgentLogin
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1877 github.com/gravitational/teleport/lib/client.(*TeleportClient).directLogin
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1816 github.com/gravitational/teleport/lib/client.(*TeleportClient).localLogin
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1619 github.com/gravitational/teleport/lib/client.(*TeleportClient).Login
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:452 main.onLogin
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:337 main.Run
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:184 main.main
    /opt/go/src/runtime/proc.go:209 runtime.main
    /opt/go/src/runtime/asm_amd64.s:1338 runtime.goexit
User Message: Post https://auth.eu-west-1.my.domain.com:3080/v1/webapi/ssh/certs: dial tcp 11.111.11.111:3080: i/o timeout

In the proxy, I can’t see any logs that helps me to find the problem (even starting it in DEBUG log level). In auth server neither.

These are the teleport.yml configs:

# AUTH SERVER
teleport:
  connection_limits:
    max_connections: 1000
    max_users: 250
  log:
    output: stderr
    severity: INFO
  storage:
    type: dynamodb
    region: eu-west-1
    table_name: test-teleport
    audit_events_uri: ['dynamodb://test-teleport-events', 'stdout://']
    audit_sessions_uri: 's3://my-teleport-sessions'
auth_service:
  enabled: yes
  cluster_name: "eu-west-1"
  authentication:
    type: local
    second_factor: otp
    u2f:
      app_id: https://auth.eu-west-1.my.domain.com
  listen_addr: 0.0.0.0:3025
  tokens:
    - "proxy,node:node/proxy-token"
    - "auth:auth-token"
  session_recording: "node"
  client_idle_timeout: "15m"
  disconnect_expired_cert: no


# PROXY SERVER
teleport:
  auth_token: "node/proxy-token"
  ca_pin: "my-ca-pin"
  auth_servers:
    - 10.100.3.201:3025
  connection_limits:
    max_connections: 1000
    max_users: 250
  log:
    output: stderr
    severity: INFO
proxy_service:
  enabled: yes
  listen_addr: 0.0.0.0:3023
  tunnel_listen_addr: 0.0.0.0:3024
  web_listen_addr: 0.0.0.0:3080
  public_addr: auth.eu-west-1.my.domain.com
  ssh_public_addr: ssh.eu-west-1.my.domain.com

You’ll need to explicitly set the ports within public_addr and ssh_public_addr in the Teleport config if they’re different from the defaults. Change public_addr to auth.eu-west-1.my.domain.com:443, restart Teleport and try again.

@galindro Please let us know how you get on!

Hey @gus, sorry by my late answer. I was on vacation last week. After change it, I could fix the mentioned error on this thread. Now I’m getting other error:

$ tsh -d login --proxy=auth.eu-west-1.my.domain.com:443,3023 --user=root root

INFO [CLIENT]    no host login given. defaulting to galindro client/api.go:770
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/tmp/ssh-AwI47aajXIUF/agent.2504" client/api.go:2000
DEBU [KEYSTORE]  Returning SSH certificate "/home/galindro/.tsh/keys/auth.eu-west-1.my.domain.com/root-cert.pub" valid until "2019-10-21 23:29:57 +0200 CEST", TLS certificate "/home/galindro/.tsh/keys/auth.eu-west-1.my.domain.com/root-x509.pem" valid until "2019-10-21 21:29:57 +0000 UTC". client/keystore.go:262
INFO [KEYAGENT]  Loading key for "root" client/keyagent.go:108
DEBU [CLIENT]    not using loopback pool for remote proxy addr: auth.eu-west-1.my.domain.com:443 client/api.go:1961
DEBU [CLIENT]    HTTPS client init(proxyAddr=auth.eu-west-1.my.domain.com:443, insecure=false) client/weblogin.go:252
Enter password for Teleport user root:
Enter your OTP token:
351532
DEBU [CLIENT]    not using loopback pool for remote proxy addr: auth.eu-west-1.my.domain.com:443 client/api.go:1961
DEBU [CLIENT]    HTTPS client init(proxyAddr=auth.eu-west-1.my.domain.com:443, insecure=false) client/weblogin.go:252
DEBU [KEYAGENT]  Adding CA key for eu-west-1 client/keyagent.go:238
DEBU [KEYSTORE]  Adding known host eu-west-1 with key: SHA256:Zj4yCoQHj1JYl2PRVZysXu8LFY5bCVNoqm7NyZiGtzE client/keystore.go:355
INFO [CLIENT]    Connecting proxy=ssh.eu-west-1.my.domain.com:3023 login='root' method=0 client/api.go:1539
DEBU [KEYAGENT]  Validated host ssh.eu-west-1.my.domain.com:3023. client/keyagent.go:280
INFO [CLIENT]    Successful auth with proxy ssh.eu-west-1.my.domain.com:3023 client/api.go:1545
DEBU [KEYSTORE]  Adding trusted cluster certificate authority "SERIALNUMBER=169780797355293687129582411851477107586,CN=eu-west-1,O=eu-west-1" to trusted pool. client/keystore.go:328
DEBU [KEYSTORE]  Returning SSH certificate "/home/galindro/.tsh/keys/auth.eu-west-1.my.domain.com/root-cert.pub" valid until "2019-10-21 23:31:03 +0200 CEST", TLS certificate "/home/galindro/.tsh/keys/auth.eu-west-1.my.domain.com/root-x509.pem" valid until "2019-10-21 21:31:03 +0000 UTC". client/keystore.go:262
DEBU [CLIENT]    Client  is connecting to auth server on cluster "eu-west-1". client/client.go:369

ERROR REPORT:
Original Error: *trace.ConnectionProblemError failed connecting to node . error: unable to connect to auth server
Stack Trace:
    /gopath/src/github.com/gravitational/teleport/lib/httplib/httplib.go:110 github.com/gravitational/teleport/lib/httplib.ConvertResponse
    /gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:341 github.com/gravitational/teleport/lib/auth.(*Client).Get
    /gopath/src/github.com/gravitational/teleport/lib/auth/clt.go:531 github.com/gravitational/teleport/lib/auth.(*Client).GetCertAuthorities
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1719 github.com/gravitational/teleport/lib/client.(*TeleportClient).GetTrustedCA
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1726 github.com/gravitational/teleport/lib/client.(*TeleportClient).UpdateTrustedCA
    /gopath/src/github.com/gravitational/teleport/lib/client/api.go:1692 github.com/gravitational/teleport/lib/client.(*TeleportClient).Login
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:452 main.onLogin
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:337 main.Run
    /gopath/src/github.com/gravitational/teleport/tool/tsh/tsh.go:184 main.main
    /opt/go/src/runtime/proc.go:209 runtime.main
    /opt/go/src/runtime/asm_amd64.s:1338 runtime.goexit
User Message: Get https://teleport.cluster.local/v2/authorities/host?load_keys=false: failed connecting to node . error: unable to connect to auth server

It seems that tsh client is trying to connect directly with auth server. Is it really necessary? I though that auth server needs to be accessible only by proxy servers not from tsh client.

# PROXY LOG
DEBU [PROXY]     conn(100.100.100.100:43902->172.17.0.2:3023, user=root) auth attempt fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:8vXQJv68aQ+NaOwamgLk5H1M3sDJWvVvievv9vbmKe8 local:172.17.0.2:3023 remote:100.100.100.100:43902 user:root srv/authhandlers.go:151
DEBU [PROXY]     conn(100.100.100.100:43902->172.17.0.2:3023, user=root) auth attempt with key ssh-rsa-cert-v01@openssh.com SHA256:8vXQJv68aQ+NaOwamgLk5H1M3sDJWvVvievv9vbmKe8, &ssh.Certificate{Nonce:[]uint8{0xa1, 0x1c, 0xeb, 0x2a, 0xf7, 0xfb, 0x3c, 0x3e, 0x1a, 0xef, 0x2b, 0xe4, 0x10, 0xac, 0xd1, 0x66, 0x73, 0x4b, 0x45, 0x67, 0x76, 0x2b, 0xf2, 0x1e, 0xb8, 0x8a, 0xca, 0x30, 0x90, 0x3c, 0xcb, 0x70}, Key:(*ssh.rsaPublicKey)(0xc00053d640), Serial:0x0, CertType:0x1, KeyId:"root", ValidPrincipals:[]string{"root"}, ValidAfter:0x5dad7e6e, ValidBefore:0x5dae276a, Permissions:ssh.Permissions{CriticalOptions:map[string]string{}, Extensions:map[string]string{"permit-agent-forwarding":"", "permit-port-forwarding":"", "permit-pty":"", "teleport-roles":"{\"version\":\"v1\",\"roles\":[\"admin\"]}", "teleport-traits":"{\"kubernetes_groups\":null,\"logins\":[\"root\"]}"}}, Reserved:[]uint8{}, SignatureKey:(*ssh.rsaPublicKey)(0xc00053d680), Signature:(*ssh.Signature)(0xc00002a1b0)} fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:8vXQJv68aQ+NaOwamgLk5H1M3sDJWvVvievv9vbmKe8 local:172.17.0.2:3023 remote:100.100.100.100:43902 user:root srv/authhandlers.go:151
DEBU [PROXY]     Successfully authenticated fingerprint:ssh-rsa-cert-v01@openssh.com SHA256:8vXQJv68aQ+NaOwamgLk5H1M3sDJWvVvievv9vbmKe8 local:172.17.0.2:3023 remote:100.100.100.100:43902 user:root srv/authhandlers.go:195
DEBU [SSH:PROXY] Incoming connection 100.100.100.100:43902 -> 172.17.0.2:3023 vesion: SSH-2.0-Go. sshutils/server.go:425
DEBU [KEEPALIVE] Starting keep-alive loop with with interval 5m0s and max count 3. srv/keepalive.go:67
DEBU [PROXY]     Handling request subsystem, want reply true. id:1 idle:15m0s local:172.17.0.2:3023 login:root remote:100.100.100.100:43902 teleportUser:root regular/sshserver.go:1080
DEBU [NODE]      parse_proxy_subsys("proxy:@eu-west-1") regular/proxy.go:70
DEBU [NODE]      newProxySubsys({default   eu-west-1 0xc000547d40 0xc00074ef00}). regular/proxy.go:172
DEBU [PROXY]     Subsystem request: proxySubsys(cluster=default/eu-west-1, host=, port=). id:1 idle:15m0s local:172.17.0.2:3023 login:root remote:100.100.100.100:43902 teleportUser:root regular/sshserver.go:1206
DEBU [SUBSYSTEM] Starting subsystem trace.fields:map[dst:172.17.0.2:3023 src:100.100.100.100:43902] regular/proxy.go:201
WARN [PROXY]     Subsystem request proxySubsys(cluster=default/eu-west-1, host=, port=) failed: unable to connect to auth server. id:1 idle:15m0s local:172.17.0.2:3023 login:root remote:100.100.100.100:43902 teleportUser:root regular/sshserver.go:1208
ERRO [NODE]      unable to connect to auth server regular/sshserver.go:1389
DEBU [PROXY]     Releasing associated resources - context has been closed. id:1 idle:15m0s local:172.17.0.2:3023 login:root remote:100.100.100.100:43902 teleportUser:root srv/monitor.go:184
DEBU [SSH:PROXY] Closed connection 100.100.100.100:43902. sshutils/server.go:427

# AUTH LOG
DEBU [DYNAMODB]  Got 0 stream shard records. dynamo/shards.go:172
DEBU [DYNAMODB]  Got 0 stream shard records. dynamo/shards.go:172
{"code":"T1000I","event":"user.login","method":"local","success":true,"time":"2019-10-21T09:47:22Z","uid":"20423bcc-1687-41f2-8b71-200107f49e41","user":"root"}DEBU [KEYGEN]    generated user key for [root] with expiry on (1571694442) 2019-10-21 21:47:22.20115193 +0000 UTC native/native.go:258
INFO [CA]        Generating TLS certificate {0x36257a0 0xc001036600 CN=root,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"logins\":[\"root\"]},STREET=,L=root 2019-10-21 21:47:22.208623691 +0000 UTC []}. common_name:root dns_names:[] locality:[root] not_after:2019-10-21 21:47:22.208623691 +0000 UTC org:[admin] org_unit:[] tlsca/ca.go:203
DEBU [DYNAMODB]  Got 2 stream shard records. dynamo/shards.go:172
DEBU [DYNAMODB]  Got 0 stream shard records. dynamo/shards.go:172
DEBU [DYNAMODB]  Got 0 stream shard records. dynamo/shards.go:172

@gus did you have enough time to check my last comment?

Hi @galindro - sorry for the delay in response.

You are correct that the tsh client does not need to be able to connect to the auth server itself, just to ports 3080 (web/API) and 3023 (SSH traffic) on the proxy server.

I believe that message you’re seeing (Client is connecting to auth server on cluster "eu-west-1".) actually comes from the proxy server rather than the tsh client. The proxy instantiates a ‘client’ of its own to connect to Teleport’s internal GRPC API on the auth server to get a certificate issued.

Can you check that firewall rules/security groups permit the proxy server to open connections back to the auth server on port 3025?

Hello @gus. Yes, it is open. It is working good in the web console. Only tsh isn’t working. Maybe we can make a remote session to see what is happening. I can share with you my screen…

@gus, one thing that I forgot toi mention is: I’m using docker to spin-up proxy and auth servers. I did not spin-up them on the host network because I followed what was done here: https://github.com/gravitational/teleport/blob/master/docker/docker-compose.yml, with some little modifications: instead of using a fixed overlay network with fixed ip address, I have just used the default bridge network and exposed all necessary ports to the container. This is my docker inspect on booth auth and proxy containers. Each one is running on different EC2 machines.

One thing really strange is: if you see the trace error report from the tsh command line, it seems that proxy server is trying to connect to itself on port 3025, because actually 172.17.0.2 is the ip address from proxy docker container. Yet, if you check the proxy inspect, you will see that it was initialized with --role=proxy and its configuration is pointing to the auth server EC2 ip address:

auth_servers:
  - 10.100.3.201:3025

tsh trace

ERROR REPORT:
Original Error: *trace.ConnectionProblemError failed connecting to node . 
ERROR REPORT:
Original Error: *trace.ConnectionProblemError dial tcp 172.17.0.2:3025: connect: connection refused
Stack Trace:
    /gopath/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:165 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).DialAuthServer
    /gopath/src/github.com/gravitational/teleport/lib/srv/regular/proxy.go:247 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).proxyToSite
    /gopath/src/github.com/gravitational/teleport/lib/srv/regular/proxy.go:239 github.com/gravitational/teleport/lib/srv/regular.(*proxySubsys).Start
    /gopath/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1206 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSubsystem
    /gopath/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1083 github.com/gravitational/teleport/lib/srv/regular.(*Server).dispatch
    /gopath/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1048 github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSessionRequests
    /opt/go/src/runtime/asm_amd64.s:1338 runtime.goexit
User Message: unable to connect to auth server

docker inspect proxy

[
    {
        "Id": "67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c",
        "Created": "2019-10-28T10:05:38.329086711Z",
        "Path": "/usr/bin/dumb-init",
        "Args": [
            "teleport",
            "start",
            "-c",
            "/etc/teleport/teleport.yaml",
            "--insecure-no-tls",
            "--roles=proxy"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 30293,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-10-28T10:05:39.183259982Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:b74994c7b350ad113c0cd4bcdef1d807c844c527e9a1c26cb34100b0a88a7a68",
        "ResolvConfPath": "/var/lib/docker/containers/67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c/hostname",
        "HostsPath": "/var/lib/docker/containers/67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c/hosts",
        "LogPath": "/var/lib/docker/containers/67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c/67fdd1ee58242776344c84b988994060c9eb1d7a078f5aa7a25ff80973899b6c-json.log",
        "Name": "/teleport-proxy",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/root/teleport.yml:/etc/teleport/teleport.yaml:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "3023/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3023"
                    }
                ],
                "3024/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3024"
                    }
                ],
                "3080/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3080"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/02632a7e45ef0b7272a56ce4e2248eb2feab6ab7057d1fb4cf5b9ee4094301f0-init/diff:/var/lib/docker/overlay2/d83b29584a35f0713a18f9ca0a3a131247f4e78994e5d9bf252e7f098b53a16e/diff:/var/lib/docker/overlay2/40f7a832020592e42d511e06a75418938b9ed7935aa1e7e39a82ae2a7e6bd244/diff:/var/lib/docker/overlay2/3d04cf7fe1b94e9c66263687e4e764d647a5c9747cf71297b5e4978215cfebac/diff:/var/lib/docker/overlay2/53281036e6f908e5ec4a78024cb042280377b77121457363a753d4ec799d36e0/diff:/var/lib/docker/overlay2/34e944ab9343577dff2e40fdbe41e72ec8408c0f56b2b9381c98a3b1c175e640/diff:/var/lib/docker/overlay2/dcb71362d282c326e64cd25e9c862a6d9abd44b3f0eb7e3c0cf08f94cb4b7f3b/diff:/var/lib/docker/overlay2/90a0b7edb7a7e68c7b1a9ad1dac11003c4775932bbffeef67e1cb01512e5ad78/diff:/var/lib/docker/overlay2/f157e62b9d11994b45894df1e75c8715468fb8161c7b711f795bf8ea40f91a3e/diff",
                "MergedDir": "/var/lib/docker/overlay2/02632a7e45ef0b7272a56ce4e2248eb2feab6ab7057d1fb4cf5b9ee4094301f0/merged",
                "UpperDir": "/var/lib/docker/overlay2/02632a7e45ef0b7272a56ce4e2248eb2feab6ab7057d1fb4cf5b9ee4094301f0/diff",
                "WorkDir": "/var/lib/docker/overlay2/02632a7e45ef0b7272a56ce4e2248eb2feab6ab7057d1fb4cf5b9ee4094301f0/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/root/teleport.yml",
                "Destination": "/etc/teleport/teleport.yaml",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "67fdd1ee5824",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3023/tcp": {},
                "3024/tcp": {},
                "3080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--insecure-no-tls",
                "--roles=proxy"
            ],
            "Image": "quay.io/gravitational/teleport:4.1.0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/usr/bin/dumb-init",
                "teleport",
                "start",
                "-c",
                "/etc/teleport/teleport.yaml"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "c65410acec558bb66e4ae5023494d22e9ffb7f99bb2f57acf61cbc0ccd907b52",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3023/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "3023"
                    }
                ],
                "3024/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "3024"
                    }
                ],
                "3080/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "3080"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/c65410acec55",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "292ae55b51064fbf74942fb47256125fb88dcddcb0b8fa90e348dceb3142eefb",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "1a99004e0516fc8ed38ce75421d6b1fa3ca5f365e22fcc5ef85ede0f1724de89",
                    "EndpointID": "292ae55b51064fbf74942fb47256125fb88dcddcb0b8fa90e348dceb3142eefb",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

docker inspect auth

[
    {
        "Id": "3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688",
        "Created": "2019-10-21T09:46:08.786502755Z",
        "Path": "/usr/bin/dumb-init",
        "Args": [
            "teleport",
            "start",
            "-c",
            "/etc/teleport/teleport.yaml",
            "--roles=auth",
            "-d"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 28771,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-10-21T09:46:09.6004433Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:b74994c7b350ad113c0cd4bcdef1d807c844c527e9a1c26cb34100b0a88a7a68",
        "ResolvConfPath": "/var/lib/docker/containers/3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688/hostname",
        "HostsPath": "/var/lib/docker/containers/3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688/hosts",
        "LogPath": "/var/lib/docker/containers/3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688/3c8264157665a189fd8095f42aac3b4131bf8196ebcee7d5788d149c1514e688-json.log",
        "Name": "/teleport-auth",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/root/teleport.yml:/etc/teleport/teleport.yaml:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "3025/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3025"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/b2b1f91fa1f0a6d4e7c4ed2523245c015ce2c84f3b1270463deeac732fa5cf25-init/diff:/var/lib/docker/overlay2/987297ad6ee7681ff06ba14959c4a8f68c8f1a716c2a2de8d170a25896a8a730/diff:/var/lib/docker/overlay2/8aad82e171aa87b8d31d56204cc8ecf0b494807bead6205a89775e163498b3bd/diff:/var/lib/docker/overlay2/f6c2b6e615a576f1ad8f4769f6fb085e3b3231302027aab5f11debc260342aa9/diff:/var/lib/docker/overlay2/d3860464988b97253faeb791edc24794f97aa1d1135201ce9084608c91a36d1a/diff:/var/lib/docker/overlay2/92e0cde8435d5c4930043d8899d2be5df869db58ed804341b4862b8d8cff6b20/diff:/var/lib/docker/overlay2/ab43d74c8bdab67146734422cf003611a47e33943f1779043707cf83dc6ae46b/diff:/var/lib/docker/overlay2/a9c2dc5c24372c68e20d6c284a60a29ab9d09c9bd9e7cfe10a2620f8bce5aa6f/diff:/var/lib/docker/overlay2/afe603ea74d45b133a0d02da8601390a1a96e9b1a71464c850f78a2b78dbcb8d/diff",
                "MergedDir": "/var/lib/docker/overlay2/b2b1f91fa1f0a6d4e7c4ed2523245c015ce2c84f3b1270463deeac732fa5cf25/merged",
                "UpperDir": "/var/lib/docker/overlay2/b2b1f91fa1f0a6d4e7c4ed2523245c015ce2c84f3b1270463deeac732fa5cf25/diff",
                "WorkDir": "/var/lib/docker/overlay2/b2b1f91fa1f0a6d4e7c4ed2523245c015ce2c84f3b1270463deeac732fa5cf25/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/root/teleport.yml",
                "Destination": "/etc/teleport/teleport.yaml",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "3c8264157665",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3025/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--roles=auth",
                "-d"
            ],
            "Image": "quay.io/gravitational/teleport:4.1.0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/usr/bin/dumb-init",
                "teleport",
                "start",
                "-c",
                "/etc/teleport/teleport.yaml"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "09c24636938c78dcdb7f6fbdaffda5dc48de4bc0acf4ab7e13f2d64f15ba9558",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3025/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "3025"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/09c24636938c",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "6df8025da2fd4b4fc95beca0db5cc02c8b2266dd96b9439786c2fad61a689175",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "36861431fdb615db3deb7a07d0d60ec4b5d1d4b834a672829fc09b4103da9f57",
                    "EndpointID": "6df8025da2fd4b4fc95beca0db5cc02c8b2266dd96b9439786c2fad61a689175",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

@galindro If you’re running inside Docker, you’ll need to either set public_addr or advertise_ip inside each Teleport service’s config file and restart the services.

Teleport services communicate with each other and state what IP/port they’re running on - this IP is automatically assumed to be whatever the IP is that Teleport “sees” internally, which is why you see traffic trying to get to the private Docker IP of the auth server. The same is true for things like AWS public/private IPs.

When running behind NAT it’s not possible for Teleport to automatically determine the “public” rather than “private” IP so you’ll need to make sure it’s configured to advertise the public IP.

1 Like

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.