Kubernetes group not applied

I’m trying to use Teleport against an Amazon EKS cluster. I am running Teleport on a stand-alone EC2 instance. I’ve configured it to use GitHub for authentication. I can successfully authenticate and start a session with the Teleport server and issue kubectl commands against the cluster, however, the Kubernetes group that I specified in my GitHub config is not applied. Instead I’m inheriting the permissions granted to the node through the instance profile assigned to the EC2 instance. Below is my config for GitHub:

kind: github
version: v3
metadata:
  # connector name that will be used with `tsh --auth=github login`
  name: github
spec:
  # client ID of Github OAuth app
  client_id: xxxxxxxxxxxxx
  # client secret of Github OAuth app
  client_secret: xxxxxxxxxxxxxxxxxxxxxx
  # connector display name that will be shown on web UI login screen
  display: Github
  # callback URL that will be called after successful authentication
  redirect_url: https://instance.compute-1.amazonaws.com:3080/v1/webapi/github/callback
  # mapping of org/team memberships onto allowed logins and roles
  teams_to_logins:
    - organization: jicowan-org # Github organization name
      team: teleport            # Github team name within that organization
      # allowed UNIX logins for team octocats/admins:
      logins:
        - ec2-user
      # list of Kubernetes groups this Github team is allowed to connect to
      kubernetes_groups: ["pod-reader"]

Can you run Teleport with --debug on the command line, capture the logs for the auth flow with Github and post them? It’d be good to see exactly what’s happening.

Also, what’s in your ~/.kube/config locally?

@gus I don’t have a role specified. It’s using the IAM role assigned to the EC2 instance that Teleport is running on. This role is mapped to the system:node RBAC group in Kubernetes. This is an EKS cluster which uses IAM for authentication. I had to tear down the proxy so I can’t access the logs. What does Teleport need to impersonate other roles?

Here’s an overview of the permissions that Teleport will need for impersonation: https://gravitational.com/teleport/docs/kubernetes_ssh/#impersonation

Thanks @gus. I created the clusterrole and clusterrolebinding along with a serviceaccount called teleport-serviceaccount, but I don’t seem to have permission to get kubernetes objects. My RBAC config:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teleport-impersonation
rules:
- apiGroups:
  - ""
  resources:
  - users
  - groups
  - serviceaccounts
  verbs:
  - impersonate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: teleport
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport-impersonation
subjects:
- kind: ServiceAccount
  # this should be changed to the name of the Kubernetes ServiceAccount being used
  name: teleport-serviceaccount
  namespace: default
- kind: Group
  name: teleport:service
  namespace: default

github.yaml

teams_to_logins:
    - organization: <my-org> 
      team: teleport
      logins:
        - root
      kubernetes_groups: ["system:masters"]

error message:

services is forbidden: User "teleport" cannot list resource "services" in API group "" in the namespace "default"

Teleport is running on a standalone EC2 instance. It is not running on the k8s cluster.

If Teleport is running outside the Kubernetes cluster, you’ll need to generate a kubeconfig for it to use as detailed here: https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes

This script is known to work on GKE and bare metal Kubernetes clusters - I don’t think it’s been tested on EKS. I’d be curious to see whether it works or whether some tweaks might be required.

Thanks again @gus. I have a kubeconfig that works. With EKS you don’t need to login using a certificate. You can assign an IAM role to the EC2 instance that teleport is running on and map that role to a Kubernetes RBAC group.

1 Like

Ok, so I am able to get it to work if I add the --as=teleport --as-group= flags to the kubectl commands. It seems like my GitHub configuration is not getting applied when I login to Teleport with my GitHub account.

If you run tsh status locally, you should get a username shown for Logged in as: - probably the email address you have registered with Github.

What’s the output of tctl get user/<username> on the auth server - specifically the section on roles (and potentially kubernetes_groups if it’s there)?

@gus when I run tsh status it says Not logged in however I am able to login to the web console using my GitHub ID. When I get the info for my GitHub user it shows me the following:

kind: user
metadata:
  expires: 2019-06-22T21:14:26.810812276Z
  id: 1561130066811413274
  name: jicowan
spec:
  created_by:
    connector:
      id: github
      identity: jicowan
      type: github
    time: 2019-06-21T15:14:26.810814876Z
    user:
      name: system
  expires: 0001-01-01T00:00:00Z
  github_identities:
  - connector_id: github
    username: jicowan
  roles:
  - admin
  status:
    is_locked: false
    lock_expires: 0001-01-01T00:00:00Z
    locked_time: 0001-01-01T00:00:00Z
  traits:
    kubernetes_groups:
    - system:masters
    logins:
    - root
version: v2

@jicowan That user output looks fine - it’s what I’d expect, and the system:masters at the bottom indicates that the Kubernetes group mapping is getting applied correctly.

If you haven’t logged into the cluster from the command line with tsh (as indicated by Not logged in) then where are you seeing that the groups aren’t getting applied?

@gus Here a description of the steps:

  1. Goto the login page for Teleport and login using my GitHub credentials
  2. Launch a terminal (SSH) session from within the browser/Teleport console
  3. Run kubectl commands

Do I need to do something different to manage kubernetes?

Yes.

The idea is that you run tsh login --proxy=<proxy_address> at the command line and a web browser should pop up, you log in via Github there, then the CLI will print a message stating that you’re now logged in for the next 12 hours.

$ tsh login --proxy=<cluster>.gravitational.co:3080
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:33179/d22e85fc-591a-47e4-86bf-929818ce9a3f
> Profile URL:  https://<cluster>.gravitational.co:3080
  Logged in as: <redacted>@gmail.com
  Cluster:      <cluster>.gravitational.co
  Roles:        clusteradmin*
  Logins:       root
  Valid until:  2019-06-22 00:30:22 -0300 ADT [valid for 12h0m0s]
  Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty


* RBAC is only available in Teleport Enterprise
  https://gravitational.com/teleport/docs/enterprise

Once that’s done, kubectl config get-contexts on your local machine should show your cluster:

$ kubectl config get-contexts
CURRENT   NAME                        CLUSTER                                               AUTHINFO                                              NAMESPACE
*         <cluster>.gravitational.co  <cluster>.gravitational.co                            <cluster>.gravitational.co

Then running anything via kubectl should go via Teleport to the Kubernetes cluster:

$ kubectl -v=7 get nodes
I0621 12:35:47.131312   27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.132563   27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.133212   27461 round_trippers.go:383] GET <cluster>.gravitational.co:3026/api?timeout=32s
I0621 12:35:47.133220   27461 round_trippers.go:390] Request Headers:
I0621 12:35:47.133224   27461 round_trippers.go:393]     Accept: application/json, */*
I0621 12:35:47.133228   27461 round_trippers.go:393]     User-Agent: kubectl/v1.12.4 (linux/amd64) kubernetes/f49fa02
I0621 12:35:47.592972   27461 round_trippers.go:408] Response Status: 200 OK in 459 milliseconds
<snip>
NAME                                             STATUS   ROLES    AGE   VERSION
gke-teleport-demo-cluster-pool-1-afbf7cbd-96cl   Ready    <none>   53d   v1.12.7-gke.10

Thanks @gus. That’s the step I was missing! I should have read through the user guide more carefully.

No worries - glad I could help :slight_smile:

@gus So now I have slightly different issue. I can login

tsh login --proxy=<instance-id>.compute-1.amazonaws.com:3080 --insecure
WARNING: You are using insecure connection to SSH proxy https://....amazonaws.com:3080
WARNING: You are using insecure connection to SSH proxy https://...:3080
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:50581/66b71d13-d389-4326-9768-51df2a3b3498
> Profile URL:  https://...compute-1.amazonaws.com:3080
  Logged in as: jicowan
  Cluster:      ....compute-1.amazonaws.com
  Roles:        admin*
  Logins:       root
  Valid until:  2019-06-22 01:44:11 -0500 CDT [valid for 12h0m0s]
  Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty 

but I can’t seem to connect to the Kubernetes server

kubectl get nodes -v 7
I0621 13:45:30.005130   44300 loader.go:359] Config loaded from file /Users/jicowan/.kube/config
I0621 13:45:30.013528   44300 round_trippers.go:416] GET https://...:3026/api?timeout=32s
I0621 13:45:30.013556   44300 round_trippers.go:423] Request Headers:
I0621 13:45:30.013565   44300 round_trippers.go:426]     User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3
I0621 13:45:30.013573   44300 round_trippers.go:426]     Accept: application/json, */*
I0621 13:45:30.248263   44300 round_trippers.go:441] Response Status: 403 Forbidden in 234 milliseconds
I0621 13:45:30.249843   44300 round_trippers.go:416] GET https://....amazonaws.com:3026/apis?timeout=32s

I can see the entries in my local kube config for the teleport proxy, but it’s behaving as if there’s no kube config on the server. When I try get nodes it says the server doesn't have a resource type "nodes". I am able to issue kubectl commands from the proxy to the k8s cluster when I SSH to the proxy, but I can’t seem to do it through a proxy session.

Given that EKS only supports IAM authN, I’d love to create a reference architecture for teleport that allows folks to use GitHub or OIDC.

Looks like it’s a permissions issue. When I issue the kubectl auth can-i "*" "*" command I get users jicowan is forbidden: User system:anonymous cannot impersonate resource users in API group "" at the cluster scope Do I need to create a specific role binding for jicowan or is it supposed to use the role I’ve assigned to it through the github.yaml file?

This is the reason why I thought that a certificate might be needed for authentication in the first place and suggested the use of https://github.com/gravitational/teleport/blob/master/examples/gke-auth/get-kubeconfig.sh (linked from https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes)

What Kubernetes group is assigned to the IAM role on the EC2 instance? It’ll need to be something with high permissions like system:masters so that Teleport is able to use the impersonate verb. I haven’t done anything on EKS myself so I’m unsure whether it does anything funky with permissions.

If the calls are currently being made as system:anonymous then that’s the reason for it getting a 403.

You shouldn’t need to create a RoleBinding in Kubernetes for anything. Teleport will give you the kubernetes_groups that you assigned in the github.yaml file, but ONLY if it’s able to send impersonate requests to the Kubernetes apiserver as a sufficiently highly privileged user.

@sasha did some work with Teleport on EKS and posted about it here - you might find something useful in that thread.

@gus the role assigned to the EC2 instance that teleport runs on is mapped to the Kubernetes teleport group. The teleport group has permissions to impersonate other roles. I can confirm this works by logging in to the instance and running kubectl get nodes --as teleport --as-group system:masters. I used the clusterrole and rolebinding from the documentation to create the role for teleport. Do I need to use system:masters? I’ll take a look at the post @sasha wrote. I wanted to try doing this on an EC2 instance before I tried running teleport on the cluster.

I also noticed that after I use tsh to login to Teleport, all of the user entries in my local kubeconfig get wiped out.