[QUESTION] Sessions vanish after restart

Hi all,

following scenario: Teleport auth and proxy are running as pods on a kubernetes cluster. The auth is configured to use etcd as well as s3 for audit sessions and is running in HA mode.

I have noticed that after a restart of one of the auth pods, a portion of the sessions in the web console vanish. Am I right in the assumption that Teleport only exposes sessions if they are found locally in /var/lib/teleport/log/sessions?

BR

It should definitely be possible to view all session recordings without relying on the local filesystem.

You mentioned that audit sessions are being uploaded to s3. Have you configured a destination for audit events? E.g.:

audit_events_uri: ['dynamodb://my_event_table']

The web UI needs the ability to scan through audit events in order to locate sessions and session recordings, so loss of the local event log would likely produce the issue you encountered.

The HA section of the admin guide includes more detailed info on audit log storage: https://gravitational.com/teleport/docs/admin-guide/#high-availability