Fail to upgrade a single node cluster

I was testing upgrade from 6.0.1 to 6.0.9 on a single node cluster, I copied the new tar and uncompress it and run ./upgrade it failed, then I tried manual steps and it failed too, do we know what am I missing in upgrade steps? Following are the steps I tried.

# sudo ./upload
Wed Oct 2 21:02:45 UTC Importing application test-appliance v0.0.2
Wed Oct 2 21:02:45 UTC Application already exists in local cluster
Wed Oct 2 21:02:45 UTC Synchronizing application with Docker registry 192.168.86.20:5000
Wed Oct 2 21:03:25 UTC Application has been uploaded

# sudo ./gravity upgrade
Wed Oct 2 21:04:20 UTC Upgrading application test-appliance from 0.0.1 to 0.0.2
Wed Oct 2 21:04:23 UTC Deploying upgrade agents on the nodes
[ERROR]: failed to deploy agents, failed to deploy agents

# ./gravity resume
Executing “/checks” locally
Wed Oct 2 21:12:47 UTC Operation failure: context deadline exceeded
Still executing “/checks” locally (10 seconds elapsed)
Still executing “/checks” locally (20 seconds elapsed)
Still executing “/checks” locally (30 seconds elapsed)
Still executing “/checks” locally (40 seconds elapsed)
Still executing “/checks” locally (50 seconds elapsed)
Still executing “/checks” locally (1 minute elapsed)
operation(update(ea902621-4caa-4a5f-964e-56a4acc215ed), cluster=testco, created=2019-10-02 21:04) finished in 1 minute
[ERROR]: failed to establish connection to server at 192.168.86.20:3012, failed to connect to agent.
Make sure the node has an agent running by issuing gravity agent deploy from the upgrade node, failed to validate requirements, failed to execute phase “/checks”

# ./gravity agent deploy
[ERROR]: failed to deploy agents, failed to deploy agents

So these all look related to the same thing, needing to isolate why the agent deployment is failing. Unfortunately the error on the CLI isn’t very telling about the root cause.

The stack trace error should be captured in the /var/log/telekube-system.log file. Alternatively, it might be possible to capture with the ./gravity --debug agent deploy flag.

As a general consideration, this error usually occurs when a member of the cluster is offline from the embedded teleports perspective. Either the teleport service on each host is failed or disconnected in some way, or something happened to gravity-site pod that isn’t allowing them to connect.

@knisbet thanks for looking into it, this is a single node cluster, below is the output of deploy agent commands.

# gravity status
Cluster status:		updating
Application:		test-appliance, version 0.0.1
Join token:		zwsSn1w5Ur
Periodic updates:	Not Configured
Remote support:		Not Configured
Last completed operation:
    * operation_update (ea902621-4caa-4a5f-964e-56a4acc215ed)
      started:	Wed Oct  2 21:04 UTC (19 hours ago)
      failed:	Wed Oct  2 21:50 UTC (18 hours ago)
Cluster endpoints:
    * Authentication gateway:
        - 192.168.86.20:32009
    * Cluster management URL:
        - https://192.168.86.20:32009
Cluster nodes:	Democo
    Masters:
        * Democo / 192.168.86.20 / node
            Status:	healthy

deploy output

# ./gravity --debug agent deploy
2019-10-03T09:46:58-07:00 DEBU             got search paths: [/var/lib/gravity assets/local] processconfig/config.go:57
2019-10-03T09:46:58-07:00 DEBU             look up configs in /var/lib/gravity processconfig/config.go:59
2019-10-03T09:46:58-07:00 DEBU             /var/lib/gravity/gravity.yaml not found in search path processconfig/config.go:67
2019-10-03T09:46:58-07:00 DEBU             look up configs in assets/local processconfig/config.go:59
2019-10-03T09:46:58-07:00 DEBU             assets/local/gravity.yaml not found in search path processconfig/config.go:67
2019-10-03T09:46:58-07:00 DEBU [LOCAL]     Creating local environment. args:{ /var/lib/gravity/local false false true 0s 0s 0x2555450 {[] 0} false} localenv/localenv.go:152
2019-10-03T09:46:58-07:00 DEBU             got search paths: [/var/lib/gravity assets/local] processconfig/config.go:57
2019-10-03T09:46:58-07:00 DEBU             look up configs in /var/lib/gravity processconfig/config.go:59
2019-10-03T09:46:58-07:00 DEBU             /var/lib/gravity/gravity.yaml not found in search path processconfig/config.go:67
2019-10-03T09:46:58-07:00 DEBU             look up configs in assets/local processconfig/config.go:59
2019-10-03T09:46:58-07:00 DEBU             assets/local/gravity.yaml not found in search path processconfig/config.go:67
2019-10-03T09:46:58-07:00 DEBU [LOCAL]     Creating local environment. args:{ /var/lib/gravity/site/update false false true 0s 0s 0x2555450 {[] 0} false} localenv/localenv.go:152
2019-10-03T09:46:58-07:00 DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:214
2019-10-03T09:46:58-07:00 DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 448.405µs. utils/dns.go:47
2019-10-03T09:46:58-07:00 DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.223.19. utils/dns.go:54
2019-10-03T09:46:58-07:00 DEBU             dialing 10.100.223.19:3009 httplib/client.go:252
2019-10-03T09:46:58-07:00 DEBU [KEYGEN]    SSH cert authority is going to pre-compute 25 keys. native/native.go:91
2019-10-03T09:46:59-07:00 INFO             2019/10/03 09:46:59 [INFO] generate received request runtime/asm_amd64.s:1337
2019-10-03T09:46:59-07:00 INFO             2019/10/03 09:46:59 [INFO] received CSR runtime/asm_amd64.s:1337
2019-10-03T09:46:59-07:00 INFO             2019/10/03 09:46:59 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-10-03T09:46:59-07:00 INFO             2019/10/03 09:46:59 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:00-07:00 INFO [CLIENT]    Connecting proxy=127.0.0.1:3023 login='root' method=0 client/api.go:1458
2019-10-03T09:47:00-07:00 INFO [CLIENT]    Successful auth with proxy 127.0.0.1:3023 client/api.go:1470
2019-10-03T09:47:00-07:00 DEBU [CLIENT]    Found clusters: [{"name":"Democo","lastconnected":"2019-10-03T16:47:00.228650737Z","status":"online"}] client/client.go:100
2019-10-03T09:47:00-07:00 DEBU [LOCAL]     Creating local environment. args:{ /var/lib/gravity/local false false false 0s 0s <nil> {[] 0} false} localenv/localenv.go:152
2019-10-03T09:47:00-07:00 DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:214
2019-10-03T09:47:00-07:00 DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 229.293µs. utils/dns.go:47
2019-10-03T09:47:00-07:00 DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.223.19. utils/dns.go:54
2019-10-03T09:47:00-07:00 DEBU             dialing 10.100.223.19:3009 httplib/client.go:252
2019-10-03T09:47:00-07:00 DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:214
2019-10-03T09:47:00-07:00 DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 902.282µs. utils/dns.go:47
2019-10-03T09:47:00-07:00 DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.223.19. utils/dns.go:54
2019-10-03T09:47:00-07:00 DEBU             dialing 10.100.223.19:3009 httplib/client.go:252
2019-10-03T09:47:00-07:00 DEBU [KEYGEN]    SSH cert authority is going to pre-compute 25 keys. native/native.go:91
2019-10-03T09:47:01-07:00 INFO             2019/10/03 09:47:01 [INFO] generate received request runtime/asm_amd64.s:1337
2019-10-03T09:47:01-07:00 INFO             2019/10/03 09:47:01 [INFO] received CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:01-07:00 INFO             2019/10/03 09:47:01 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-10-03T09:47:01-07:00 INFO             2019/10/03 09:47:01 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:01-07:00 INFO [CLIENT]    Connecting proxy=127.0.0.1:3023 login='root' method=0 client/api.go:1458
2019-10-03T09:47:02-07:00 INFO [CLIENT]    Successful auth with proxy 127.0.0.1:3023 client/api.go:1470
2019-10-03T09:47:02-07:00 INFO [CLIENT]    Client= connecting to node=192.168.86.20:3022 client/client.go:352
2019-10-03T09:47:05-07:00 INFO [CLI]       Successfully connected to teleport on node DemocoProd (192.168.86.20:3022). cli/rpcagent.go:228
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] generate received request runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] received CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] signed certificate with serial number 342259627591069372679530342636988104252407942430 runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] generate received request runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] received CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:05-07:00 INFO             2019/10/03 09:47:05 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-10-03T09:47:06-07:00 INFO             2019/10/03 09:47:06 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:06-07:00 INFO             2019/10/03 09:47:06 [INFO] signed certificate with serial number 100369884078499068349853886713353092917280478119 runtime/asm_amd64.s:1337
2019-10-03T09:47:06-07:00 INFO             2019/10/03 09:47:06 [INFO] generate received request runtime/asm_amd64.s:1337
2019-10-03T09:47:06-07:00 INFO             2019/10/03 09:47:06 [INFO] received CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:06-07:00 INFO             2019/10/03 09:47:06 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-10-03T09:47:08-07:00 INFO             2019/10/03 09:47:08 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-10-03T09:47:08-07:00 INFO             2019/10/03 09:47:08 [INFO] signed certificate with serial number 676189641797581597018401646394731666668113232213 runtime/asm_amd64.s:1337
2019-10-03T09:47:08-07:00 DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:214
2019-10-03T09:47:08-07:00 DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 22.752194ms. utils/dns.go:47
2019-10-03T09:47:08-07:00 DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.223.19. utils/dns.go:54
2019-10-03T09:47:08-07:00 DEBU             dialing 10.100.223.19:3009 httplib/client.go:252
2019-10-03T09:47:08-07:00 INFO [CLIENT]    Client= connecting to node=192.168.86.20:3022 client/client.go:352
2019-10-03T09:47:18-07:00 INFO [RPC:DEPLO] "2019-10-03T16:47:18Z DEBU             got search paths: [/var/lib/gravity assets/local] processconfig/config.go:57\n2019-10-03T16:47:18Z DEBU             look up configs in /var/lib/gravity processconfig/config.go:59\n2019-10-03T16:47:18Z DEBU             /var/lib/gravity/gravity.yaml not found in search path processconfig/config.go:67\n2019-10-03T16:47:18Z DEBU             look up configs in assets/local processconfig/config.go:59\n2019-10-03T16:47:18Z DEBU             assets/local/gravity.yaml not found in search path processconfig/config.go:67\n2019-10-03T16:47:18Z DEBU [LOCAL]     Creating local environment. args:{ /var/lib/gravity/local true false true 0s 0s 0x23a54d0 {[] 0} false} localenv/localenv.go:152\n" cmd:gravity enter -- --notty /usr/bin/gravity -- package unpack Democo/rpcagent-secrets:6.0.9 /var/lib/gravity/site/update/agent/secrets --debug --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009 --insecure node:192.168.86.20:3022 stream:stdout utils/ssh.go:306
2019-10-03T09:47:18-07:00 INFO [RPC:DEPLO] "2019-10-03T16:47:18Z DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:214\n" cmd:gravity enter -- --notty /usr/bin/gravity -- package unpack Democo/rpcagent-secrets:6.0.9 /var/lib/gravity/site/update/agent/secrets --debug --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009 --insecure node:192.168.86.20:3022 stream:stdout utils/ssh.go:306
2019-10-03T09:47:18-07:00 INFO [RPC:DEPLO] "2019-10-03T16:47:18Z DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 528.883µs. utils/dns.go:47\n2019-10-03T16:47:18Z DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.223.19. utils/dns.go:54\n2019-10-03T16:47:18Z DEBU             dialing 10.100.223.19:3009 httplib/client.go:252\n" cmd:gravity enter -- --notty /usr/bin/gravity -- package unpack Democo/rpcagent-secrets:6.0.9 /var/lib/gravity/site/update/agent/secrets --debug --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009 --insecure node:192.168.86.20:3022 stream:stdout utils/ssh.go:306
2019-10-03T09:47:19-07:00 INFO [RPC:DEPLO] "Democo/rpcagent-secrets:6.0.9 unpacked at /var/lib/gravity/site/update/agent/secrets\n" cmd:gravity enter -- --notty /usr/bin/gravity -- package unpack Democo/rpcagent-secrets:6.0.9 /var/lib/gravity/site/update/agent/secrets --debug --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009 --insecure node:192.168.86.20:3022 stream:stdout utils/ssh.go:306
2019-10-03T09:47:19-07:00 WARN [RPC:DEPLO] "/bin/bash: /usr/bin/systemctl: No such file or directory\n" cmd:/usr/bin/systemctl stop gravity-agent.service node:192.168.86.20:3022 stream:stderr utils/ssh.go:322
2019-10-03T09:47:19-07:00 DEBU [RPC:DEPLO] Command /usr/bin/systemctl stop gravity-agent.service failed: Process exited with status 127 cmd:/usr/bin/systemctl stop gravity-agent.service error:[
ERROR REPORT:
Original Error: *ssh.ExitError Process exited with status 127
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:258 github.com/gravitational/gravity/lib/utils.SSHRunAndParse
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:167 github.com/gravitational/gravity/lib/utils.(*sshCommands).run
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:135 github.com/gravitational/gravity/lib/utils.(*sshCommands).Run
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:221 github.com/gravitational/gravity/lib/rpc.deployAgentOnNode
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:135 github.com/gravitational/gravity/lib/rpc.DeployAgents.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: Process exited with status 127
] node:192.168.86.20:3022 utils/ssh.go:259
2019-10-03T09:47:19-07:00 WARN [RPC:DEPLO] ignoring failed subcommand command:{/usr/bin/systemctl stop gravity-agent.service false map[PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/writable/bin] false} error:[
ERROR REPORT:
Original Error: *ssh.ExitError Process exited with status 127
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:258 github.com/gravitational/gravity/lib/utils.SSHRunAndParse
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:167 github.com/gravitational/gravity/lib/utils.(*sshCommands).run
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:135 github.com/gravitational/gravity/lib/utils.(*sshCommands).Run
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:221 github.com/gravitational/gravity/lib/rpc.deployAgentOnNode
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:135 github.com/gravitational/gravity/lib/rpc.DeployAgents.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: Process exited with status 127
] node:192.168.86.20:3022 utils/ssh.go:147
2019-10-03T09:47:22-07:00 INFO [RPC:DEPLO] "gravitational.io/gravity:6.0.9 exported to file /var/lib/gravity/site/update/agent/gravity\n" cmd:gravity enter -- --notty /usr/bin/gravity -- package export --file-mask=755 gravitational.io/gravity:6.0.9 /var/lib/gravity/site/update/agent/gravity --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009 --insecure node:192.168.86.20:3022 stream:stdout utils/ssh.go:306
2019-10-03T09:47:22-07:00 DEBU [RPC:DEPLO] Command /var/lib/gravity/site/update/agent/gravity agent --debug install  failed: Process exited with status 4294967295 cmd:/var/lib/gravity/site/update/agent/gravity agent --debug install  error:[
ERROR REPORT:
Original Error: *ssh.ExitError Process exited with status 4294967295
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:258 github.com/gravitational/gravity/lib/utils.SSHRunAndParse
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:167 github.com/gravitational/gravity/lib/utils.(*sshCommands).run
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:135 github.com/gravitational/gravity/lib/utils.(*sshCommands).Run
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:221 github.com/gravitational/gravity/lib/rpc.deployAgentOnNode
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:135 github.com/gravitational/gravity/lib/rpc.DeployAgents.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: Process exited with status 4294967295
] node:192.168.86.20:3022 utils/ssh.go:259
2019-10-03T09:47:22-07:00 WARN [RPC:DEPLO] Subcommand failed, sequence interrupted. command:{/var/lib/gravity/site/update/agent/gravity agent --debug install  true map[PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/writable/bin] false} error:[
ERROR REPORT:
Original Error: *ssh.ExitError Process exited with status 4294967295
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:258 github.com/gravitational/gravity/lib/utils.SSHRunAndParse
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:167 github.com/gravitational/gravity/lib/utils.(*sshCommands).run
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:135 github.com/gravitational/gravity/lib/utils.(*sshCommands).Run
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:221 github.com/gravitational/gravity/lib/rpc.deployAgentOnNode
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:135 github.com/gravitational/gravity/lib/rpc.DeployAgents.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: Process exited with status 4294967295
] node:192.168.86.20:3022 utils/ssh.go:144
2019-10-03T09:47:22-07:00 WARN             Failed to deploy agent. error:[
ERROR REPORT:
Original Error: *ssh.ExitError Process exited with status 4294967295
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:258 github.com/gravitational/gravity/lib/utils.SSHRunAndParse
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:167 github.com/gravitational/gravity/lib/utils.(*sshCommands).run
	/gopath/src/github.com/gravitational/gravity/lib/utils/ssh.go:135 github.com/gravitational/gravity/lib/utils.(*sshCommands).Run
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:221 github.com/gravitational/gravity/lib/rpc.deployAgentOnNode
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:135 github.com/gravitational/gravity/lib/rpc.DeployAgents.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: {/var/lib/gravity/site/update/agent/gravity agent --debug install  true map[PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/writable/bin] false}
] node:192.168.86.20:3022 rpc/deploy.go:138
2019-10-03T09:47:22-07:00 WARN             Command failed. error:[
ERROR REPORT:
Original Error: trace.aggregate {/var/lib/gravity/site/update/agent/gravity agent --debug install  true map[PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/writable/bin] false}
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/utils/collecterrors.go:64 github.com/gravitational/gravity/lib/utils.Collect
	/gopath/src/github.com/gravitational/gravity/lib/utils/collecterrors.go:27 github.com/gravitational/gravity/lib/utils.CollectErrors
	/gopath/src/github.com/gravitational/gravity/lib/rpc/deploy.go:144 github.com/gravitational/gravity/lib/rpc.DeployAgents
	/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/rpcagent.go:272 github.com/gravitational/gravity/tool/gravity/cli.deployAgents
	/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/rpcagent.go:206 github.com/gravitational/gravity/tool/gravity/cli.rpcAgentDeploy
	/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/run.go:856 github.com/gravitational/gravity/tool/gravity/cli.Execute
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:157 github.com/gravitational/gravity/e/tool/gravity/cli.execute
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:39 main.run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:26 main.main
	/go/src/runtime/proc.go:200 runtime.main
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: failed to deploy agents, failed to deploy agents
] gravity/main.go:27
[ERROR]: failed to deploy agents, failed to deploy agents

Hmm, it looks like most of the commands it’s trying to run via teleport to bring the upgrade agent online are failing with varios status codes. I’m not quite sure off hand what would be causing those specific failures. You can probably try and run the commands it’s printing to see if that gives you any additional insight into the problem.