Installation failed with Error waiting for Agent to join

Hi
I was trying to install gravity and running into following, it is ubuntu 16.04, and not much found in logs except below error,

installation is failing at

Error waiting for Agent to join

Error in logs

ERROR REPORT:
Original Error: syscall.Errno no such file or directory
Stack Trace:
/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:236 github.com/gravitational/gravity/lib/systeminfo.collectFilesystemUsage
/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:138 github.com/gravitational/gravity/lib/systeminfo.collect
/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:35 github.com/gravitational/gravity/lib/systeminfo.New
/gopath/src/github.com/gravitational/gravity/lib/rpc/server/server.go:206 github.com/gravitational/gravity/lib/rpc/server.realSystemInfo.getSystemInfo
/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peer.go:242 github.com/gravitational/gravity/lib/rpc/server.(*serverPeer).Reconnect
/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:228 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer.func1
/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:227 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer
/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: failed to get mount info on /media/user/Ubuntu\04016.04.6\040LTS\040amd64
no such file or directory] utils/logginghook.go:56
2020-07-22T23:05:19+03:00 INFO Retrying in 6.633242453s. error:[
ERROR REPORT:
Original Error: *errors.errorString waiting for agents to join
Stack Trace:
/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:235 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).canContinue
/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:204 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents.func1
/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:197 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents
/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:104 github.com/gravitational/gravity/lib/install/engine/cli.(*Engine).execute

It looks that the joining agent has failed to validate one of the mounts (/media/user/Ubuntu\04016.04.6\040LTS\040amd64). The agent is collecting the mount metadata for the installer to later be able to answer path questions. Can you check whether the mount is existing?

Hi
I am seeing similar issue during installation as well, this is for Ubuntu fresh install
Mount exists under /media/user/Ubuntu 016.04.6 LTS amd64
Also it is happening during installation, and what else we can look?, crashreport have nothing inside it.

It looks like the location of a USB drive - possibly used for the installation?
I wonder why it is listed in the active mounts but is not such, i.e. if you ran:

findmnt /media/$USER/Ubuntu*

does it return anything?
If you unmount the USB - does the error persist? Or is USB not attached?

@Dmitri_Shelenin
for me the USB is not attached, I tried removing the folder and still got the same error, I tried looking at journelctl logs and did not find any thing other than installer is started,
i have gravity 6.1.29, can you suggest what can be the next steps?

020-07-22T22:49:56+03:00 DEBU [LOCAL]     Creating local environment. args:{ /mnt/installation false false false 0s 0s <nil> {[] 0} true} utils/logginghook.go:56
2020-07-22T22:49:56+03:00 INFO [INSTALLER] Flavor is not set, picking first flavor. flavor:single utils/logginghook.go:56
2020-07-22T22:49:56+03:00 INFO [INSTALLER] No server profile specified, picking the first. role:node utils/logginghook.go:56
2020-07-22T22:49:56+03:00 INFO [CLI]       Will use provider "generic" specified on the CLI. utils/logginghook.go:56
2020-07-22T22:49:57+03:00 DEBU             /pack/v1/repositories/gravitational.io/packages/rpcagent-secrets/0.0.1/envelope method:GET utils/logginghook.go:56
2020-07-22T22:49:57+03:00 DEBU             /pack/v1/repositories/gravitational.io/packages/rpcagent-secrets/0.0.1/file method:HEAD utils/logginghook.go:56
2020-07-22T22:49:57+03:00 DEBU             /pack/v1/repositories/gravitational.io/packages/rpcagent-secrets/0.0.1/file method:GET utils/logginghook.go:56
2020-07-22T22:49:57+03:00 INFO [AGENT:RPC] Fetched cloud metadata. metadata:CloudMetadata(<empty>) provider:onprem utils/logginghook.go:56
2020-07-22T22:49:57+03:00 INFO [INSTALLER] Failed to query install operation. error:[
ERROR REPORT:
Original Error: *trace.BadParameterError expected 1 cluster, got: []
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/ops/utils.go:229 github.com/gravitational/gravity/lib/ops.GetWizardOperation
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:238 github.com/gravitational/gravity/lib/install.(*Installer).maybeStartAgent
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:74 github.com/gravitational/gravity/lib/install.New
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:152 github.com/gravitational/gravity/e/tool/gravity/cli.newCLInstaller
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:101 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:29 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main
	/go/src/runtime/proc.go:200 runtime.main
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: expected 1 cluster, got: []] utils/logginghook.go:56
2020-07-22T22:49:57+03:00 INFO [SERVICE:I] Execute. req: utils/logginghook.go:56
2020-07-22T22:49:57+03:00 INFO [INSTALLER] Execute. req: utils/logginghook.go:56
2020-07-22T22:49:57+03:00 INFO [INSTALLER] Installed gravity binary. mode:cli path:/usr/bin/gravity utils/logginghook.go:56
2020-07-22T22:49:57+03:00 DEBU             Found gravity state locator: {/mnt/gravity}. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             createSite({"app_package":"gravitational.io/test-app:1.0.0","account_id":"00000000-0000-0000-0000-000000000001","email":"installer@sharpeinstein2444","provider":"onprem","domain_name":"sharpeinstein2444","license":"","labels":null,"resources":null,"location":"","flavor":"single","install_token":"en4WwQmpqI","service_user":{"name":"user","uid":"1000","gid":"1000"},"cloud_config":{},"dns_overrides":{"hosts":{},"zones":{}},"dns_config":{"addrs":["127.0.0.2"],"port":53},"docker":{}}) utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/test-app:1.0.0. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/kubernetes:6.1.29. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/rbac-app:6.1.29. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/dns-app:0.4.0. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/bandwagon:6.0.1. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/logging-app:6.0.4. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/monitoring-app:6.0.10. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/tiller-app:6.1.0. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Getting dependencies for gravitational.io/site:6.1.29. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             Package dependency gravitational.io/gravity:6.1.29 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             Package dependency gravitational.io/web-assets:6.1.29 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             Package dependency gravitational.io/teleport:3.2.13 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             Package dependency gravitational.io/planet:6.1.26-11512 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/rbac-app:6.1.29 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/dns-app:0.4.0 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/bandwagon:6.0.1 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/logging-app:6.0.4 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/monitoring-app:6.0.10 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/tiller-app:6.1.0 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/site:6.1.29 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             App dependency gravitational.io/kubernetes:6.1.29 is present. utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [OPS]       CreateSite(ops.NewSiteRequest{AppPackage:"gravitational.io/test-app:1.0.0", AccountID:"00000000-0000-0000-0000-000000000001", Email:"installer@sharpeinstein2444", Provider:"onprem", DomainName:"sharpeinstein2444", License:"", Labels:map[string]string(nil), Resources:[]uint8(nil), Location:"", Flavor:"single", InstallToken:"en4WwQmpqI", ServiceUser:storage.OSUser{Name:"user", UID:"1000", GID:"1000"}, CloudConfig:storage.CloudConfig{GCENodeTags:[]string(nil)}, DNSOverrides:storage.DNSOverrides{Hosts:map[string]string{}, Zones:map[string][]string{}}, DNSConfig:storage.DNSConfig{Addrs:[]string{"127.0.0.2"}, Port:53}, Docker:storage.DockerConfig{StorageDriver:"", Args:[]string(nil)}}). utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU [OPS]       createInstallExpandOperation. domain:sharpeinstein2444 req:{operation_install install_initiated onprem {{  false   { []}} {10.244.0.0/16 10.100.0.0/16 8472} {          }} map[node:{Generic Linux node map[gravitational.io/k8s-role: role:node]  { 1}} worker:{Generic Worker Linux node map[gravitational.io/k8s-role: role:worker]  { 0}}]} utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Create install operation. token:en4WwQmpqI utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU [OPS]       selected subnets: &{10.244.0.0/16 10.100.0.0/16} domain:sharpeinstein2444 opid:c8f0e8c2-6278-4e05-bc7f-d550c6b9b9e2 utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [OPS]       Operation has been created domain:sharpeinstein2444 opid:c8f0e8c2-6278-4e05-bc7f-d550c6b9b9e2 opstate:in_progress optype:operation_install progress:0 utils/logginghook.go:56
2020-07-22T22:49:58+03:00 DEBU             got operation: &ops.SiteOperationKey{AccountID:"00000000-0000-0000-0000-000000000001", SiteDomain:"sharpeinstein2444", OperationID:"c8f0e8c2-6278-4e05-bc7f-d550c6b9b9e2"} utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [INSTALLER] Start progress feedback loop. operation:c8f0e8c2-6278-4e05-bc7f-d550c6b9b9e2 utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [SERVER-PE] Monitoring. monitored:172.16.0.6:61009 utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [SERVER-PE] Reconnecting reconnected:peer(addr=172.16.0.6:3012->server=172.16.0.6:61009) utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [SERVER-PE] Monitoring peers. health.checker:peer(addr=172.16.0.6:61009) utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO [AGENT:RPC] Listening. addr:172.16.0.6:3012 utils/logginghook.go:56
2020-07-22T22:49:58+03:00 INFO             Retrying in 378.900703ms. error:[
ERROR REPORT:
Original Error: syscall.Errno no such file or directory
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:236 github.com/gravitational/gravity/lib/systeminfo.collectFilesystemUsage
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:138 github.com/gravitational/gravity/lib/systeminfo.collect
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:35 github.com/gravitational/gravity/lib/systeminfo.New
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/server.go:206 github.com/gravitational/gravity/lib/rpc/server.realSystemInfo.getSystemInfo
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peer.go:242 github.com/gravitational/gravity/lib/rpc/server.(*serverPeer).Reconnect
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:228 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer.func1
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
	/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:227 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: failed to get mount info on /media/user
	no such file or directory] utils/logginghook.go:56
2020-07-22T22:49:59+03:00 DEBU [PROCESS]   Added group. key:operation(id=c8f0e8c2-6278-4e05-bc7f-d550c6b9b9e2) mode:installer process:172.16.0.6_61009 utils/logginghook.go:56
2020-07-22T22:49:59+03:00 INFO [AGENT-GRO] Monitoring peers. health.checker: utils/logginghook.go:56
2020-07-22T22:49:59+03:00 INFO             Retrying in 575.788756ms. error:[
ERROR REPORT:
Original Error: *errors.errorString waiting for agents to join
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:235 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).canContinue
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:204 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents.func1
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
	/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:197 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:104 github.com/gravitational/gravity/lib/install/engine/cli.(*Engine).execute
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:80 github.com/gravitational/gravity/lib/install/engine/cli.(*Engine).Execute
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:258 github.com/gravitational/gravity/lib/install.(*Installer).execute
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:206 github.com/gravitational/gravity/lib/install.(*Installer).startExecuteLoop.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: waiting for agents to join] utils/logginghook.go:56
2020-07-22T22:49:59+03:00 INFO             Retrying in 596.995505ms. error:[
ERROR REPORT:
Original Error: syscall.Errno no such file or directory
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:236 github.com/gravitational/gravity/lib/systeminfo.collectFilesystemUsage
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:138 github.com/gravitational/gravity/lib/systeminfo.collect
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:35 github.com/gravitational/gravity/lib/systeminfo.New
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/server.go:206 github.com/gravitational/gravity/lib/rpc/server.realSystemInfo.getSystemInfo
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peer.go:242 github.com/gravitational/gravity/lib/rpc/server.(*serverPeer).Reconnect
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:228 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer.func1
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
	/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:227 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: failed to get mount info on /media/user
	no such file or directory] utils/logginghook.go:56
2020-07-22T22:49:59+03:00 INFO             Retrying in 1.006873384s. error:[
ERROR REPORT:
Original Error: *errors.errorString waiting for agents to join
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:235 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).canContinue
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:204 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents.func1
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
	/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:197 github.com/gravitational/gravity/lib/install/engine/cli.(*executor).waitForAgents
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:104 github.com/gravitational/gravity/lib/install/engine/cli.(*Engine).execute
	/gopath/src/github.com/gravitational/gravity/lib/install/engine/cli/cli.go:80 github.com/gravitational/gravity/lib/install/engine/cli.(*Engine).Execute
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:258 github.com/gravitational/gravity/lib/install.(*Installer).execute
	/gopath/src/github.com/gravitational/gravity/lib/install/install.go:206 github.com/gravitational/gravity/lib/install.(*Installer).startExecuteLoop.func1
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: waiting for agents to join] utils/logginghook.go:56
2020-07-22T22:49:59+03:00 INFO             Retrying in 932.629141ms. error:[
ERROR REPORT:
Original Error: syscall.Errno no such file or directory
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:236 github.com/gravitational/gravity/lib/systeminfo.collectFilesystemUsage
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:138 github.com/gravitational/gravity/lib/systeminfo.collect
	/gopath/src/github.com/gravitational/gravity/lib/systeminfo/systeminfo.go:35 github.com/gravitational/gravity/lib/systeminfo.New
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/server.go:206 github.com/gravitational/gravity/lib/rpc/server.realSystemInfo.getSystemInfo
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peer.go:242 github.com/gravitational/gravity/lib/rpc/server.(*serverPeer).Reconnect
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:228 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer.func1
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:238 github.com/gravitational/gravity/lib/utils.RetryWithInterval.func1
	/gopath/src/github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff/retry.go:37 github.com/gravitational/gravity/vendor/github.com/cenkalti/backoff.RetryNotify
	/gopath/src/github.com/gravitational/gravity/lib/utils/retry.go:237 github.com/gravitational/gravity/lib/utils.RetryWithInterval
	/gopath/src/github.com/gravitational/gravity/lib/rpc/server/peers.go:227 github.com/gravitational/gravity/lib/rpc/server.(*peers).reconnectPeer
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: failed to get mount info on /media/user
	no such file or directory] utils/logginghook.go:56

Also based on todays logs after node reboot
jounalrctl logs

Jul 26 11:03:47 TEST-USER01 systemd[1]: Started Auto-generated service for the /mnt/installation/.gravity/gravity-installer.service.
Jul 26 11:03:54 TEST-USER01 gravity[787]: Sun Jul 26 08:03:54 UTC        Starting installer
Jul 26 11:03:54 TEST-USER01 gravity[787]: [ERROR]: no network interfaces detected
Jul 26 11:03:54 TEST-USER01 gravity[787]:
Jul 26 11:03:54 TEST-USER01 systemd[1]: gravity-installer.service: Main process exited, code=exited, status=255/n/a
Jul 26 11:03:54 TEST-USER01 systemd[1]: gravity-installer.service: Unit entered failed state.
Jul 26 11:03:54 TEST-USER01 systemd[1]: gravity-installer.service: Failed with result 'exit-code'.
Jul 26 11:03:59 TEST-USER01 systemd[1]: gravity-installer.service: Service hold-off time over, scheduling restart.
Jul 26 11:03:59 TEST-USER01 systemd[1]: Stopped Auto-generated service for the /mnt/installation/.gravity/gravity-installer.service.
Jul 26 11:03:59 TEST-USER01 systemd[1]: Started Auto-generated service for the /mnt/installation/.gravity/gravity-installer.service.
Jul 26 11:04:00 TEST-USER01 gravity[1179]: Sun Jul 26 08:04:00 UTC        Starting installer
Jul 26 11:04:06 TEST-USER01 gravity[1179]: [ERROR]: dial tcp 172.16.0.6:61009: connect: connection refused
Jul 26 11:04:06 TEST-USER01 gravity[1179]:
Jul 26 11:04:06 TEST-USER01 systemd[1]: gravity-installer.service: Main process exited, code=exited, status=252/n/a
Jul 26 11:04:06 TEST-USER01 systemd[1]: gravity-installer.service: Unit entered failed state.
Jul 26 11:04:06 TEST-USER01 systemd[1]: gravity-installer.service: Failed with result 'exit-code'.

Gravity logs

2020-07-26T11:03:54+03:00 DEBU             State dir locator written to /etc/.gravity.config. utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU             got search paths: [/var/lib/gravity assets/local] utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU             look up configs in /var/lib/gravity utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU             /var/lib/gravity/gravity.yaml not found in search path utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU             look up configs in assets/local utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU             assets/local/gravity.yaml not found in search path utils/logginghook.go:56
2020-07-26T11:03:54+03:00 DEBU [LOCAL]     Creating local environment. args:{/mnt/gravity /mnt/gravity/local false false false 0s 0s 0x270a580 {[127.0.0.2] 53} false} utils/logginghook.go:56
2020-07-26T11:03:54+03:00 INFO [INSTALLER] Set installer write state directory. dir:/mnt/installation/.gravity utils/logginghook.go:56
2020-07-26T11:03:54+03:00 WARN             Command failed. error:[
ERROR REPORT:
Original Error: *trace.NotFoundError no network interfaces detected
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/config.go:1168 github.com/gravitational/gravity/tool/gravity/cli.checkLocalAddr
	/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/config.go:273 github.com/gravitational/gravity/tool/gravity/cli.(*InstallConfig).CheckAndSetDefaults
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/config.go:91 github.com/gravitational/gravity/e/tool/gravity/cli.(*InstallConfig).CheckAndSetDefaults
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:25 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main
	/go/src/runtime/proc.go:200 runtime.main
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: no network interfaces detected] utils/logginghook.go:56
2020-07-26T11:04:00+03:00 DEBU             State dir locator written to /etc/.gravity.config. utils/logginghook.go:56
2020-07-26T11:04:00+03:00 DEBU             got search paths: [/var/lib/gravity assets/local] utils/logginghook.go:56
2020-07-26T11:04:00+03:00 DEBU             look up configs in /var/lib/gravity utils/logginghook.go:56
...
2020-07-26T11:04:00+03:00 DEBU [LOCAL]     Creating local environment. args:{/mnt/gravity /mnt/gravity/local false false false 0s 0s 0x270a580 {[127.0.0.2] 53} false} utils/logginghook.go:56
2020-07-26T11:04:00+03:00 INFO [INSTALLER] Set installer write state directory. dir:/mnt/installation/.gravity utils/logginghook.go:56
2020-07-26T11:04:00+03:00 INFO [INSTALLER] Set advertise address. addr:172.16.0.6 utils/logginghook.go:56
2020-07-26T11:04:00+03:00 INFO             System user exists: User(user, uid=1000, gid=1000). utils/logginghook.go:56
2020-07-26T11:04:00+03:00 INFO [PROCESS]   Teleport config: &service.Config{DataDir:"/mnt/installation/.gravity", Hostname:"TEST-USER01", Token:"", AuthServers:[]utils.NetAddr{utils.NetAddr{Addr:"localhost:61025", AddrNetwork:"tcp", Path:""}, utils.NetAddr{Addr:"localhost:61025", AddrNetwork:"tcp", Path:""}}, Identities:[]*auth.Identity(nil), AdvertiseIP:"", CachePolicy:service.CachePolicy{Enabled:true, TTL:0, NeverExpires:false, RecentTTL:(*time.Duration)(nil)}, SSH:service.SSHConfig{Enabled:true, Addr:utils.NetAddr{Addr:"0.0.0.0:61022", AddrNetwork:"tcp", Path:""}, Namespace:"", Shell:"/bin/bash", Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, Labels:map[string]string(nil), CmdLabels:services.CommandLabels(nil), PermitUserEnvironment:false, PAM:(*pam.Config)(0xc00013b8c0), PublicAddrs:[]utils.NetAddr(nil)}, Auth:service.AuthConfig{Enabled:true, EnableProxyProtocol:true, SSHAddr:utils.NetAddr{Addr:"localhost:61025", AddrNetwork:"tcp", Path:""}, Authorities:[]services.CertAuthority(nil), Roles:[]services.Role(nil), ClusterName:(*services.ClusterNameV2)(0xc000a447e0), StaticTokens:(*services.StaticTokensV2)(0xc0005ab680), StorageConfig:backend.Config{Type:"bolt", Params:backend.Params{"path":"/mnt/installation/.gravity"}}, Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, NoAudit:false, Preference:(*services.AuthPreferenceV2)(0xc000582fa0), ClusterConfig:(*services.ClusterConfigV3)(0xc0002d58c0), LicenseFile:"/var/lib/teleport/license.pem", PublicAddrs:[]utils.NetAddr(nil)}, Keygen:sshca.Authority(nil), Proxy:service.ProxyConfig{Enabled:true, DisableTLS:false, DisableWebInterface:true, DisableWebService:true, DisableReverseTunnel:false, ReverseTunnelListenAddr:utils.NetAddr{Addr:"0.0.0.0:61024", AddrNetwork:"tcp", Path:""}, EnableProxyProtocol:true, WebAddr:utils.NetAddr{Addr:"0.0.0.0:61080", AddrNetwork:"tcp", Path:""}, SSHAddr:utils.NetAddr{Addr:"0.0.0.0:61023", AddrNetwork:"tcp", Path:""}, TLSKey:"", TLSCert:"", Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, PublicAddrs:[]utils.NetAddr(nil), SSHPublicAddrs:[]utils.NetAddr(nil), Kube:service.KubeProxyConfig{Enabled:false, ListenAddr:utils.NetAddr{Addr:"0.0.0.0:3026", AddrNetwork:"tcp", Path:""}, APIAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, ClusterOverride:"", CACert:[]uint8(nil), PublicAddrs:[]utils.NetAddr(nil), KubeconfigPath:""}}, HostUUID:"", Console:(*io.PipeWriter)(0xc000010ae0), ReverseTunnels:[]services.ReverseTunnel(nil), OIDCConnectors:[]services.OIDCConnector(nil), PIDFile:"", Trust:(*usersservice.UsersService)(0xc000bb75f0), Presence:(*keyval.backend)(0xc000baf800), Provisioner:(*usersservice.UsersService)(0xc000bb75f0), Identity:(*usersservice.UsersService)(0xc000bb75f0), Access:(*usersservice.UsersService)(0xc000bb75f0), ClusterConfiguration:(*usersservice.UsersService)(0xc000bb75f0), CipherSuites:[]uint16{0xcca8, 0xcca9, 0xc02f, 0xc02b, 0xc030, 0xc02c, 0x9c, 0x9d}, Ciphers:[]string{"aes128-gcm@openssh.com", "aes128-ctr", "aes192-ctr", "aes256-ctr"}, KEXAlgorithms:[]string{"curve25519-sha256@libssh.org", "ecdh-sha2-nistp256", "ecdh-sha2-nistp384", "ecdh-sha2-nistp521"}, MACAlgorithms:[]string{"hmac-sha2-256-etm@openssh.com", "hmac-sha2-256"}, DiagnosticAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, Debug:false, UploadEventsC:(chan *events.UploadEvent)(nil), FileDescriptors:[]service.FileDescriptor(nil), PollingPeriod:10000000000, ClientTimeout:0, ShutdownTimeout:0, CAPin:"", Clock:clockwork.Clock(nil)}. mode:installer utils/logginghook.go:56
2020-07-26T11:04:00+03:00 INFO [PROCESS]   Gravity config: processconfig.Config{Hostname:"172.16.0.6", Mode:"installer", Profile:processconfig.ProfileConfig{HTTPEndpoint:"", OutputDir:""}, Devmode:false, ClusterName:"", WebAssetsDir:"/mnt/installation/.gravity/assets", DataDir:"/mnt/installation/.gravity", HealthAddr:utils.NetAddr{Addr:":61010", AddrNetwork:"tcp", Path:""}, BackendType:"bolt", ETCD:keyval.ETCDConfig{Clock:clockwork.Clock(nil), Nodes:[]string(nil), Key:"", TLSKeyFile:"", TLSCertFile:"", TLSCAFile:"", RetryInterval:0}, OpsCenter:processconfig.OpsCenterConfig{SeedConfig:(*ops.SeedConfig)(0xc000bb7110)}, Pack:processconfig.PackageServiceConfig{ListenAddr:utils.NetAddr{Addr:"0.0.0.0:61009", AddrNetwork:"tcp", Path:""}, PublicListenAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, AdvertiseAddr:utils.NetAddr{Addr:"172.16.0.6:61009", AddrNetwork:"tcp", Path:""}, PublicAdvertiseAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, ReadDir:"/mnt/installation"}, Charts:processconfig.ChartsConfig{Backend:"local"}, Users:processconfig.Users{processconfig.User{Owner:false, Password:"en4WwQmpqI", T
...
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported ciphers: ["aes128-gcm@openssh.com" "aes128-ctr" "aes192-ctr" "aes256-ctr"]. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported KEX algorithms: ["curve25519-sha256@libssh.org" "ecdh-sha2-nistp256" "ecdh-sha2-nistp384" "ecdh-sha2-nistp521"]. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported MAC algorithms: ["hmac-sha2-256-etm@openssh.com" "hmac-sha2-256"]. utils/logginghook.go:56

2020-07-26T11:04:04+03:00 INFO [PROXY:SER] Starting on 0.0.0.0:61024 using cache that will expire after connection to database is lost after 20h0m0s, will cache frequently accessed items for 2s utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [PROC:1]    Broadcasting event. event:ProxyReverseTunnelReady utils/logginghook.go:56
2020-07-26T11:04:04+03:00 INFO [PROCESS]   Received ProxyReverseTunnelReady event. mode:installer utils/logginghook.go:56
2020-07-26T11:04:04+03:00 INFO [PROCESS]   Received ProxyIdentity event. mode:installer utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Listening on [::]:61024. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported ciphers: ["aes128-gcm@openssh.com" "aes128-ctr" "aes192-ctr" "aes256-ctr"]. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported KEX algorithms: ["curve25519-sha256@libssh.org" "ecdh-sha2-nistp256" "ecdh-sha2-nistp384" "ecdh-sha2-nistp521"]. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [SSH:PROXY] Supported MAC algorithms: ["hmac-sha2-256-etm@openssh.com" "hmac-sha2-256"]. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [PROC]      Adding service to supervisor. service:proxy.ssh utils/logginghook.go:56

2020-07-26T11:04:04+03:00 INFO [AUDIT:1]   Creating directory /mnt/installation/.gravity/log/upload. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [PROC:1]    Service has started. service:proxy.ssh utils/logginghook.go:56
2020-07-26T11:04:04+03:00 INFO [PROXY]     SSH proxy service is starting on 0.0.0.0:61023. utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [PROC:1]    Service has started. service:proxy.reversetunnel.agent utils/logginghook.go:56
2020-07-26T11:04:04+03:00 DEBU [PROC:1]    Service has started. service:proxy.shutdown utils/logginghook.go:56
2020-07-26T11:04:04+03:00 INFO [AUDIT:1]   Creating directory /mnt/installation/.gravity/log/upload/sessions. utils/logginghook.go:56
...
2020-07-26T11:04:06+03:00 INFO [CLI]       Will use provider "generic" specified on the CLI. utils/logginghook.go:56
2020-07-26T11:04:06+03:00 WARN             Command failed. error:[
ERROR REPORT:
Original Error: utils.exitCodeError dial tcp 172.16.0.6:61009: connect: connection refused
Stack Trace:
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:96 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:29 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run
	/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main
	/go/src/runtime/proc.go:200 runtime.main
	/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: dial tcp 172.16.0.6:61009: connect: connection refused] utils/logginghook.go:56

The previous error was still about a bogus mount (/media/user) which was apparently still listed in mounts.
The last one is curious - can you share the output of ip link show to see if maybe you actually have no interfaces the installer is looking for (ipv4 and no lo) for whatever reason?

@Dmitri_Shelenin thanks for taking a look at this,

I guess it did not came up earlier because machine is reboot and ip was not assigned, but later it was fine, any suggestion what to look next?

ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1200 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 02:c6:8d:79:2c:7c brd ff:ff:ff:ff:ff:ffz

I do not see any mount

lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1.6G  0 rom  
fd0      2:0    1    4K  0 disk 
sda      8:0    0  300G  0 disk 
├─sda2   8:2    0    1K  0 part 
├─sda5   8:5    0  975M  0 part [SWAP]
└─sda1   8:1    0  299G  0 part /

fstab output

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=b62569f6-afd7-4190-898c-d7f0802be56d /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=0da0632f-34cb-462a-92a2-b6e2b4b11f18 none            swap    sw              0       0
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0

I will try to work around this problem by filtering out non-existing mountpoints.
You can also share the contents of /etc/mtab or check whether you have differences between /etc/mtab and /proc/mounts?

I’ve merged a change that should help you avoid this error. It will be available in 7.0.14 when it’s published or you can build a custom version.