Gravity installation failing on Centos 8

I am trying to install my application on Centos 8 machine. The installation get stuck on starting enterprise installer.

ERROR REPORT:

Original Error: *trace.BadParameterError expected 1 cluster, got: []

Stack Trace:

/gopath/src/github.com/gravitational/gravity/lib/ops/utils.go:229 github.com/gravitational/gravity/lib/ops.GetWizardOperation

/gopath/src/github.com/gravitational/gravity/lib/install/install.go:238 github.com/gravitational/gravity/lib/install.(*Installer).maybeStartAgent

/gopath/src/github.com/gravitational/gravity/lib/install/install.go:74 github.com/gravitational/gravity/lib/install.New

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:152 github.com/gravitational/gravity/e/tool/gravity/cli.newCLInstaller

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:101 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:29 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run

/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main

/go/src/runtime/proc.go:200 runtime.main

/go/src/runtime/asm_amd64.s:1337 runtime.goexit

Can you share the exact environment details (uname -a and the OS release information) and which command line you use to install?

uname -a

Linux ip-172-31-60-79.us-west-2.compute.internal 5.6.2-1.el8.elrepo.x86_64 #1 SMP Thu Apr 2 11:03:39 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

NAME=“CentOS Linux”
VERSION=“8 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“CentOS Linux 8 (Core)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://www.centos.org/
BUG_REPORT_URL=“https://bugs.centos.org/
CENTOS_MANTISBT_PROJECT=“CentOS-8”
CENTOS_MANTISBT_PROJECT_VERSION=“8”

using sudo ./gravity install to install

Seems to be an issue with ports not being open. On starting the installer service, i got the following error.

/mnt/installation/gravity install --advertise-addr “172.31.60.79” --cloud-provider “generic” --state-dir “/mnt/gravity” --token “secret” --from-service “/mnt/installation”
Thu Sep 24 18:58:05 UTC Starting enterprise installer
[ERROR]: dial tcp 172.31.60.79:61009: connect: connection refused

I tried to open the ports using firewall-cmd. They seem to be open, but still getting connection refused.

[root@ip-172-31-60-79 sysconfig]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client ssh
ports: 61008-61010/tcp 61022-61024/tcp 61009/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

[root@ip-172-31-60-79 sysconfig]# /mnt/installation/gravity install --advertise-addr “172.31.60.79” --cloud-provider “generic” --state-dir “/mnt/gravity” --token “secret” --from-service “/mnt/installation”
Thu Sep 24 19:14:32 UTC Starting enterprise installer
[ERROR]: dial tcp 172.31.60.79:61009: connect: connection refused

This is very much similar to https://github.com/gravitational/gravity/pull/2151 (sadly there was no issue recorded for this). It has been fixed in 7.x but has not been ported to 6.1.x. Are you using 6.1.x for this installation? If so, the fix will be available in 6.1.40

Still seeing this in 6.1.41.

NAME=“CentOS Linux”
VERSION=“8 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“CentOS Linux 8 (Core)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://www.centos.org/
BUG_REPORT_URL=“https://bugs.centos.org/

CENTOS_MANTISBT_PROJECT=“CentOS-8”
CENTOS_MANTISBT_PROJECT_VERSION=“8”

Gravity Version
dependencies:
apps:

gravity-system.log
Original Error: utils.exitCodeError dial tcp 172.31.47.250:61009: connect: connection refused
Stack Trace:
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:96 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:29 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:102 github.com/gravitational/gravity/e/tool/gravity/cli.execute
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:51 github.com/gravitational/gravity/e/tool/gravity/cli.getExec.func1
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/logging.go:71 github.com/gravitational/gravity/tool/gravity/cli.(*CmdExecer).Execute
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:46 github.com/gravitational/gravity/e/tool/gravity/cli.Run
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main
/go/src/runtime/proc.go:200 runtime.main
/go/src/runtime/asm_amd64.s:1337 runtime.goexit
User Message: dial tcp 172.31.47.250:61009: connect: connection refused] utils/logginghook.go:56

Please share the journal logs of the gravity-installer service. And also whether the node is running SELinux in enforcing mode. In this case, parts of the audit log relevant to the install would also help.

Journal logs;
06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd[1]: Started System Security Services Daemon.
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd[1]: Reached target User and Group Name Lookups.
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd[1]: Starting Login Service…
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity-cli[1542]: [RUNNING]: /mnt/installation/gravity install --advertise-addr “172.31.47.250” --cloud-provider "generi>
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity[1542]: Tue Oct 6 03:28:06 UTC Starting enterprise installer
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity-cli[1542]: [FAILURE]: /mnt/installation/gravity install --advertise-addr “172.31.47.250” --cloud-provider "generi>
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity[1542]: [ERROR]: no network interfaces detected
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd-udevd[1492]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity[1542]:
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd[1]: gravity-installer.service: Main process exited, code=exited, status=255/n/a
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal systemd[1]: gravity-installer.service: Failed with result ‘exit-code’.
Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal kernel: ppdev: user-space parallel port driver

Audit log;
type=CRYPTO_KEY_USER msg=audit(1602003451.739:1345): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=session fp=? direction=both spid=4115 suid=74 rport=50593 laddr=172.31.47.250 lport=22 exe="/usr/sbin/sshd" hostname=? addr=222.186.31.166 terminal=? res=success’UID=“root” AUID=“unset” SUID=“sshd”
type=CRYPTO_KEY_USER msg=audit(1602003451.739:1346): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:bc:2b:2a:0e:09:37:04:be:ae:ca:46:bc:0b:39:6f:f6:a2:01:57:0a:04:bd:67:fe:57:a8:8f:91:3b:59:54:66 direction=? spid=4115 suid=74 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset” SUID=“sshd”
type=USER_ERR msg=audit(1602003451.739:1347): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=222.186.31.166 addr=222.186.31.166 terminal=ssh res=failed’UID=“root” AUID=“unset”
type=CRYPTO_KEY_USER msg=audit(1602003451.740:1348): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:95:28:a7:f3:21:97:2c:3a:e4:a5:6e:4f:09:95:7c:03:a6:08:0a:d5:89:68:16:ad:83:84:1d:da:ca:e3:08:4f direction=? spid=4114 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset” SUID=“root”
type=CRYPTO_KEY_USER msg=audit(1602003451.740:1349): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:87:80:f1:fb:98:27:4c:ea:f0:e1:6b:1f:d3:bc:5d:7b:04:8a:f3:f3:a2:05:28:41:86:e4:b0:51:51:ee:e7:e5 direction=? spid=4114 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset” SUID=“root”
type=CRYPTO_KEY_USER msg=audit(1602003451.740:1350): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:bc:2b:2a:0e:09:37:04:be:ae:ca:46:bc:0b:39:6f:f6:a2:01:57:0a:04:bd:67:fe:57:a8:8f:91:3b:59:54:66 direction=? spid=4114 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset” SUID=“root”
type=USER_LOGIN msg=audit(1602003451.740:1351): pid=4114 uid=0 auid=4294967295 ses=4294967295 msg='op=login acct=“root” exe="/usr/sbin/sshd" hostname=? addr=222.186.31.166 terminal=ssh res=failed’UID=“root” AUID=“unset”
type=SERVICE_START msg=audit(1602003478.167:1352): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”
type=SERVICE_STOP msg=audit(1602003488.469:1353): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm=“systemd” exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success’UID=“root” AUID=“unset”

The problem is given with the error:

Oct 06 03:28:06 ip-172-31-47-250.us-west-2.compute.internal gravity[1542]: [ERROR]: no network interfaces detected

While it’s true that we don’t have a good start up error reporting story between the installer client/server when the sever fails to start, this error should be taken care of before you can install.
Can you verify that the box has nics and you can use an address as advertise address for the installation?

I managed to find a workaround to get the install to work. I am not sure of the root cause, but the following steps worked for me.Disable firewalld. Install and enable iptabes. flush iptable rules. restart install.

Install an enable iptables.
yum install iptables-services
systemctl start iptables
systemctl enable iptables
service iptables save
iptables --flush