Multi-Chart Example is not working

I will start by describing my problem and then go into detail about some discrepancies I am seeing.

I am using the example helm multi-chart example from the quickstart github repo.


In that example it uses the app.yaml to call the install.yaml using the install hook, which then calls an install.sh which executes a couple commands to launch multiple helm charts. This makes sense to me but when I run the example it hangs on the install hook and times out. Here is the console output:

Tue Dec 17 21:47:36 UTC Install application mattermost:2.2.0
Tue Dec 17 21:47:46 UTC         Still executing install hook for mattermost:2.2.0 (10 seconds 
elapsed)

The install hook is essentially just executing the same command but inside a shell script, instead of the install.yaml command variable. I have even tried just doing a single chart install using this shell command method and it still won’t work. Not sure if its having trouble accessing the local repo? When I just do a normal install without the shell script it all works fine.

Something else to note it that the manifest file app.yaml is different for this multi-chart vs the single chart example. The single chart uses:
apiVersion: cluster.gravitational.io/v2
kind: Cluster
metadata:
name: mattermost
resourceVersion: “2.2.0”

Where the multi-chart uses:
apiVersion: bundle.gravitational.io/v2
kind: Bundle
metadata:
name: mattermost
resourceVersion: “2.2.0”

I can’t find anywhere in the documentation where bundle is used, I have tried both bundle and chart for the shell version to no avail.

Furthermore, inside the shell script is tries to use resources from /var/lib/gravity/resources/charts/mattermost, but the resources directory doesn’t exist for me. Maybe it gets deleted during the instillation process?

I’m just a little lost here and am trying to get this basic multichart example to work before I switch to a more complicated personal multichart use case.

Sorry for the confusion, bundle and cluster are the same thing. We’ve been updating our nomenclature to try and make the applications more understandable. So a cluster manifest is used to create a cluster image, which is the installer tarball. However, you will still see bundle around in places we’ve missed updating this nomenclature.

Depending on the version of gravity you are using, the install log or system log(/var/log/gravity-install.log or /var/log/gravity-system.log) will print the container logs when the step times out. Alternatively, the cluster is largely operational at this point, so you can do gravity shell to get a shell within the planet container, and then use kubectl to get the logs for the install container. This will be a job and pod within the kube-system namespace by default.

Without seeing what’s happening within that container, the sort of normal culprits would be a typo in the install script.

It’s not clear to me whether you are looking within the host system or the pod. This path is within the install container, we create a init container that extracts the resources included with the build to this path. So you won’t see it on the host, it is only included with the container.

Encountered the same problem in non-airgapped environment. We are trying to work out the multiple helm charts scenarios without any post-install hook. Still not sure where to look for the issue. To be specific we followed the multi-helm example and recreated the same env for our helm charts by deploying them using a simple script which was just performing the commands like

#!/bin/bash
set -x

echo "Installing chart1..."
/usr/local/bin/helm install /var/lib/gravity/resources/charts/chart1

echo "Installing chart2..."
/usr/local/bin/helm install /var/lib/gravity/resources/charts/chart2 

and so on for about 4 charts that we require. The charts on the host machine are kept in

some_folder/resources/charts/

To be elaborative enough we used the following command to build the tar

tele build -o some.tar some_folder/resources/app.yaml

app.yaml has everything similar to the mattermost example like with only one install hook

hooks:
  install:
    job: file://install.yaml

And finally the install.yaml is

 apiVersion: batch/v1
 kind: Job
 metadata:
   name: install
 spec:
   template:
     metadata:
       name: install
     spec:
       restartPolicy: OnFailure
       containers:
         - name: install
           image: quay.io/gravitational/debian-tall:0.0.1
           command: ["/bin/bash", "-c", "/var/lib/gravity/resources/install.sh"]

On gravity install the installer fails with the given logs ->

Logs of gravity-install

<omitted output>
Wed Dec 18 05:17:44 UTC [DEBUG] [aws08] Application gravitational.io/tiller-app:6.2.0 does not have postInstall hook.
Wed Dec 18 05:17:45 UTC [INFO] [aws08] Executing phase: /runtime/site.
Wed Dec 18 05:17:45 UTC [INFO] [aws08] Executing install hook for site:6.2.5.
Created Pod "gravity-install-fcef2f-m7jcj" in namespace "kube-system".

Container "gravity-install" created, current state is "waiting, reason PodInitializing".

Pod "gravity-install-fcef2f-m7jcj" in namespace "kube-system", has changed state from "Pending" to "Running".
Container "gravity-install" changed status from "waiting, reason PodInitializing" to "running".

2019-12-18T05:18:01Z DEBU             got search paths: [/var/lib/gravity/resources/config] processconfig/config.go:57
2019-12-18T05:18:01Z DEBU             look up configs in /var/lib/gravity/resources/config processconfig/config.go:59
2019-12-18T05:18:01Z DEBU             using ETCD backend processconfig/config.go:237
2019-12-18T05:18:01Z INFO [PROCESS]   Process ID: 172.31.40.31. mode:site process/process.go:285
2019-12-18T05:18:01Z INFO [PROCESS]   Teleport config: &service.Config{DataDir:"/var/lib/gravity/site/teleport", Hostname:"aws08", Token:"", AuthServers:[]utils.NetAddr{utils.NetAddr{Addr:"0.0.0.0:3025", AddrNetwork:"tcp", Path:""}}, Identities:[]*auth.Identity(nil), AdvertiseIP:"", CachePolicy:service.CachePolicy{Enabled:true, TTL:0, NeverExpires:false, RecentTTL:(*time.Duration)(nil)}, SSH:service.SSHConfig{Enabled:true, Addr:utils.NetAddr{Addr:"0.0.0.0:3022", AddrNetwork:"tcp", Path:""}, Namespace:"", Shell:"/bin/bash", Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, Labels:map[string]string(nil), CmdLabels:services.CommandLabels(nil), PermitUserEnvironment:false, PAM:(*pam.Config)(0xc0007cd020), PublicAddrs:[]utils.NetAddr(nil)}, Auth:service.AuthConfig{Enabled:true, EnableProxyProtocol:true, SSHAddr:utils.NetAddr{Addr:"0.0.0.0:3025", AddrNetwork:"tcp", Path:""}, Authorities:[]services.CertAuthority(nil), Roles:[]services.Role(nil), ClusterName:(*services.ClusterNameV2)(0xc0007f5f10), StaticTokens:(*services.StaticTokensV2)(0xc000acb280), StorageConfig:backend.Config{Type:"dir", Params:backend.Params{"path":"/var/lib/gravity/site/teleport"}}, Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, NoAudit:false, Preference:(*services.AuthPreferenceV2)(0xc0007f77c0), ClusterConfig:(*services.ClusterConfigV3)(0xc0003c6b40), LicenseFile:"/var/lib/teleport/license.pem", PublicAddrs:[]utils.NetAddr(nil)}, Keygen:sshca.Authority(nil), Proxy:service.ProxyConfig{Enabled:true, DisableTLS:false, DisableWebInterface:true, DisableWebService:true, DisableReverseTunnel:false, ReverseTunnelListenAddr:utils.NetAddr{Addr:"0.0.0.0:3024", AddrNetwork:"tcp", Path:""}, EnableProxyProtocol:true, WebAddr:utils.NetAddr{Addr:"0.0.0.0:3080", AddrNetwork:"tcp", Path:""}, SSHAddr:utils.NetAddr{Addr:"0.0.0.0:3023", AddrNetwork:"tcp", Path:""}, TLSKey:"", TLSCert:"", Limiter:limiter.LimiterConfig{Rates:[]limiter.Rate(nil), MaxConnections:1000, MaxNumberOfUsers:250, Clock:timetools.TimeProvider(nil)}, PublicAddrs:[]utils.NetAddr(nil), SSHPublicAddrs:[]utils.NetAddr(nil), Kube:service.KubeProxyConfig{Enabled:false, ListenAddr:utils.NetAddr{Addr:"0.0.0.0:3026", AddrNetwork:"tcp", Path:""}, APIAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, ClusterOverride:"", CACert:[]uint8(nil), PublicAddrs:[]utils.NetAddr(nil), KubeconfigPath:""}}, HostUUID:"", Console:(*io.PipeWriter)(0xc000010ef8), ReverseTunnels:[]services.ReverseTunnel(nil), OIDCConnectors:[]services.OIDCConnector(nil), PIDFile:"", Trust:(*usersservice.UsersService)(0xc0008f9b00), Presence:(*keyval.electingBackend)(0xc0008f9ad0), Provisioner:(*usersservice.UsersService)(0xc0008f9b00), Identity:(*usersservice.UsersService)(0xc0008f9b00), Access:(*usersservice.UsersService)(0xc0008f9b00), ClusterConfiguration:(*usersservice.UsersService)(0xc0008f9b00), CipherSuites:[]uint16{0xcca8, 0xcca9, 0xc02f, 0xc02b, 0xc030, 0xc02c, 0x9c, 0x9d}, Ciphers:[]string{"aes128-gcm@openssh.com", "aes128-ctr", "aes192-ctr", "aes256-ctr"}, KEXAlgorithms:[]string{"curve25519-sha256@libssh.org", "ecdh-sha2-nistp256", "ecdh-sha2-nistp384", "ecdh-sha2-nistp521"}, MACAlgorithms:[]string{"hmac-sha2-256-etm@openssh.com", "hmac-sha2-256"}, DiagnosticAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, Debug:false, UploadEventsC:(chan *events.UploadEvent)(nil), FileDescriptors:[]service.FileDescriptor(nil), PollingPeriod:10000000000, ClientTimeout:0, ShutdownTimeout:0, CAPin:"", Clock:clockwork.Clock(nil)}. mode:site process/process.go:302
2019-12-18T05:18:01Z INFO [PROCESS]   Gravity config: processconfig.Config{Hostname:"leader.telekube.local", Mode:"site", Profile:processconfig.ProfileConfig{HTTPEndpoint:"", OutputDir:""}, Devmode:false, ClusterName:"", WebAssetsDir:"", DataDir:"/var/lib/gravity/site", HealthAddr:utils.NetAddr{Addr:"0.0.0.0:3010", AddrNetwork:"tcp", Path:""}, BackendType:"etcd", ETCD:keyval.ETCDConfig{Clock:clockwork.Clock(nil), Nodes:[]string{"https://127.0.0.1:2379"}, Key:"/gravity/local", TLSKeyFile:"/var/lib/gravity/secrets/etcd.key", TLSCertFile:"/var/lib/gravity/secrets/etcd.cert", TLSCAFile:"/var/lib/gravity/secrets/root.cert", RetryInterval:0}, OpsCenter:processconfig.OpsCenterConfig{SeedConfig:(*ops.SeedConfig)(nil)}, Pack:processconfig.PackageServiceConfig{ListenAddr:utils.NetAddr{Addr:"0.0.0.0:3009", AddrNetwork:"tcp", Path:""}, PublicListenAddr:utils.NetAddr{Addr:"0.0.0.0:3007", AddrNetwork:"tcp", Path:""}, AdvertiseAddr:utils.NetAddr{Addr:"172.31.40.31:3009", AddrNetwork:"tcp", Path:""}, PublicAdvertiseAddr:utils.NetAddr{Addr:"", AddrNetwork:"", Path:""}, ReadDir:""}, Charts:processconfig.ChartsConfig{Backend:"local"}, Users:processconfig.Users(nil), InstallLogFiles:[]string(nil), ImportDir:"", ServiceUser:(*systeminfo.User)(0xc000a986c0), InstallToken:""}. mode:site process/process.go:303
2019-12-18T05:18:01Z DEBU [PROCESS]   Init from "/opt/gravity-import". mode:site process/process.go:464
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/cert-authority:0.0.1. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/planet-172.31.40.31-secrets:6.2.4-11603. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/planet-config-172314031Aspace:6.2.4-11603. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/site-export:0.0.1. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/teleport-master-config-172314031Aspace:3.2.13. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at Aspace/teleport-node-config-172314031Aspace:3.2.13. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/bandwagon:6.0.1. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/dns-app:0.3.0. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/gravity:6.2.5. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/kubernetes:6.2.5. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/logging-app:6.0.2. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/monitoring-app:6.0.4. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/planet:6.2.4-11603. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/rbac-app:6.2.5. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/site:6.2.5. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/sygmoid-installer:2.2.0. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/teleport:3.2.13. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/tiller-app:6.2.0. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z INFO [IMPORTER]  Looking at gravitational.io/web-assets:6.2.5. dir:/opt/gravity-import process/import.go:89
2019-12-18T05:18:01Z DEBU [IMPORTER]  Importing cluster data. dir:/opt/gravity-import process/import.go:200
2019-12-18T05:18:01Z DEBU [IMPORTER]  Importing packages. dir:/opt/gravity-import process/import.go:164
2019-12-18T05:18:01Z DEBU [BLOB]      Got enough success writes for cb5a15b5d6011dd6488d547c3419e92de3e48f1be3183cd1b65f5cc952253c6a [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:01Z DEBU [IMPORTER]  Imported Aspace/cert-authority:0.0.1 in 85.111898ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:01Z DEBU [BLOB]      Got enough success writes for 03fb56007e9d649f4b9edaec1f45bb8d0d6a2a6d4a55ead5ee9a0e42ef4b572c [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:01Z DEBU [IMPORTER]  Imported Aspace/planet-172.31.40.31-secrets:6.2.4-11603 in 61.228124ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:01Z DEBU [BLOB]      Got enough success writes for 5f08ab7703218ed9538abe43d89de145cc63fed808cbfd9d7f74d75905d11f74 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:01Z DEBU [IMPORTER]  Imported Aspace/planet-config-172314031Aspace:6.2.4-11603 in 98.59177ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:01Z DEBU [BLOB]      Got enough success writes for a4c40e0d74ca2dbbbc12d1bb2f32b45f76e3839f446217a95e4bdcab4316a8ea [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:01Z DEBU [IMPORTER]  Imported Aspace/teleport-master-config-172314031Aspace:3.2.13 in 106.194973ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:02Z DEBU [BLOB]      Got enough success writes for f2c63d03f61e1b65a3b0cdef1174fbf7e8930d1921ed12dd044ce0525ae5a763 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:02Z DEBU [IMPORTER]  Imported Aspace/teleport-node-config-172314031Aspace:3.2.13 in 41.815775ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:03Z DEBU [BLOB]      Got enough success writes for a03a63849b666ab7656ba504cd637874f891ccb3967e12a971b83d6cd2036967 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:03Z DEBU [IMPORTER]  Imported gravitational.io/bandwagon:6.0.1 in 1.587558364s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:04Z DEBU [BLOB]      Got enough success writes for cb80d761c2b3c6a26e174d83a6eb8a96871cac4cd66cd5b55108bf5854dc78b6 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:04Z DEBU [IMPORTER]  Imported gravitational.io/dns-app:0.3.0 in 908.807311ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:06Z DEBU [BLOB]      Got enough success writes for d94a58c2412b6c680311b30a32bc5c1689da58156d716aa5c73b3c798c87aa3a [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:06Z DEBU [IMPORTER]  Imported gravitational.io/gravity:6.2.5 in 1.930347883s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:06Z DEBU [BLOB]      Got enough success writes for ce34ee5f7df69a20878be6d370ed8772d8c34c6d7e31421602a43c7e719b4400 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:06Z DEBU [IMPORTER]  Imported gravitational.io/kubernetes:6.2.5 in 90.299437ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:07Z DEBU [BLOB]      Got enough success writes for 55e31b086464ed17a16e27a63ba83e5cb7a6e9d878e917b071f6795be8da17b8 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:07Z DEBU [IMPORTER]  Imported gravitational.io/logging-app:6.0.2 in 1.403867378s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:10Z DEBU [BLOB]      Got enough success writes for 7eeec679755fa0e9446e381fd4b73e3529b76643b3d58f63ecc868b8801960a1 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:10Z DEBU [IMPORTER]  Imported gravitational.io/monitoring-app:6.0.4 in 2.937624813s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:16Z DEBU [BLOB]      Got enough success writes for 9df345dd7aa74ae48d0c7d2fcce6627b71f3f2b461b96981d8c1f36636a03195 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:16Z DEBU [IMPORTER]  Imported gravitational.io/planet:6.2.4-11603 in 5.207132594s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:16Z DEBU [BLOB]      Got enough success writes for 894391cf76884a72fe6cc71140a508e7a16271a12fb17b261405f44d4d7e3585 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:16Z DEBU [IMPORTER]  Imported gravitational.io/rbac-app:6.2.5 in 68.075505ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:16Z DEBU [BLOB]      Got enough success writes for 96920199b6450f7f987c939b9ff779035520856b99aa3f3983312e53396e99bb [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:16Z DEBU [IMPORTER]  Imported gravitational.io/site:6.2.5 in 435.298334ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:18Z DEBU [BLOB]      Got enough success writes for 2a3f093d9292850e0fb995bfc1eb54a5e6e5641bf686f46f9aec9ef88a14af66 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:18Z DEBU [IMPORTER]  Imported gravitational.io/sygmoid-installer:2.2.0 in 1.880985352s. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:18Z DEBU [BLOB]      Got enough success writes for 32f0f175806cd35de2c50f5a771e7ff4d59d8719fc0b8777697782dd1201d38f [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:18Z DEBU [IMPORTER]  Imported gravitational.io/teleport:3.2.13 in 409.65983ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:19Z DEBU [BLOB]      Got enough success writes for 084390f3ca0d2dd805b91f70eb1ccb844a110cb70b13fa59e553e96235c0d049 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:19Z DEBU [IMPORTER]  Imported gravitational.io/tiller-app:6.2.0 in 243.417898ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:19Z DEBU [BLOB]      Got enough success writes for 6f60a0a25ea6642186587eab7dd7ead7b15f7c0cb269ed0ebe0113debd0310a3 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:19Z DEBU [IMPORTER]  Imported gravitational.io/web-assets:6.2.5 in 250.241564ms. dir:/opt/gravity-import process/import.go:188
2019-12-18T05:18:19Z DEBU [IMPORTER]  Closing backend. dir:/opt/gravity-import process/import.go:109
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generate received request runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] received CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] signed certificate with serial number 262540636618560386158188552329125601178111394437 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generate received request runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] received CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] signed certificate with serial number 289125573576691353172630590606569663296411103074 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generate received request runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] received CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] generating key: rsa-2048 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] encoded CSR runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z INFO             2019/12/18 05:18:19 [INFO] signed certificate with serial number 22651691747850937588004698011244423357514672252 runtime/asm_amd64.s:1337
2019-12-18T05:18:19Z DEBU [BLOB]      Got enough success writes for 70ee955a415090a9dd8e959804b87cbb9050d2ac2f3018b549503fcc018851d1 [172.31.40.31]. addr:https://172.31.40.31:3009 id:172.31.40.31 cluster/cluster.go:305
2019-12-18T05:18:19Z INFO [PROCESS]   Initialized RPC credentials: gravitational.io/rpcagent-secrets:0.0.1. mode:site process/process.go:489
configmap/gravity-opscenter created
Creating cluster configmap
configmap/gravity-site created
serviceaccount/gravity-site created
role.rbac.authorization.k8s.io/gravity-site created
rolebinding.rbac.authorization.k8s.io/gravity-site created
clusterrole.rbac.authorization.k8s.io/gravity-site created
clusterrolebinding.rbac.authorization.k8s.io/gravity-site created
daemonset.apps/gravity-site created
service/gravity-site created
role.rbac.authorization.k8s.io/gravity-site created
rolebinding.rbac.authorization.k8s.io/gravity-site created
Pod "gravity-install-fcef2f-m7jcj" in namespace "kube-system", has changed state from "Running" to "Succeeded".
Container "gravity-install" changed status from "running" to "terminated, exit code 0".

Wed Dec 18 05:18:27 UTC [INFO] [aws08] Executing postInstall hook for site:6.2.5.
Created Pod "site-app-post-install-71d352-rh4n8" in namespace "kube-system".

Container "post-install-hook" created, current state is "waiting, reason PodInitializing".

Pod "site-app-post-install-71d352-rh4n8" in namespace "kube-system", has changed state from "Pending" to "Running".
Container "post-install-hook" changed status from "waiting, reason PodInitializing" to "running".

[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz
Container "post-install-hook" changed status from "running" to "terminated, exit code 255".

Container "post-install-hook" restarted, current state is "running".

[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz
Container "post-install-hook" changed status from "running" to "terminated, exit code 255".

Container "post-install-hook" changed status from "terminated, exit code 255" to "waiting, reason CrashLoopBackOff".

Pod "site-app-post-install-71d352-rh4n8" in namespace "kube-system", has changed state from "Running" to "Succeeded".
Container "post-install-hook" restarted, current state is "terminated, exit code 0".

Wed Dec 18 05:18:53 UTC [INFO] [aws08] Executing phase: /runtime/kubernetes.
Wed Dec 18 05:18:53 UTC [DEBUG] [aws08] Application gravitational.io/kubernetes:6.2.5 does not have install hook.
Wed Dec 18 05:18:53 UTC [DEBUG] [aws08] Application gravitational.io/kubernetes:6.2.5 does not have postInstall hook.
Wed Dec 18 05:18:55 UTC [INFO] [aws08] Executing phase: /app/sygmoid-installer.
Wed Dec 18 05:18:55 UTC [INFO] [aws08] Executing install hook for sygmoid-installer:2.2.0.
Created Pod "install-60b3ee-782pn" in namespace "kube-system".
Container "install" created, current state is "waiting, reason PodInitializing".

Pod "install-60b3ee-782pn" in namespace "kube-system", has changed state from "Pending" to "Running".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Container "install" restarted, current state is "waiting, reason RunContainerError".

Wed Dec 18 05:25:00 UTC [ERROR] [aws08] Phase execution failed: Job has reached the specified backoff limit, gravitational.io/sygmoid-installer:2.2.0 install hook failed.

I am able to connect to https://gravity-site.kube-system.svc.cluster.local:3009/healthz from inside the pod as mentioned in of the github issues

ubuntu@aws08:~/workspace/gravity$ kubectl run new -it --rm --image=alpine
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.

/ # nslookup kubernetes.default.svc.cluster.local.
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default.svc.cluster.local.
Address 1: 10.100.0.1 kubernetes.default.svc.cluster.local

/ # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20190108-r0)
(2/4) Installing nghttp2-libs (1.39.2-r0)
(3/4) Installing libcurl (7.66.0-r0)
(4/4) Installing curl (7.66.0-r0)
Executing busybox-1.30.1-r2.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 7 MiB in 18 packages


/ # curl -k https://gravity-site.kube-system.svc.cluster.local:3009/healthz 
{"info":"service is up and running","status":"ok"}/ # 

Even did the exact same thing after

sudo ./gravity exec -i bash 

and then ran the above commands of curl which we returning success output.
sygmoid-installer is the name of our installer.

Got this working. The issue was the install.sh was not having executable permissions. I still don’t know how gravity is working this thing out because in the install.yaml it is mentioned to use /bin/bash

"/bin/bash", "-c", "/var/lib/gravity/resources/install.sh"

If we are using /bin/bash then what is the need of executable permission on install.sh

The build process simply tar’s up the application directory, and untar’s it back to /var/lib/gravity/resources for the hook to use it. So this process maintains the permissions that were on the file when the application was built.

I believe this has to do with passing the -c parameter, which according to the man page is for passing a string of commands. As such install.sh becomes a command, just like invoking a program, where the kernel will need to read the header, and pick an interpreter to run the script.

If the -c option wasn’t passed, bash would like run the file simply as an interpreter. Using the -c parameter is likely just a carry over from the way it was written previously, where I was trying very quickly to demo how to run a script from the install hook.

Thanks for your reply, I have understood now. But it would be a kind help if you mention this in your doc or on the github page.