Gravity crashes after upgrade/install

After upgrade sometime gravity crashes and takes 15-20 mins to come back, is this something expected.

This operation will also remove docker images that you manually pushed to the docker registry on this node. Are you sure?
confirm (yes/no):
Thu Apr 23 07:44:13 UTC Stop registry service
Thu Apr 23 07:44:14 UTC Delete registry state directory /var/lib/gravity/planet/registry
Thu Apr 23 07:44:15 UTC Start registry service
Thu Apr 23 07:44:16 UTC Sync application state with registry

root@10.10.10.10:/mnt/installation# gravity status
panic: getwd: no such file or directory

goroutine 1 [running]:
github.com/gravitational/gravity/lib/utils.must(0x0, 0x3d2c220, 0xc0000fc780, 0x182beb9)
        /gopath/src/github.com/gravitational/gravity/lib/utils/commands.go:131 +0x54
github.com/gravitational/gravity/lib/utils.init.ializers()
        /gopath/src/github.com/gravitational/gravity/lib/utils/commands.go:62 +0x5d

Even sometime after installation too.

root@10.10.10.10:/home/ubuntu# gravity status
runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailable
pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f3bbf9ea428 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f3bbf9ea428
stack: frame={sp:0x7ffcee9e9fd8, fp:0x0} stack=[0x7ffcee1eb548,0x7ffcee9ea570)
00007ffcee9e9ed8:  000000c0000eb3c0  0000000000000001
00007ffcee9e9ee8:  0000000000000010  000000000000000f
00007ffcee9e9ef8:  0000000000000008  0000000000000000
00007ffcee9e9f08:  000000c0000eb388  0000000000000001
00007ffcee9e9f18:  000000000000000e  0000000000000000
00007ffcee9e9f28:  0000000000000000  0000000000000000
00007ffcee9e9f38:  0000000000000000  082f493013020000
00007ffcee9e9f48:  2403c6006406e600  000000000000ffff
00007ffcee9e9f58:  000000ff000000ff  0000000000000005
00007ffcee9e9f68:  0000000000000000  0000000000000000
00007ffcee9e9f78:  0000000e00000007  0000000000000002
00007ffcee9e9f88:  8000000000000006  0000000000000000
00007ffcee9e9f98:  0000000000000000  0000000000000000
00007ffcee9e9fa8:  00007f3bbfd7a700  0000000003ea83bb
00007ffcee9e9fb8:  00000000067f4420  0000000000000011
00007ffcee9e9fc8:  0000000003c95bb4  0000000000000000
00007ffcee9e9fd8: <00007f3bbf9ec02a  0000000000000020
00007ffcee9e9fe8:  0000000000000000  0000000000000000
00007ffcee9e9ff8:  0000000000000000  0000000000000000
00007ffcee9ea008:  0000000000000000  0000000000000000
00007ffcee9ea018:  0000000000000000  0000000000000000
00007ffcee9ea028:  0000000000000000  0000000000000000
00007ffcee9ea038:  0000000000000000  0000000000000000
00007ffcee9ea048:  0000000000000000  0000000000000000
00007ffcee9ea058:  0000000000000000  0000000000000000
00007ffcee9ea068:  000000000000000a  00007f3bbfd7b770
00007ffcee9ea078:  00007f3bbfa2abc8  00007f3bbfd7a540
00007ffcee9ea088:  0000000000000001  00007f3bbfd7a5c3
00007ffcee9ea098:  0000000000000011  0000000003c95bb4
00007ffcee9ea0a8:  00007f3bbfa2f409  00007f3bbfd7a540
00007ffcee9ea0b8:  00007f3bbfd7a540  000000000000000a
00007ffcee9ea0c8:  00000000067f4420  0000000000000011
runtime: unknown pc 0x7f3bbf9ea428
stack: frame={sp:0x7ffcee9e9fd8, fp:0x0} stack=[0x7ffcee1eb548,0x7ffcee9ea570)
00007ffcee9e9ed8:  000000c0000eb3c0  0000000000000001
00007ffcee9e9ee8:  0000000000000010  000000000000000f
00007ffcee9e9ef8:  0000000000000008  0000000000000000
00007ffcee9e9f08:  000000c0000eb388  0000000000000001
00007ffcee9e9f18:  000000000000000e  0000000000000000
00007ffcee9e9f28:  0000000000000000  0000000000000000
00007ffcee9e9f38:  0000000000000000  082f493013020000
00007ffcee9e9f48:  2403c6006406e600  000000000000ffff
00007ffcee9e9f58:  000000ff000000ff  0000000000000005
00007ffcee9e9f68:  0000000000000000  0000000000000000
00007ffcee9e9f78:  0000000e00000007  0000000000000002
00007ffcee9e9f88:  8000000000000006  0000000000000000
00007ffcee9e9f98:  0000000000000000  0000000000000000
00007ffcee9e9fa8:  00007f3bbfd7a700  0000000003ea83bb
00007ffcee9e9fb8:  00000000067f4420  0000000000000011
00007ffcee9e9fc8:  0000000003c95bb4  0000000000000000
00007ffcee9e9fd8: <00007f3bbf9ec02a  0000000000000020
00007ffcee9e9fe8:  0000000000000000  0000000000000000
00007ffcee9e9ff8:  0000000000000000  0000000000000000
00007ffcee9ea008:  0000000000000000  0000000000000000
00007ffcee9ea018:  0000000000000000  0000000000000000
00007ffcee9ea028:  0000000000000000  0000000000000000
00007ffcee9ea038:  0000000000000000  0000000000000000
00007ffcee9ea048:  0000000000000000  0000000000000000
00007ffcee9ea058:  0000000000000000  0000000000000000
00007ffcee9ea068:  000000000000000a  00007f3bbfd7b770
00007ffcee9ea078:  00007f3bbfa2abc8  00007f3bbfd7a540
00007ffcee9ea088:  0000000000000001  00007f3bbfd7a5c3
00007ffcee9ea098:  0000000000000011  0000000003c95bb4
00007ffcee9ea0a8:  00007f3bbfa2f409  00007f3bbfd7a540
00007ffcee9ea0b8:  00007f3bbfd7a540  000000000000000a
00007ffcee9ea0c8:  00000000067f4420  0000000000000011

goroutine 1 [chan receive, locked to thread]:
text/template/parse.(*lexer).nextItem(...)
        /go/src/text/template/parse/lex.go:194
text/template/parse.(*Tree).peek(...)
        /go/src/text/template/parse/parse.go:95
text/template/parse.(*Tree).parse(0xc00016c000)
        /go/src/text/template/parse/parse.go:277 +0xbc1
text/template/parse.(*Tree).Parse(0xc00016c000, 0x37c2bf9, 0x12b, 0x0, 0x0, 0x0, 0x0, 0xc00014b9b0, 0xc000085630, 0x2, ...)
        /go/src/text/template/parse/parse.go:230 +0x215
text/template/parse.Parse(0x36d5476, 0xb, 0x37c2bf9, 0x12b, 0x0, 0x0, 0x0, 0x0, 0xc000085630, 0x2, ...)
        /go/src/text/template/parse/parse.go:55 +0x125
text/template.(*Template).Parse(0xc000096ac0, 0x37c2bf9, 0x12b, 0xc00014b860, 0xc00014b890, 0xc00014b8c0)
        /go/src/text/template/template.go:196 +0x113
html/template.(*Template).Parse(0xc00014b980, 0x37c2bf9, 0x12b, 0x0, 0x0, 0x0)
        /go/src/html/template/template.go:189 +0x85
github.com/gravitational/gravity/vendor/github.com/gravitational/trace.init.ializers()
        /gopath/src/github.com/gravitational/gravity/vendor/github.com/gravitational/trace/trace.go:597 +0x92

goroutine 5 [chan receive]:
github.com/gravitational/gravity/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x6798120)
        /gopath/src/github.com/gravitational/gravity/vendor/k8s.io/klog/klog.go:941 +0x8b
created by github.com/gravitational/gravity/vendor/k8s.io/klog.init.0
        /gopath/src/github.com/gravitational/gravity/vendor/k8s.io/klog/klog.go:403 +0x6c

goroutine 6 [runnable]:
text/template/parse.(*lexer).run(0xc0000ca280)
        /go/src/text/template/parse/lex.go:226
created by text/template/parse.lex
        /go/src/text/template/parse/lex.go:221 +0x134

rax    0x0
rbx    0x7f3bbfd7a700
rcx    0x7f3bbf9ea428
rdx    0x6
rdi    0x1154
rsi    0x1154
rbp    0x3ea83bb
rsp    0x7ffcee9e9fd8
r8     0x7f3bbfd7b770
r9     0x7f3bc01b8700
r10    0x8
r11    0x202
r12    0x67f4420
r13    0x11
r14    0x3c95bb4
r15    0x0
rip    0x7f3bbf9ea428
rflags 0x202
cs     0x33
fs     0x0
gs     0x0```

Hi @mtariq,

Which version are you upgrading to and from of Gravity? Any logs or errors you’ve captured that you can share?

Thanks

The first error about getwd indicates that the directory you’re running the command from suddenly disappears - is this possible in your setup? What kind of mount point is the /mnt/installation directory?

The second error seems indicative of a system resource starvation - can you provide more info about the environment - is this a VM or a hardware node?
Also if you could look into kernel log (dmesg) and share any errors regarding resource starvation - it would be helpful.