Recover master from Out of disk

#1

gravity master stop when free disk is left less than 3GB and was pointing to https://gravitational.com/gravity/docs/cluster/#garbage-collection,
Does not gravity auto purge logs? I did removing logs from /var/lib/gravity/planet/log but now I am seeing below status, what is the best way to make it run for a long with auto rotate logs?

gravity status
Cluster status:		degraded
Application:		XXX-XXX, version 0.0.1
Join token:		XXXXXX
Periodic updates:	Not Configured
Remote support:		Not Configured
Active operations:
    * operation_gc (3701c232-736f-4340-be19-a34b397c3b46)
      started:	Fri Apr 12 01:11 UTC (3 days ago)
      Prune unused docker images on node "ubuntu", 0% complete
Last completed operation:
    * operation_install (d4603403-888f-41ea-a615-e64916bd96f0)
      started:		Tue Apr  9 21:32 UTC (5 days ago)
      completed:	Tue Apr  9 21:32 UTC (5 days ago)
Cluster endpoints:
    * Authentication gateway:
        - 192.168.86.169:32009
    * Cluster management URL:
        - https://192.168.86.169:32009
Cluster nodes:	XXXX-XXXXXX
    Masters:
        * ubuntu (192.168.86.169, node)
            Status:	degraded
            [×]		Unable to create inotify watch (no space left on device)

Gravity GC also seems stuck

gravity gc
This operation will also remove docker images that you manually pushed to the docker registry. Are you sure?
confirm (yes/no):
yes
[ERROR]: the cluster is collecting_garbage

0 Likes

#2

Hi maaz,

You have to increase number of inotify watches. Also /var/lib/gravity/planet/log is a directory for journald logs; it is not recommended to delete something from there manually. From the inside of gravity you can vaccum logs with journalctl --vacuum-size=500M or journalctl --vacuum-time=2d.

0 Likes