Handle Networking changes in Gravity


#1

If an enterprise put the box behind Proxy or update Node IP address/Gateway/Subnet/Proxy?, what is the best way to handle these changes with gravity?


#2

If you mean behind an HTTP proxy an you’re application requires outbound connections, you can take a look at the runtime environment docs for setting the http_proxy environment variables: https://gravitational.com/gravity/docs/cluster/#configuring-runtime-environment-variables

For changes to a nodes networking configuration, I recommend removing the node, making the changes, and re-adding the node to the cluster. The gravity cluster uses the advertise ip address used at install time in its internal and kubernetes state, and likely won’t react well to changes in the nodes IP address.
https://gravitational.com/gravity/docs/cluster/#removing-a-node
https://gravitational.com/gravity/docs/cluster/#adding-a-node


#3

@knisbet thanks, can we modify runtime variables/trigger that job from our App UI or Gravity UI as well? the goal is ADMIN should be able to do it from our APP UI if possible.

will we have to do same even if it is a single node cluster? if company have DHCP or used AWS where stop/start can change the private IP, then it means they need CLI access to make it up again? any easy way to make to workaround this, like put FQDN for each node instead of IP etc.


#4

Re: modify runtime variables
I don’t think setting runtime configuration exposed in the gravity 5.5 WebUI, however, I do think it should be available in gravity 6.0. If you want to expose this from your own UI, which I assume you’re planning to run within a pod, there is an internal API available, but we don’t currently commit to compatibility on it during upgrades. One thing you can do, is if you look at how we do the hook containers, we mount the gravity binary into the pod as an initContainer, which would allow the UI to exec the gravity process to change the configuration and monitor the change (it can trigger a rolling restart of the cluster).

Example I’ve pulled from one of my clusters:

  initContainers:
  - args:
    - |2

      TMPDIR=/tmp/state /opt/bin/gravity app unpack --service-uid=1000 gravitational.io/dns-app:0.1.0 /var/lib/gravity/resources
      mv /var/lib/gravity/resources/resources/* /var/lib/gravity/resources
      rm -r /var/lib/gravity/resources/resources
    command:
    - /bin/sh
    - -c
    - -e
    env:
    - name: APP_PACKAGE
      value: gravitational.io/dns-app:0.1.0
    image: leader.telekube.local:5000/gravitational/debian-tall:0.0.1
    imagePullPolicy: Always
    name: init
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/gravity/local
      name: gravity
    - mountPath: /tmp/state
      name: state-dir
    - mountPath: /opt/bin
      name: bin
    - mountPath: /usr/local/bin/kubectl
      name: kubectl
    - mountPath: /usr/local/bin/helm
      name: helm
    - mountPath: /etc/ssl/certs
      name: certs
    - mountPath: /var/lib/gravity/resources
      name: resources
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-fbtrb
      readOnly: true
  volumes:
  - hostPath:
      path: /usr/bin
      type: ""
    name: bin
  - hostPath:
      path: /usr/bin/kubectl
      type: ""
    name: kubectl
  - hostPath:
      path: /usr/bin/helm
      type: ""
    name: helm
  - hostPath:
      path: /etc/ssl/certs
      type: ""
    name: certs
  - hostPath:
      path: /var/lib/gravity/local
      type: ""
    name: gravity
  - emptyDir: {}
    name: resources
  - emptyDir: {}
    name: state-dir
  - name: default-token-fbtrb
    secret:
      defaultMode: 420
      secretName: default-token-fbtrb

Re: Will we have to do the same if it’s a single node cluster?
I believe so, I see no of no easy way to tackle changing IP addresses. If you are absolutely sure the single node will never be expanded into a multi-node cluster, it might be possible to set the advertise address to 127.0.0.1 or 127.0.0.2. However, I’ve never tried this configuration, and don’t offer any guarantees.

I think on AWS to tackle this, there is a way to assign secondary interface IPs to a node, and those would be static as the node is taken down, re-created, rebooted, etc. And use this as the IP assignment to gravity, if the plan is to shut down a node when not in use.

Also, I don’t know anything about you’re application, but what we’ve seen at other sites, is to treat the cluster itself as temporal, and provide easy ways to restore cluster state. So for example, take good backups of the app, and just install a new gravity cluster when needed, and restoring the app from backup. It’s not instant of course like booting a node and it figuring itself out while booting, but does force good disaster recovery practices.

If this is something you definitely need, we can always look at it of course through professional services, just the current structure of gravity doesn’t provide an easy way to support changing the IP address on a node.


#5

thanks will give it a try