Container storage solutions with Gravity

I am wondering what is the best way to deploy container storage solutions with Gravity, I ran into a few issues

1- These solutions required privilege containers & Gravitational disable it by default.
I tried installing Rook-ceph, OpenEBS, Rancher Longhorn but not able to run them since all these require some operators run under privileged mode which is disabled in Gravity, I am not able to work around that with pod security policies using system capabilities like SYS_ADMIN since some are triggered from code via k8s API. Can we by default set allow privilege containers in gravity k8s?

2- Allow 2 stage Application deployment?
In my local k8s cluster, I will first install these containers storage solutions and once they are up I will deploy my application so this is like 2 steps process. Can we achieve this is gravity?

@sasha @knisbet any help on this is appreciated, kind of stuck here?

Hey @maaz, sorry I didn’t look at this one right of way since I don’t have all the answers without doing a bit of digging and testing.

Re: Rook / OpenEBS / Rancher Longhorn. We don’t currently have an easy way to enable privileged containers within gravity, although I have encountered privileged containers in other packages as a defaul. Traditionally we’ve tried to promote best practices around security by hard coding those best practices. But like you’re experiencing, we’ve come across a few projects now that don’t provide any guidance on how to run without requiring privileged containers, even though its probably not that hard to do.

The unfortunate answer is we don’t provide an easy way to do this, but I do think allowing or exposing configuration for privileged containers may be something we need to evaluate.

The only workaround I can think of that works with the current release, is it’s possible to extend the planet base image we use within gravity, that changes the systemd units for apiserver and kubelet to allow privileged containers. https://gravitational.com/gravity/docs/pack/#user-defined-base-image
Be careful with this approach though, as Gravity and Planet do tend to be tied together, so you will need to make sure you track our planet releases in order to avoid any surprises.

Re: 2-stage application deployment
This is absolutely possible, it just requires a bit of scripting or programming work to setup the install job. Our hook system for install, actually lets you run just about any container or pod as an install hook. So in your application manifest, when defining the install hook, you can point it to you’re own container which will do the install, or you can just write a script that the container will run. The hook itself will mount the app directory, which means it can be as simple as a script that’s included in you’re app directory to perform the install.

I put together a similar example to this a couple months ago just to demo how to install two helm charts: https://github.com/gravitational/quickstart/blob/kevin/examples/multi-cluster/examples/multi-helm-install/resources/install.sh

The way this works is:

  1. The app.yaml defines an install hook as a separate file (install.yaml)
  2. The install.yaml, just references a random linux image that contains bash, in this case ubuntu:18.04
  3. Gravity when it runs the Job as the install hook, will tweak the job to add mounts for the contents of the application directory, which include an install.sh bash script.
  4. The install.sh bash script runs with whatever steps or logic is embedded in the script. So this could be a very simple kubectl apply -f /var/lib/gravity/resources/step1.yaml, kubectl wait <condition>, kubectl apply -f /var/lib/gravity/resources/step2.yaml script, or something with far more complex logic.

I hope that helps.

Great @knisbet thanks for the help, I will look into it,
a followup question I have, did you tested any PV/Storage solution for onPrem deployment since cloud have their specific solutions?

The most common PV solution I’ve seen for on-prem is old school nfs. Basically as a requirement for installing the software, the customer should also provide an NFS volume. Although the applications using this model have been using gravity for several years when there were less options.

For a more general DB, as part of our enterprise offering, we help our customers and have done some customization for running postgres, cassandra (and pithos), etc.