GitOps

The Shivering-Isles Infrastructure uses GitOps as central concept to maintain the Kubernetes cluster and deploy changes to production. Centralising around git as Single Source of Truth without dynamic state provides an easier way to verify changes. It also reduces the amount of trust put into the CI system by enforcing signed commits on the GitOps operator side.

The current tool of choice to implement GitOps in the Shivering-Isles Infrastructure is FluxCD in combination with a monorepo.

GitOps Security

To secure GitOps based deployments and reduce the risks of compromise, the GitOps deployment in the Shivering-Isles Infrastructure only accepts signed commits. This prevents a deployment of workload if an attackers mananges to push a commit onto the GitOps repository. The git forge itself is in charge of preventing rollbacks in the commit history. Rollbacks could be prevented by using git tags instead of git branches as reference, but are less practical.

Further all secrets stored in the GitOps repository are encrypted using SOPS along with insensitive, but irrelevant information, such as dns names.

Local testing

It has proven to be useful to pre-render certain aspects of the GitOps deployments before pushing them.

To render kustomizations locally this command can be used:

kubectl kustomize ./path/to/kustomization --load-restrictor LoadRestrictionsNone

To improve debugging, the relevant top-level kustomization can be exteded with:

buildMetadata:
  - originAnnotations
  - managedByLabel
  - transformerAnnotations

See the upstream documentation for details.