Git and Git branching

If you are new to Git, you will first need to familiarize yourself with distributed version control. One good source is: What is Git: become a pro at Git with this guide | Atlassian Git Tutorial. Also, make sure to read up on Git workflows as they become relevant when we talk about GitOps flows: Git Workflow | Atlassian Git Tutorial.

GitOps

GitOps is a methodology where you operate a system through code. Instead of making changes to the system directly through an API or CLI, you make Git commits. The system pulls in those Git commits itself. The component that we are using to pull in those changes in my current project and Kubernetes (K8s) is called Flux. We move from a push-based model to a pull-based one when using GitOps.

You can read more on the GitOps approach and how it is making good traction in the K8s space here: Guide To GitOps (weave. works).

Continuous Integration(CI) vs. Continuous Deployment(CD)

In GitOps, there is a sharp separation between CI and CD. You will keep the CI more or less exactly as you have it today. You will still do unit tests, static code analysis, generate reports, integrations tests, and the rest of it in your CI pipeline. The CI is left entirely up to the individual teams to manage.

The CD is a bit different. You will keep all of your yaml files in a separate repo away from your application code repos. The goal is to describe your entire system with all the yaml files necessary in that repo. We do not want to make manual changes to the system configuration repo each time there is a new image. Instead, we want to automate it and have the GitOps component make those changes whenever new images become available. Flux has a feature that does exactly that. It listens for new images in a given container registry. The GitOps imageupdate component then updates a given branch and path in the repo with the new image tag and makes a git commit for you.

Managing environment-specific configurations

Sometimes you need to overwrite specific parts of your YAML system configurations for individual environments. Maybe you need a larger replica count in the test environment compared to the development environment, for example. We use a tool called kustomize to make those environment-specific changes. The different environments have Flux installed and configured to run those environment-specific overlays.

Keeping secrets secret in Git

We want to have the entire system configuration stored in Git, including K8s secrets. We use a tool called Sealed Secrets from Bitnami to encrypt the secrets before storing them in Git. The cluster administrators have installed the sealed secrets controller in all clusters and added the private key used for decrypting the sealed secrets. It is then up to the team to use the public key to generate a sealed secret and keep that in the repo instead of the regular K8s secret yaml.

GitOps branching strategy

There are not any fixed rules on how you manage your environments and software releases using GitOps. This post has two suggestions, but please feel free to use the one you want. There is also no best git branching strategy. It is a bit like saying that gasoline is better than water. It all depends on your context.

Goals

What we want to achieve regardless of which approach you use is the following:

• We only build, push and change container images when there are changes in them
• We want to "promote" image versions across environments. For example, after you have tested an image in the test environment, you want to take that exact image and push it into production.
• We want to be as agile as possible and continuously push to production instead of bulking up big releases. It is easier to find bugs with small and rapid changes.

Trunk based workflow

The Trunk-based workflow focuses on getting changes quickly into production. Instead of creating stabilization branches, like with Git flow, we push directly onto the main branch. If you have made a feature branch, you create a PR into the main/trunk branch. The environments pull in the changes and apply them. One of the tenants of trunk-based development is rapid small changes integrated into the trunk each day. The team will typically use feature toggles to hide features that are in progress.

It is also possible to introduce a bit more testing before pushing the change into production. For example, you could have a dev, test, and main branch. You would have the automatic image update functionality in the dev environment that updates the base part in our YAML customize setup, then merge dev into test, and test into prod when deploying into the environments accordingly.

Git Flow

Vincent Driessen introduced git-flow at Nvie. It is a flow that suits teams that want more control of their releases than the trunk-based flow. Teams that maybe have some junior developers where their code needs to be checked by the senior developers. Also, perhaps the team is working in a highly regulated industry where they need to follow many regulations, procedures, and rules before letting the code into production. Compared to the trunk-based model, this flow is more complex. You typically start with a development branch where the individual developers merge features into it with pull requests. Then once the team has a set of new features that they want to have into production, they create a test branch from dev. The team gets feedback from testers working in the test environment about potential bugs and minor changes. The testers should know that missing features or significant changes are for the next release. The developers merge the test branch into production once the test is stable and the testers are happy.

We can create this flow with a couple of steps. The dev environment gets all new images with a particular format, e.g., application:development-1.2.3, and the image update gets committed in the dev overlay on the dev branch. Pretty straightforward.

It is a bit different for test and production. The test environment will be listening for images in another format, e.g., application:version-1.2.3. GitOps will commit for you using the automatic image update policy. However, the change is done in the base kustomize and not in the overlay, in the test branch. For the production overlay, there is no configured automatic image update policy at all. There is no image update policy in prod because we only want to introduce the change in production by a pull request from the test environment. We must remember to merge the test branch back into the development branch after production to get all the small stabilization features into dev.

Conclusion

GitOps help us to keep a sharp separation between continuous integration and continuous deployment. You keep all of your YAML that represents the system in question separate from your application code. You can now change the system configuration without having to run the continuous integration part of your pipeline. We use Flux from Weave Works, Kustomize, part of the Kubectl CLI, and Sealed Secret from Bitnami in our GitOps setup. It is working well.