CI/CD with GitLab & K8s

MG 2021 - 2024

Continuous Integration & Delivery with GitLab and Kubernetes

A number of Continuous Integration / Continuous Delivery projects were delivered as part of implementation of CAPI based platform and workload cluster migrations.

Highlights

packer builds for custom AMIs
atlantis for automation of Terraform IaC MRs
helm test & lint
container security scanning
image publishing to AWS container registry & GitLab registry

One of the client's applications needed to spin up short lived custom EC2 instances for certain workloads. These instances were launched using a custom AMI image that used to be built manually, and wasn't often updated.

As part of the project a new CI/CD pipeline was built that used packer to build the AMI automatically, ensuring that the EC2 instance that packer uses during the build process is spun up in a custom VPC without access from the internet. The pipeline included a lint stage that used packer validate command to ensure the manifests are correct, a build stage that ran packer build with AMI creation disabled for merge requests, and a deploy stage that ran packer build with AMI creation enabled for commits to the main branch.

That allowed catching any syntax issues early, and catching any build issues as part of iteration on the merge request before reaching the main branch.

Atlantis

All of the client's infrastructure was provisioned and managed using IaC (Infrastructure as Code) processes using Terraform. Manifests were stored in git repos in GitLab, changes were reviewed using Merge Requests, and applied by running terraform from the maintainer's workstation after merging.

This was setup very well and processes were in place to avoid issues. This wasn't always possible and sometimes problems would occur anyway, such as terraform state reflecting an MR branch that was not merged, conflicts due to concurrent work, and similar.

In order to mitigate such issues Atlantis was introduced to serve as an agent that ran terraform plan as part of the Merge Request iteration to provide visibility on what a change would do when merged, and ran terraform apply when an MR was merged so all changes were applied sequentially. This eliminated the issues with conflicts and divergence between manifests in the main branch and actual state.

Atlantis setup involved creating an AWS IAM Role that had appropriate access to apply the required infrastructure changes, allowing that role to be assumed by a Kubernetes service account, and deploying the atlantis agent on a Kubernetes cluster (using the upstream Helm chart) ensuring it uses the appropriate service account which allows it to assume the AWS Role through IRSA, and exposing it through ingress so that GitLab hosts can call it via webhooks triggered by events like commit push, merge, or comment.

Helm Test & Lint

A number of new Helm charts were created as part of building of the K8s/CAPI Platform, and each such Helm chart included check, test, and deploy stages in its GitLab CI pipeline by utilising GitLab's include directive to re-use common step definitions from another repository containing CI related code.

The check stage for Helm charts ran two jobs - helm lint to verify that the chart is well formed, and helm template to make sure that the templates render correctly. The test stage utilised helm-unittest plugin to run the charts' unit tests. These allowed to catch any syntax or logic issues quickly and validating that the templates render resource manifests as expected according to different options.

The CI repository, apart from the common pipeline step definitions, also included code to build and publish a docker image with all the tools needed in CI pipelines like kubectl, helm (+plugins), packer (+plugins), etc. That image could then be used in all other repositories' pipelines as base for runtime environment of CI/CD jobs utilising GitLab's docker executor.

Container Security Scanning

All built OCI images were scanned for security vulnerabilities using GitLab's container scanning feature which wraps trivy security scanner to perform static analysis of container images and search for vulnerabilities, misconfigurations, and sensitive information. Any discovered issues were then saved in a scanning report artifact which GitLab displays in the Merge Request.

This was enabled and enforced on all relevant projects through the use of security policy projects and scan execution policies.

Image Publishing

Most of the application as well as utility repos' CI/CD pipelines produced an OCI container image to be used to run the app/tool. Those images were also published to GitLab's OCI registry using docker push and GitLab's job tokens.

Helm charts were also published to the OCI registry utilising Helm's OCI support.