Continuous Integration for helm charts

I’ve recently been working on a comprehensive helm chart for docker-mailserver.

Since it’s a complicated chart with some logic re which features are in/out, I thought it’d be helpful to setup some continuous integration (CI), so that I can have CI test for valid syntax, as well as run unit and regression testing as I make ongoing changes.

I did a bit of searching, and didn’t find all that much around an established way to do CI with helm charts. After some effort, I’m quite satisfied with the level of coverage I have, so I wanted to share what works for me.

Helm syntax

To give me confidence that my chart will work when a user deploys it (and this found me lots of bugs!), I first run it through helm’s standard --lint command. This confirms that my chart template syntax is good - I haven’t left out any template fields, and I don’t have any unbalanced if/else statements. It’s quick to do, and catches the first basic level of errors:

# helm lint docker-mailserver
==> Linting docker-mailserver
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, no failures
# add-helm-chart ±

Kubernetes manifest syntax

The helm lint doesn’t tell me whether or not Kubernetes will accept my generated manifests though. For this, I use @garethr’s kubeval package. kubevel takes a kubernetes version number (i.e. 1.13.0) as an argument, and then validates any manifests passed to it via stdin. (note: this is also a great way to confirm your chart is compatible with older versions of Kubernetes)

In theory, evaluating all generated manifests is as easy as running helm template <my chart> | kubeval -v 1.13.0.

Unfortunately, kubeval currently can’t process CustomResourceDefinitions (which I’m using to create certificates with cert-manager), so I have to resort to a dirty-but-effective workaround. Any custom resource templates have “-crd” suffixed to their names, and then I exclude them from kubeval processing like this:

mkdir manifests
helm template helm-chart/docker-mailserver --output-dir manifests
find manifests/ -name '*.yaml' | grep -v crd | xargs kubeval -v $KUBERNETES_VERSION

Here’s some truncated example output:

# find manifests/ -name '*.yaml' | grep -v crd | xargs kubeval -v 1.13.0
<snip>
The document manifests//docker-mailserver/templates/service_rainloop.yaml contains a valid Service
The document manifests//docker-mailserver/templates/service.yaml is empty
The document manifests//docker-mailserver/templates/service.yaml contains a valid Service
The document manifests//docker-mailserver/templates/pvc.yaml is empty
The document manifests//docker-mailserver/templates/pvc.yaml contains a valid PersistentVolumeClaim
The document manifests//docker-mailserver/templates/ingress_rainloop.yaml contains a valid Ingress
The document manifests//docker-mailserver/templates/deployment_rainloop.yaml is empty
The document manifests//docker-mailserver/templates/deployment_rainloop.yaml contains a valid Deployment
The document manifests//docker-mailserver/templates/configmap.yaml is empty
The document manifests//docker-mailserver/templates/configmap.yaml contains a valid ConfigMap
The document manifests//docker-mailserver/templates/secret.yaml is empty
The document manifests//docker-mailserver/templates/secret.yaml contains a valid Secret
The document manifests//docker-mailserver/templates/pvc_rainloop.yaml is empty
The document manifests//docker-mailserver/templates/pvc_rainloop.yaml contains a valid PersistentVolumeClaim
#

Unit tests

While the above 2 tests confirm that syntactically, the generated config is good, they wouldn’t catch a logic error, like the mis-typing of a resource name leading to a field in values.yaml which doesn’t have the desired effect.

The answer here was a helm plugin, helm-unittest, a BDD-styled unit test framework for helm charts.

Once installed, the plugin integrates with the helm cli command to execute your defined tests against your charts. For example you might write a test which confirms that changing a value in values.yaml has the intended downstream effect in your Deployment definition.

Here’s an example of one of my tests:

- it: should configure imaps port 10993 for rainloop if enabled (and haproxy enabled)
  set:
    rainloop.enabled: true
    haproxy.enabled: true      
  asserts:
    - matchRegex:
        path: data.dovecot\.cf
        pattern: rainloop

Run the unit tests by installing the plugin, and then running helm unittest <my chart>. Here’s some sample output:

# helm unittest docker-mailserver

### Chart [ docker-mailserver ] docker-mailserver

 PASS  haproxy	docker-mailserver/tests/haproxy_test.yaml
 PASS  pvc_rainloop	docker-mailserver/tests/pvc_rainloop_test.yaml
 PASS  pvc creation	docker-mailserver/tests/pvc_test.yaml
 PASS  service_rainloop	docker-mailserver/tests/service_rainloop_test.yaml
 PASS  configmap	docker-mailserver/tests/configmap_test.yaml
 PASS  deployment_rainloop	docker-mailserver/tests/deployment_rainloop_test.yaml
 PASS  deployment tests	docker-mailserver/tests/deployment_test.yaml
 PASS  disable_spf_tests	docker-mailserver/tests/spf_test.yaml
 PASS  ingress_rainloop	docker-mailserver/tests/ingress_rainloop_test.yaml
 PASS  oobe	docker-mailserver/tests/oobe_test.yaml
 PASS  secret	docker-mailserver/tests/secret_test.yaml

Charts:      1 passed, 1 total
Test Suites: 11 passed, 11 total
Tests:       34 passed, 34 total
Snapshot:    11 passed, 11 total
Time:        9.801965981s

#

In addition to these pre-defined tests, helm-unittest supports creating a “snapshot” of the generated manifest. This snapshot gets stored with your tests, and allows you to compare the generated manifest with a known-good snapshot taken at a point in time. If your config legitimately changes, you update your snapshots by running helm unittest -u <chart name>

Integrating tests with Travis-CI

Once the 3 tests above can be run locally, you can run them all via your automated CI environment. I used Travis CI - here’s an example of my .travis.yml file which includes all 3 stages of testing above:

language: C
sudo: false
before_install:
  - wget https://kubernetes-helm.storage.googleapis.com/helm-v2.13.1-linux-amd64.tar.gz
  - tar xzvf helm-v2.13.1-linux-amd64.tar.gz
  - mv linux-amd64/helm helm
  - chmod u+x helm
  - wget https://github.com/garethr/kubeval/releases/download/0.7.3/kubeval-linux-amd64.tar.gz
  - tar xzvf kubeval-linux-amd64.tar.gz
  - chmod u+x kubeval
  - mv helm kubeval /home/travis/bin/
  - helm init -c
env:
  - KUBERNETES_VERSION="1.12.0"
jobs:
  include:
    - stage: lint chart syntax
      script:
        - helm lint helm-chart/docker-mailserver
    - stage: kubeval generated manifests
      script:   
        - mkdir manifests
        - helm template helm-chart/docker-mailserver --output-dir manifests
        - find manifests/ -name '*.yaml' | grep -v crd | xargs kubeval -v $KUBERNETES_VERSION
    - stage: execute unit tests
      script:  
        - mkdir -p helm-chart/docker-mailserver/config/opendkim/keys/example.com
        - cp helm-chart/docker-mailserver/demo-mode-dkim-key-for-example.com.key helm-chart/docker-mailserver/config/opendkim/keys/example.com/mail.private
        - echo "sample data for unit test" > helm-chart/docker-mailserver/config/opendkim/ignore.txt
        - travis_retry helm plugin install https://github.com/lrills/helm-unittest
        - helm unittest helm-chart/docker-mailserver

As an aside, I discovered the travis_retry wrapper during this process. Sometimes a build fails for network timeout reasons, and travis_retry will re-attempt (rather than fail) any commands up to 3 times before failing.

Got any ideas for improvements? Hit me up below, or jump over to the Discord server!