Beware the hidden newlines in Kubernetes secrets

TL;DR

When creating secrets for Kubernetes, always use echo -n to create the files you intend to import as secrets using kubectl create secret.

The longer story

I’ve been working on a new series of Geek’s Cookbook recipes, utilising my new favorite container orchestrator, Kubernetes.

In building a recipe for a LetsEncrypt-certified MQTT broker, I thought I’d take the opportunity to learn to use Kubernetes Secrets, so that I could publish my .yaml configs verbatim, without worrying about exposing my CloudFlare API key.

Secrets allow you to safely store and pass sensitive information to your pods without storing these in your config files. Secrets can be passed to pods either as environment variables or as filesystem created on the filesystem.

Semi-distracted, I read (skimmed) though the docs, echo’d my secrets into individual files, and used kubectl to create a secret for me:

echo "myapikeyissosecret" > cloudflare-key.secret
kubectl create secret -n mqtt generic mqtt-credentials \
   --from-file=cloudflare-key.secret \
   --from-file=cloudflare-email.secret \
   --from-file=letsencrypt-email.secret

And then I found that certbot / cloudflare scripts would fail. I shelled into my pod (kubectl exec -n mqtt mqtt-8876bcf6d-g99bc  -it -- /bin/ash), and ran env to confirm the secrets were there, and yes, there they were!

(Queue the ominous music - I failed to take cognisance of the blank lines separating some of the environment variables)

Atfer much debugging, I finally found that the secrets had ended up as environment variables within the pod, with an trailling newline. Which was, of course, breaking my carefully scripted curl API calls.

Google found me some resources discussing the hidden newlines, and looking back on the docs, I see that I was instructed to use echo -n, but unaware of the implication of -n, I’d simply used echo.

So, as a note for future dumbasses following in my footsteps - always use echo -n when creating secrets to import with kubectl!