In preparing an author bio (harder than it sounds!) for a PHPList book I've been authoring, I dug up this old copy of a JAJC manual (a crusty ol' win32 jabber client)  I authored in 2003, using Docbook XML.

It's probably no longer valid, but here it is anyway, for posterity :)

Sadly, the docbook XML source is lost in time - it would have been fun to compare it to the sexiness that is markdown and leanpub today!

I work on Kubernetes cluster which are a whole lot more locked-down than typical installation instructions / application defaults would suggest. In one such cluster, we use PodSecurityPolicies to apply a minimal set of privileges to each pod, and make exceptions on a case-by-case basis.

On the same cluster, we use the Istio service mesh to secure traffic between our pods using mutualTLS. We take advantage of Istio's CNI plugin to allow the Istio sidecar to inject the "traffic interception" rules when pods start up, without requiring privileged access for every pod with a sidecar.

The CNI plugin creates a daemonset (a pod per node), which requires privileged access to inject the interception rules. Our default, restrictive PSP policy prevents these istio-cni-node pods from ever starting though, as illustrated below:

  Type     Reason        Age                    From                  Message
  ----     ------        ----                   ----                  -------
  Warning  FailedCreate  114s (x17 over 7m22s)  daemonset-controller  Error creating: pods "istio-cni-node-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0].hostPath.pathPrefix: Invalid value: "/opt/cni/bin": is not allowed to be used spec.volumes[1].hostPath.pathPrefix: Invalid value: "/etc/cni/net.d": is not allowed to be used]

The error above is pointing out that PSPs (quite rightly) prevented an arbitrary pod from mounting critical host directories, and having its way with them.

In this case, access to /opt/cni/bin and /etc/cni/net.d is a requirement for using Istio CNI (and the alternative of allowing every pod privileged access is much worse!), so we deploy a PSP, ClusterRole, and ClusterRoleBinding as illustrated below (you can grab a copy here):

here's a handy one-liner to pin a running pod to the node it's currently on:

kubectl patch deployment -n $NAMESPACE $DEPLOYMENT -p '{"spec": {"template": {"spec": {"nodeSelector": {"kubernetes.io/hostname": "'$(kubectl get pods -n $NAMESPACE -o jsonpath='{ ..nodeName }')'"}}}}}' || (echo Failed to identify current node of $DEPLOYMENT pod; exit 1)

The long version

I've been supporting with the Portainer team with a helm chart for their new v2, Kubernetes-supporting version. Recently the boss told me:

"Sometimes, when using one of these small/development, multi-node Kubernetes clusters like k3s or microk8s, Kubernetes will schedule the pod to a particular node, but when the pod moves to a different node, the data is lost. Find a way to ensure that the pod always remains on the same node"!

"Nonsense", I replied. "The Kubernetes storage provisioner will be smart enough to ensure that an allocated PV doesn't just move to a different node". And to prove how smart I was, I illustrated by creating a multi-node KinD cluster:

❯ cat kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

❯ kind create cluster --config kind.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.1) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

I created the namespace, added the helm repo, and deployed the chart:

❯ kubectl create namespace portainer
namespace/portainer created
❯ helm repo add portainer https://portainer.github.io/k8s/
❯ helm repo update
❯ helm upgrade --install -n portainer portainer portainer/portainer
Release "portainer" does not exist. Installing it now.
NAME: portainer
LAST DEPLOYED: Wed Dec  9 21:08:09 2020
NAMESPACE: portainer
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace portainer -o jsonpath="{.spec.ports[0].nodePort}" services portainer)
  export NODE_IP=$(kubectl get nodes --namespace portainer -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

I examined the PV created by the deployment and saw, as expected, a nodeSelector:

> kubectl get pv -o yaml
<snip>
nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kind-worker

"Boom!", I said. "There's no problem, because Kubernetes won't let the pod run on a different node, due to the nodeSelector".

Not so fast!

"Try microk8s", the boss said, "it happens all the time..."

So I did. Grumbling about how much harder it is to setup a multi-node microk8s environment, I used Multipass to create 2 Ubuntu 20.04 VMs, and then followed the instructions re setting up a microk8s cluster.

Sure enough, as it turns out, when I examined the microk8s PV, there was no nodeSelector. Microk8s, it turns out, uses a simple hostPath-type provisioner!

Where's my data?

So this presents a problem for any application deployed on a multi-node microk8s cluster, as well as any other cluster using a hostPath-based storage provisioner. We came up with what I think is an elegant solution though..

This command will return the current node of a pod (provided that pod has been scheduled):

kubectl get pods <podname> -o jsonpath='{ ..nodeName }'

And this command will patch a deployment, adding a nodeSelector:

kubectl patch deployments <deploymentname> -p '{"spec": {"template": {"spec": {"nodeSelector": {"kubernetes.io/hostname": "<nodename>"}}}}}'

Combined, we get this neat little command, a variation which is now featured on the Portainer install docs:

kubectl patch deployment -n $NAMESPACE $DEPLOYMENT -p '{"spec": {"template": {"spec": {"nodeSelector": {"kubernetes.io/hostname": "'$(kubectl get pods -n $NAMESPACE -o jsonpath='{ ..nodeName }')'"}}}}}' || (echo Failed to identify current node of $DEPLOYMENT pod; exit 1)

It should be noted that pinning a pod to a node obviously reduces resiliency in the event that a node fails, and something like this shouldn't be attempted seriously in production. If you're using microk8s though, you're probably not in serious production, so go wild!

BTW, this is what I do, all day, every day. I enjoy it, and I'm good at it. If this sort of stuff is what you need, I'd be interested to work with you.

I've been spending a lot more time with my iPad Pro, since the screen of my Macbook Pro cracked a few days into New Zealand's COVID-19 lockdown.

The 2020 iPad Pro itself was a long-awaited purchase, and it replaced my 2017 9.7" iPad Pro (which I didn't use much, since the screen was too small for comfort)

Enticed by posts like the one below (❤️ ya' sweet-setup!), I pulled the trigger on ordering the Magic Keyboard to replace the Smart Folio I'd originally bought (its up for sale if you're interested)

Magic Keyboard: Turning the iPad Into Something New – The Sweet Setup

Our accounting office is right next to a Telus store full of Android and Windows fanatics. I haven’t been able to get any person in the store to even consider an iPhone or Mac for themselves, let alone convince them the iPad is a great business device. The Magic Keyboard is the first accessory that …

Josh GinterThe Sweet Setup

When the keyboard finally arrived, it didn't disappoint. The keys are luxurious and "tappy", unlike the cloth-covered smart folio case, and the backlight is just what I needed for late-night geeking. The trackpad is a gamechanger for serious work away from my desk.

However..

It turns out that the default keyboard layout offered on the Apple Store in New Zealand is UK English, and not US English, as evidenced below:

For some reason, Magic Keyboard for the 11-inch gets to default to US English.

Upset that my new keyboard (which costs almost as much as a new entry-level iPad!) had smaller keys than its US counterpart, not to mention extra keys I didn't need, I contacted Apple to arrange a return, and placed my order for the US English version instead, resigned to wait the 3-4 weeks for delivery. Apple agreed that I could continue to use the keyboard, provided I was able to return it undamaged on receipt of my replacement.

What changed my mind

So what changed my mind? I've been trialling using the "Blink Shell" app for SSH shell on the iPad. My first choice, Termius, annoyingly doesn't permit text-selection using iPadOS's cursor. I'm less than pleased that they've not responded at all to my request, either:

Blink is certainly less polished than Termius, and I still haven't mastered SSH agent-forwarding, but it does let me click and drag with my cursor to select large blocks of text.

The killer feature of Blink here is that it allows you to create any custom keymappings. So I discovered that I can remap the otherwise useless symbol at the top left of the British keyboard, to the ESC key!

Now not only do I have an escape key which I can use in all sorts of terminal sequences, but I haven't lost the use of any of the other keys (including the ~ key, which would have been my go-to escape key on the US English version).

So, I just pushed the "cancel order" button on my pending US English Magic Keyboard, and I'm now resigned to these smaller English buttons, in return for having the next-best thing to a "physical" escape key!

I’m using Cybozu’s TopoLVM to provide local LVM-based storage to a bare-metal Kubernetes cluster, in an intelligent fashion.

I’m also using Jetstack’s cert-manager with the --namespace argument, to watch for certificate resources in a particular namespace only, so I wasn’t able to use cert-manager with TopoLVM, which is normally a pre-requisite.

The deployment docs tell me that I can avoid cert-manager if I use a self-signed certificate for the TopoLVM mutatingwebhook, which I thought wouldn’t be too difficult. I ran the following to generate the necessary cert, key, and cacert (valid for 100 years):

openssl genrsa -out rootCA.key 4096

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 35600 -out rootCA.crt

openssl genrsa -out controller.topolvm-system.svc.key 2048 

openssl req -new -sha256 -days 36500  -key controller.topolvm-system.svc.key -subj '/CN=controller.topolvm-system.svc' -out controller.topolvm-system.svc.csr

openssl x509 -req -in controller.topolvm-system.svc.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out controller.topolvm-system.svc.crt -days 36500 -sha256

The documentation said to add the caBundle field to the mutatingwebhook YAML in PEM format, so I initially added the entire rootCA.crt, including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----. This failed due to bad base64 encoding, so I removed the BEGIN and END lines, and the PEM data from the certificate was accepted as valid base64.

However, the webhook wasn’t trusted, and any pods I deployed failed with messages about certificate signed by unknown authority.

Turns out what was required was to base64-encode the PEM file, and paste the resulting base64-encoded string into caBundle. I.e, I set caBundle to the output of cat rootCA.crt | base64.

I found some kindred fellow-sufferers, whose confusion and eventual frustration echo my own!

I’m working on a project which requires a CockroachDB instance in multiple namespaces (prod/uat/dev), in an Isio-enabled Kubernetes cluster.

There are currently some pretty significant drawbacks to using Istio with “headless TCP” services, one of which being you can only have a single instance of a service with a specific TCP port, within the entire service mesh. So, no multiple cockroachDB instances on port 26257.

The problem hinges on the fact that for TCP-based services, Istio can only intercept traffic to a given TCP-based service by recognizing its port number.

A useful workaround to this issue, however, is Istio’s namespace isolation. Namespace isolation allows you to configure Istio to only configure the envoy sidecar proxies to access a subset of services on the mesh, within the scope of a namespace.

How would you use Istio namespace isolation?

My project is an easy example. Say you have 3 namespaces (dev/uat/prod), each of which should talk to their own CockroachDB instance (in their namespace), but be blissfully unaware of any other CockroachDB instances in other namespaces.

What you want is a “Sidecar” custom resource, which limits the scope of the service mesh config deployed to your sidecars. For example, the following CR restricts istio-proxy sidecars to being “aware” of only other services in the same namespace, or in the istio-system namespace:

kind: Sidecar
metadata:
  name: default
  namespace: istio-system
spec:
  egress:
  - hosts:
    - "./*"
    - "istio-system/*"

If this Sidecar is applied within the “cluster mesh” namespace (istio-system by default), and it doesn’t explictly match any source workloads, then the default will apply to all namespaces. This is a simple way to create a default namespace isolation policy.

Or so I thought…

Turns out there’s a wrinkle, that’s outlined in the docs:

NOTE 2: A Sidecar configuration in the MeshConfig root namespace will be applied by default to all namespaces without a Sidecar configuration. This global default Sidecar configuration should not have any workloadSelector.

The subtle implication here is that if you:

a) Setup a default Sidecar resource in the istio-system namespace, and b) Create a Sidecar resource in your namespace with a workloadSelector which doesn’t include all pods (for example, permitting a pod named “logmaster” to connect to pod “logcatcher” in namespace “logmonkey”)

Then any pods not matched by the aforementioned workloadSelector will have NO Sidecar resource applied, and thus no namespace isolation.

To re-iterate:

If you want namespace isolation, either specify a default Sidecar resource in your MeshConfig root namespace (default istio-system) and don’t create any more Sidecar resources in your target namespace, OR create a default Sidecar resource in each namespace, followed by more workload-specific Sidecar resources.

In my case, since I legitimately need some pods in each prod/uat/dev namespace to be able to communicate with select services in different namespaces, I’ve created a default Sidecar resource in each namespace, which restricts all pods to commuicating only within their own namespace:

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: istio-config
  namespace: prod
spec:
  egress:
  - hosts:
    - "./*"

Having established this per-namespace default, I can now apply more lenient Sidecar resources to only a subset of my pods, as in the following example:

apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: permit-logs-to-logmonkey
spec:
  egress:
  - hosts:
    - ./*
    - logmonkey/logcatcher.logmonkey.svc.cluster.local
  workloadSelector:
    labels:
      app.kubernetes.io/name: batman-app

Now I can happily spin up multiple headless TCP services in separate namespaces, without having Istio arbitrarily forward the traffic to the first-available service listening on a particular port :)

Update (29 Apr 2020)

Per these threads, the isolation described above is only applied if you’ve also set global.outboundTrafficPolicy.mode to REGISTRY_ONLY, rather than the more common value of ALLOW_ANY!

Got any questions / suggestions? I’m an Istio-n00b too, hit me up!

When Kubernetes 1.16 was released, several beta APIs were deprecated in favor of their stable counterparts. This means that your deployments will fail, if you’re still using the old APIs (as hundreds of public helm charts are!)

As release manager Lachlan Evenson mentioned in his Kubernetes Podcast interview, the intention of the deprecation was to “forcefully nudge” the community towards using the stable APIs, already available for 7 versions:

LACHLAN EVENSON: Let me start by saying that this is the first release that we've had a
big API deprecation, so the proof is going to be in the pudding.

CRAIG BOX: Yes.

LACHLAN EVENSON: And we do have an API deprecation policy. So as you mentioned, Craig, 
the Apps v1 has been around since 1.9. If you go and read the API deprecation policy, 
you can see that we have a three-release announcement. So around the 1.12, 1.13 time 
frame, we actually went and announced this deprecation, and over the last few releases,
we've been reiterating that.

But really, what we want to do is get the whole community on those stable APIs because it
really starts to become a problem when we're supporting all these many now-deprecated APIs,
and people are building tooling around them and trying to build reliable tooling. So this
is the first test for us to move people, and I'm sure it will break a lot of tools that
depend on things. But I think in the long run, once we get onto those stable APIs, people 
can actually guarantee that their tools work, and it's going to become easier in the long 
run.

So we've put quite a bit of work in announcing this. There was a blog sent out about six 
months ago by Valerie Lancey in the Kubernetes community which said, hey, go use 'kubectl
convert', where you can actually say, I want to convert this resource from this API version 
to that API version, and it actually makes that really easy. But I think there'll be some 
problems in the ecosystem, but we need to do this going forward, pruning out the old APIs 
and making sure that people are on the stable ones.

So what are you to do if you want to deploy a public helm chart, which has not yet been updated for the deprecated APIs?

You could fork, update, and publish your own helm repo. It’s do-able, but that’s a lot of work!

Eventually annoyed at the process, I wrote a little script to “transmogrify” Kubernetes manifests, replacing the deprecated APIs with the stable ones.

How does it work?

You simply point the script at your manifest directory, and it if finds any of the deprecated APIs, it’ll replace them inline with the stable versions.

Here’s an example:

root@cn1:~# /tmp/transmogrify_for_k8s_1.16.sh /tmp/ansible.bbGF94temp/prometheus/templates/
Deprecated API found in [/tmp/ansible.bbGF94temp/prometheus/templates/kube-state-metrics-deployment.yaml].. Transmogrifying...
Deprecated API found in [/tmp/ansible.bbGF94temp/prometheus/templates/alertmanager-deployment.yaml].. Transmogrifying...
Deprecated API found in [/tmp/ansible.bbGF94temp/prometheus/templates/node-exporter-daemonset.yaml].. Transmogrifying...
Deprecated API found in [/tmp/ansible.bbGF94temp/prometheus/templates/server-deployment.yaml].. Transmogrifying...
root@cn1:~#

Running the script a second time will result in no output, since there are no longer any deprecated APIs:

root@cn1:~# /tmp/transmogrify_for_k8s_1.16.sh /tmp/ansible.bbGF94temp/prometheus/templates/
root@cn1:~#

Here be dragons ?

I’ve made a lot of assumptions here - for one thing, the script assumes that each Kubernetes element is in its own YAML file. Weird things might happen if you point the script at a .yaml file combining multiple elements.

I’ve just finished a design for a client which required restricting various web endpoints to NZ IP addresses only.

I hadn’t used Azure Front Door prior to this project, because frankly it seemed like overkill for a small, NZ-based platform. I figured, “meh, I’ll just whitelist the IP ranges using an NSG, and be done with it. How hard can it be?”

Well…

It turns out little ol’ NZ has 7,295,124 IP addresses. (That’s just under double our population!) To whitelist these IPs, I’d need 221 individual IP entries! (point of geekiness - the first range assigned to NZ was 31 years ago, on 10/08/88, to Massey University)

So, the IP whitelist option was out.

But Azure Front Door was a viable approach. It’s got an attractive, minimal configuration (unlike the beast that is Application Gateway v1), and it supports fancy WAF rules, as well as custom rules, for which geo-location of the remote source is a configurable filter:

So we setup an Azure Front Door, with a dirty ol’ application gateway as its backend. Then in the NSG protecting the application gateway (applied at the subnet level), we whitelisted the IP ranges which Front Door traffic would be expected to ingress from, and locked all you unwashed barbarians outside the gates.

This all went mostly according to plan, but here are some gotchas to save you time:

Gotchas to avoid

Avoid using Frontdoor-managed certificates

Front Door will provide you with a free SSL certificate for your frontends. But I’ve had poor experience making it work. Mostly, choosing this option causes the “Updating” process to take 2+ hours.

Instead, the way to go is to put your certificates into KeyVault (you should be doing this already), and then to follow the following, super-user-unfriendly process:

1. Launch a cloud shell from the Azure portal, select PowerShell (of course, you weren’t using it by default, were you?), and paste in the following incantation: New-AzureRmADServicePrincipal -ApplicationId "ad0e1c7e-6d38-4ba4-9efd-0bc77ba9f037" (these are the instructions directly from Azure! They couldn’t make this into a button?)

1. Now that the service principal has been added to your AADC, create a new access policy, and grant it the secret.get permission in Key Vault.

Avoid hard-coding the backend host header

When you setup your backend hosts, you’re prompted to choose a backend host header. If you set this value, then the “Host” field on every forwarded request will be rewritten to what you chose. If you’ve routing more than one domain name to the backend, then this static value will break ability of the backend host to determine what content to send, based on the request header.

Rather, leave the backend host header blank, so that Front Door will just pass through the Host header from the original request:

Avoid omitting the host header with curl when you test

I baffled myself for a while after implementing the above, because I was running curl <hostname of frontdoor-protected-website>, and receiving a 404 error from my backend in response. Turns out, you have to submit a host header of some sort, so curl -H 'Host: myawesomesite.com' <hostname of frontdoor-protected-website> did the trick.

Super hard, it turns out. But I hope this post will make it easier for you.

In order to produce some documentation for a client on setting up DKIM under Office365, I undertook to migrate a test domain of mine to Office365, and set it up. This way, I’d produce better documentation, having had personal, hands-on experience (and screenshots). How hard could it be?

Friends don’t let friends use non-working email addresses

First, I made a noob mistake, and bought my Office365 subscription with the email address I intended to use with it (i.e., a currently non-working email address). After my purchase, when I attempted to sign in, I received a mysterious error and was advised to “try again later”. Of course, not having a working email address, I wasn’t able to do an “account recovery” or reset my password in any way.

Fortunately, Safari had saved my (randomly generated) password to iCloud Keychain, and I was able to recover it and login the next day.

CNAME schenanigans

Second, to use DKIM, you need some DNS records added to your domain. Unlike SPF, which is relatively easy to setup on Office365, DKIM requires some mental gymnastics to identify what records to add.

Here’s the CNAMES I had to add to protect elpenguino.net:

selector1._domainkey --> selector1.selector1-elpenguino-net._domainkey.elpenguinonet.onmicrosoft.com
selector1._domainkey --> selector2-elpenguino-net._domainkey.elpenguinonet.onmicrosoft.com

Part of the CNAME destination is my initial domain, and the other is the domainGUID. Neither match my actual domain.

Powershell, much?

So after I figured out the magic DNS records needed, I thought I’d be able to turn on DKIM signing, and get cracking. No. Turns out that for reasons unknown (but apparently rather common), to enable DKIM signing for my domain, I needed to break out some PowerShell.

Homebrew saved me, and with a quick brew cask install powershell, I had a PowerShell CLI on my Macbook.

Assuming you’re in a similar situation, you’ll want to run the following commands via PowerShell:

Prepare to login: $UserCredential = Get-Credential

Actually login: $Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection

Do something else: Import-PSSession $Session -DisableNameChecking

Enable DKIM signing for yourdomain.com:; New-DkimSigningConfig -DomainName "yourdomain.com" -Enabled $true

Here’s my error-filled beginner’s attempt to drive PowerShell:

So, finally I can navigate to my Office365 Exchange Admin settings, and enable DKIM signing.

Best of luck to you, fair adventurer!

I was recently invited by the Portainer team to speak at the Docker Auckland meetup, about the new RBAC extension for Portainer.

RBAC is a standard feature of the super-expensive, super-enterprisey Docker Enterprise Edition, but if you want more than admin-only access to your Docker CE cluster, you’re (until now) out of luck.

However, if you’re using Portainer to administer your Docker enviroment (and if not, why not?), you can now get all that Enterprisey goodness, for a whopping $US9.95 per year (per portainer instance, and each portainer instance can manage multiple swarms).

Since I’ve recently been introducting my kids to Star Wars, I felt it’d be useful to explain to this room-full-of-geeks how RBAC works, in a familiar context, and so I demonstrated RBAC using Star Wars characters (please don’t sue me, Disney!)

Contents

• Organizational structure
• Grant teams access to their own endpoints
• Examine effective permissions
• User experience
• Endpoint Groups
• Available Roles
• What have we learned?
• Tell me more!

Organizational structure

Let’s start with an organizational structure. Assume we have the following teams and endpoints.. (Portainer’s term for a docker / swarm instance)

• Jedi Council, whose endpoint is named “Tython”
• Galactic Republic, whose swarm cluster is named “Coruscant”
• Galactic Empire, whose swarm cluster is named “Death Star”
• Rebel Alliance, whose swarm cluster is named “Yavin-4”
• Nerf Herders, whose swarm cluster is named “Tatooine”

Grant teams access to their own endpoints

In general, each team should have administrative access to their own endpoint, so we grant each team the “Endpoint Administrator” role to their endpoint. So far, so good.

Examine effective permissions

You’ll have noticed that some team members overlap. For example, Han Solo is a member of both Nerf Herders and The Rebel Alliance. Let’s check his effective permissions to see what access he has:

In this case, membership of the 2 teams has granted Han Solo the team’s level of access to each endpoint. Say the Leia didn’t quite trust Han-the-scoundrel on the production endpoint, and wanted to restrict his role to read-only activities. We’d achieve this by granting a user-specific role to han.solo in the Yavin-4 endpoint. (User-specific roles override team roles)

User experience

How does this look, to Han? Let’s take a look.. we’ll login as han.solo, first attempt to administer the Tatooine endpoint (for which Han is an Endpoint Administrator), and then attempt to do the same to the Yavin-4 endpoint.

Endpoint Groups

We’ve seen how to restrict access to endpoints per team and per user. It’s also possible to group endpoints together, and to apply permissions at an endpoint-group level:

Once the endpoints are grouped, apply a role to the endpoint group:

In this example, I gave the user yoda the Endpoint Administrator role on the heroes endpoint group, which reflects on his effective permissions:

Available Roles

Portainer made a calculated decision to not allow the creation of custom roles. You get these 4 roles, and that’s it:

Endpoint Administrator: The Endpoint Administrator has complete control over the resources deployed within a given endpoint, but is not able to make any changes to the infrastructure that underpins an endpoint (ie no host management), nor able to make any changes to Portainer internal settings

Helpdesk: The Helpdesk role has read-only access over the resources deployed within a given endpoint but is not able to make any changes to any resource, nor open a console to a container, or make changes to a container’s volumes

Standard User: The Standard User role has complete control over the resources that user deploys, or if the user is a member of a team, complete control over resources that users of that team deploy

Read-Only User: A Read-Only User has read only access over the resources they are entitled to see (resources created by members of their team, and public resources)

What have we learned?

So, what have we learned, young padawan? We’ve learned that to administer one or more clusters with effective user management, we can Portainer plus the RBAC extension, to provide a “single pane of glass” to our Docker Swarm enviroments. And that Han Solo can’t be trusted with a production endpoint!

Tell me more!

Here are some resources:

• My blog posts tagged with portainer
• My geeky recipe re running Portainer on Docker Swarm
• Portainer’s blog
• Portainer’s support forums
• Portainer’s Slack server
• Portainer’s github repo

In The Design of Engineering Culture - DZone DevOps, a great read on “designing” your engineering culture, these two paragraphs jumped out at me:

In addition to motivation, change requires attention. A fair amount of people’s attention is devoted to doing their work when everything is business as usual. Any extraordinary circumstances (such as financial stress, workplace conflicts, or outside personal issues) will deplete people’s attention budgets further. Be aware of the different factors outlined in Paloma Medina’s BICEPS model of the core needs and motivations of people in the workplace. If people feel that their needs aren’t being met, that can take their attention and make them less amenable to change. For example, if an engineer feels like they aren’t being treated equally (the “E” in BICEPS) by their manager and that is causing significant stress, it will be harder for them to focus on something that feels less important, such as remembering to use a new post-mortem template.

Company- or department-level stressors — such as restructuring, staff turnover, or organizational financial concerns — will reduce the overall capacity for change. Especially over the long term, bad habits can form, adversarial thinking can become ingrained, and people can become so burnt out that they lose most of their attention or capacity to change at an individual level. This sort of situation makes meaningful change more necessary but also more difficult to enact. In exceptionally stressful circumstances, you may need to wait until big picture issues have been resolved before trying to make meaningful changes — for example, don’t try to force a new post-mortem facilitation training program on people right after (or during) a big round of layoffs.

I’ve worked in organizations where the company-level stressors (restructuring and staff turnover) impact staff so badly that despite wanting to change, the organisation is unable to muster the motivation to make changes stick.

Of course, half-hearted attempts at change, which fail due lack of support, simple feed the burn-out and make it harder for the next attempted change to succeed.​

The lesson here, I believe, is to be intentional about your company culture when things are going well, rather than reactively once you’re in the weeds.

..Because an organization that doesn’t actively create the culture it wants will end up with a culture anyway. It will be the disorganized total of its employees’ thoughts and experiences–based on everything from how they’re treated to where they sit. - Brent Gleeson

This post by James Thomas struck a chord with me. The post details the loss of the space shuttle Columbia (including 7 humans) in 2006.

The fact is we know that PowerPoint kills. Most often the only victims are our audience’s inspiration and interest. This, however, is the story of a PowerPoint slide that actually helped kill seven people.

• ​What happened
• How was the decision made?
• How was this critical data so badly mis-interpreted?
• The PowerPoint-killer: a simple document
• Real-world example of the triumph of words vs slides

​What happened

The shuttle suffered accidental damage on launch, but made it safely into space. The subsequent tragic loss resulted from the decision to attempt re-entry. The re-entry decision was significantly influenced by the presentation of technical data using PowerPoint, which lead to mis-interpretation of the risks…

At eighty-two seconds into the launch a piece of spray on foam insulation (SOFI) fell from one of the ramps that attached the shuttle to its external fuel tank. As the crew rose at 28,968 kilometres per hour the piece of foam collided with one of the tiles on the outer edge of the shuttle’s left wing… It was impossible to tell from Earth how much damage this foam, falling nine times faster than a fired bullet, would have caused when it collided with the wing… There were a number of options.  The astronauts could perform a spacewalk and visually inspect the hull.  NASA could launch another Space Shuttle to pick the crew up.  Or they could risk re-entry.
​

How was the decision made?

NASA officials sat down with Boeing Corporation engineers who took them through three reports; a total of 28 slides.. The salient point was whilst there was data showing that the tiles on the shuttle wing could tolerate being hit by the foam this was based on test conditions using foam more than 600 times smaller than that that had struck Columbia.

Despite the data showing that the “production fault” was 600x more significant than any previous test “in dev”, NASA felt confident that the data showed there was not enough damage to put the crew’s lives in danger, and went ahead with re-entry.

On re-entry, the wing overheated, the shuttle disintegrated and was lost, along with the lives of all 7 crew.

How was this critical data so badly mis-interpreted?

Edward Tufte, a Professor at Yale University and expert in communication reviewed the slideshow the Boeing engineers had given NASA, in particular the above slide. His findings were tragically profound.

​Read the entire post, or better yet, read Tufte’s full report. To be concise, the way that facts are presented via powerpoint (spacing, font sizing, intendation, titles, etc), and our “powerpoint fatigue” leads us to misleading or lazy conclusions. In this case, the (poor) formatting and presentation of the data led NASA to conclude that the Boeing engineers’s data showed minimal risk on re-entry.

The PowerPoint-killer: a simple document

Jeff Bezos enforces a “no Powerpoint, narrative memos only” rule for Amazon execs (he makes attendees read memos at the start of the meeting). ​

Basecamp has a similar strategy - new ideas must be written down as “pitches”, and given time to percolate.

I enjoy writing (obviously, you’re reading my blog!), and the act of writing and editing helps me to clarify my own thoughts / arguments far more effectively than creating a handful of slides with bullet points would do.

Real-world example of the triumph of words vs slides

I was recently asked to present technical recommendations to a client re how to reduce their AWS bill while improving their database I/O performance. The client specifically prioritised content over presentation. I spent a few days preparing / “percolating” two carefully-structured, technically sound papers detailing my recommendations. The recommendations were supported by calculations in a simple spreadsheet. I delivered the documents to the client a day ahead of our presentation / Q&A session (via video).

All of the Q&A attendees read my documents in preparation for the presentation - we spent about 30 seconds introducing the topic, and jumped directly into a technically deep discussion about the proposed solution, backed by the facts.

In summary, pre-presenting carefully prepared (and internally reviewed) “narrative” technical documents was universally praised by attendees as being “easy for non-technical audience to understand”, and delivering lasting value (since the documents now form the basis of a scope of work).

When cleaning up issues/PRs in Funky Penguin’s Geek’s Cookbook repository today, I noticed that PRs committed from the GitHub website included verified commits, but my own commits (from my latop) were not verified.

Determined to correct this, I worked through GitHub’s documentation on commit signature verification.

Here’s the basic path I followed:

Installed GPG:

brew install gpg

I generated myself a key, using defaults and GitHub’s recommendation of 4096 bits for my keysize:

[funkypenguin:~] 1 % gpg --full-generate-key
gpg (GnuPG) 2.2.15; Copyright (C) 2019 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory '/Users/funkypenguin/.gnupg' created
gpg: keybox '/Users/funkypenguin/.gnupg/pubring.kbx' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: David Young
Email address: davidy@funkypenguin.co.nz
Comment: https://www.funkypenguin.co.nz
You selected this USER-ID:
    "David Young (https://www.funkypenguin.co.nz) <davidy@funkypenguin.co.nz>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /Users/funkypenguin/.gnupg/trustdb.gpg: trustdb created
gpg: key 525EF417604A0541 marked as ultimately trusted
gpg: directory '/Users/funkypenguin/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/Users/funkypenguin/.gnupg/openpgp-revocs.d/3198CBB012FC221DD99BCA3F525EF417604A0541.rev'
public and secret key created and signed.

pub   rsa4096 2019-04-22 [SC]
      3198CBB012FC221DD99BCA3F525EF417604A0541
uid                      David Young (https://www.funkypenguin.co.nz) <davidy@funkypenguin.co.nz>
sub   rsa4096 2019-04-22 [E]

[funkypenguin:~] %

I listed my keys to find my key ID (in the example below, the key ID is 525EF417604A0541)

[funkypenguin:~] 130 % gpg --list-secret-keys --keyid-format LONG
gpg: WARNING: server 'gpg-agent' is older than us (2.2.10 < 2.2.15)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
/Users/funkypenguin/.gnupg/pubring.kbx
--------------------------------------
sec   rsa4096/525EF417604A0541 2019-04-22 [SC]
      3198CBB012FC221DD99BCA3F525EF417604A0541
uid                 [ultimate] David Young (https://www.funkypenguin.co.nz) <davidy@funkypenguin.co.nz>
ssb   rsa4096/1CC86B12BD8AEEE6 2019-04-22 [E]

[funkypenguin:~] %

I exported the key, ASCII-armoured:

[funkypenguin:~] % gpg --armour --export 525EF417604A0541
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFy+OLsBEADNKlxkp3tLddEK02BvHeqfoo8XxAgB87AM5hpvLkAbui8fnEgb
XhJ8v6SnhMNHPthsCq3LRVRggtPkIT0LemB2nibJgCqJhgzC5NE+Uu7WvDt5X860
GWZL7oqFnjq23VBAPNQRiDMiVsCwSqWSsyqCaJzL7UJZw8C88j05sEJEHx9anoBU
<snipped for brevity>
upLSmBs9tzGliP8+XYdPSSe8cQNEEv/sdrsj81VdBW9Zen2RSxEIqTLIHvbwljD0
jKj6Pat3l3oQfi1Be5DORer5r8YiVbdeKBm01vMp9pBkE4/VDUZrMsnQ27uc30sL
2m7atbQoAq3tCNJgZ+jjWx+oFG3hslEnVWe9lkhDpeGryVzQHb5pDYvLQTuTXRV/
d97VRAI+A7gQQFBT88Erdio1fmaa6VoytPBJIJEZ/viBrQGn
=rQDl
-----END PGP PUBLIC KEY BLOCK-----
[funkypenguin:~] %

I followed GitHub’s instructions (Settings -> SSH/GPG keys) to add the public key to my account

I configured my local git (globally to use my GPG key, and enabled GPGP signing of commits by default:

git config --global user.signingkey 525EF417604A0541
git config --global commit.gpgsign true

Finally, since I don’t want to have to type in my passphrase every time I commit, I installed GPG Suite using brew:

brew cask install gpg-suite-no-mail

The first time I ran a git commit, I was prompted by “Pinentry Mac” for my GPG passphrase. I typed my passphrase, and chose to save it in KeyChain.

I pushed a commit, and boom - a verified commit popped out the other end!