A Kubernetes ClusterSecret

No, at this moment ClusterSecret, unlike ClusterRole, doesn’t officially exist in any version of Kubernetes yet. I’ve seen some discussion like this, so looks like it will be a while to have a ClusterSecret.

But why do I need a ClusterSecret in the first place? The reason is very simple: To be DRY. Imagine I have a few apps deployed into several different namespaces and they all need to pull from my private docker registry. This looks like:

├── namespace-1
│   ├── image-pull-secret
│   └── deployment-app-1
├── namespace-2
│   ├── image-pull-secret
│   └── deployment-app-2
...

It’s a tad straight forward that all the image-pull-secret secrets are the same but as there’s no ClusterSecret they have to be duplicated all over the place. And to make things nicer, if the private registry changes its token, all of these secrets need to be updated at once.

Of course I’m not the first one to be frustrated by this and there are tools built by the community already. ClusterSecret operator is one of them. But when I looked at kubernetes-reflector I immediately liked its simple approach: it can reflects 1 source secret or configmap to many mirror ones in all namespaces! Also it’s easy to integrate with existing SealedSecret operator with reflector.

Here’s how to install kubernetes-reflector quickly with all default settings(copied from its README). I chose to save this file and let my FluxCD to install it for me.

kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

Now I can create a image pull secret for my private docker registry in kube-system namespace and then the reflector will copy it to a few namespaces which match the regex for the namespace whitelist.

The command to create a image pull secret is

kubectl create secret docker-registry image-pull-secret -n kube-system --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

The full sealed secret command will be

kubectl create secret docker-registry image-pull-secret -n kube-system --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> | \
  kubeseal --controller-namespace=sealed-secrets --controller-name=sealed-secrets -o yaml > image-pull-secret.yaml

Then I’ll add a few magic annotation to let the reflector pick up the job

# this is image-pull-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: image-pull-secret
  namespace: kube-system
spec:
  encryptedData:
    .dockerconfigjson: AgA4E6mcpri...
  template:
    metadata:
      creationTimestamp: null
      name: image-pull-secret
      namespace: kube-system
      annotations:
        reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
        reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
        reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "wordpress-.*"
status: {}

So when I deploy this file, first the SealedSecret operator will decrypt this into a normal secret with those annotations(note. adding annotations won’t break the encryption, but changing name or namespace could). And then the reflector will create the image-pull-secret secrets in all namespaces which start with wordpress- prefix.

Mission accomplished 🙂

Using Sealed Secrets in a Raspberry Pi Kubernetes Cluster

Sealed Secrets is a bitnami Kubernetes operator aimed to one-way encrypt secrets into sealed secrets so that they can be safely checked-in into GitHub or other VCS. It’s rather easy to install and use Sealed Secrets in a Kubernetes cluster on AMD64 architecture, but not so on my Raspberry Pi cluster.

First, the container image for the sealed-secrets-controller wasn’t built for ARM architecture. I managed to build it in my Raspberry Pi 2 with following commands:

git clone https://github.com/bitnami-labs/sealed-secrets.git
cd sealed-secrets
# golang build tools are needed here
make controller.image
# you can tag it to your docker registry instead of mine
docker tag quay.io/bitnami/sealed-secrets-controller:latest raynix/sealed-secrets-controller-arm:latest
docker push raynix/sealed-secrets-controller-arm

The next step is to use kustomize to override the default sealed-secrets deployment schema to use my newly built container image that runs on ARM

# kustomization.yaml
# controller.yaml is from https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.7/controller.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: sealed-secrets
images:
  - name: quay.io/bitnami/sealed-secrets-controller
    newName: raynix/sealed-secrets-controller-arm
    newTag: latest
patchesStrategicMerge:
  - patch.yaml

resources:
  - controller.yaml
  - ns.yaml
# ns.yaml
# I'd like to install the controller into its own namespace
apiVersion: v1
kind: Namespace
metadata:
  name: sealed-secrets
# patch.yaml
# apparently the controller running on Raspberry Pi 4 needs more time to initialize
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sealed-secrets-controller
spec:
  template:
    spec:
      containers:
        - name: sealed-secrets-controller
          readinessProbe:
            initialDelaySeconds: 100

Then the controller can be deployed with command kubectl apply -k .

The CLI installation is much easier on a Linux laptop. After kubeseal is installed. The public key used to encrypt secrets can be obtained from the controller deployed above. Since I installed the controller in it’s own namespace sealed-secrets instead of the default kube-system the command to encrypt secrets is a bit different:

kubectl create secret generic test-secret --from-literal=username=admin --from-literal=password=password --dry-run -o yaml | \
  kubeseal --controller-namespace=sealed-secrets -o yaml > sealed-secrets.yaml

Then the generated file sealed-secrets.yaml can be deploy with kubectl apply -f sealed-secrets.yaml and a secret called test-secret will be created. Now feel free to check-in sealed-secrets.yaml into a public GitHub repository!

🙂