I haven’t done an upgrade for my External Secrets Operator(ESO) for a while in my Kubernetes clusters. The version stayed at 0.5.x for 4 years already 😛 Lucky for me, AI/Claude has become hugely helpful now. So I asked Claude to summarise the upgrade path I should take – I guess a big quick hop to latest won’t work simply.

Although Claude suggested the following steps: 0.5.8 → 0.9.x → 0.12.1 → 0.16.1 (do the storedVersions cleanup) → convert manifests to v1 → 0.17.x → 0.20.4 → 1.x → 2.9, I think it’s safe to skip a few still.

The first step I took is to upgrade to v0.16 because that’s where the API version external-secrets.io/v1 is promoted and v1beta1 is still supported. The official release can be retrieved using helm, eg.

helm template external-secrets external-secrets/external-secrets --version 0.16.1 -n external-secrets > upstream_manifests/external-secret.yaml

The generated yaml file can then be used with any GitOps workflow, or it can be done directly with helm too but that’s not what I prefer.

Once ESO v0.16 is running, I double checked if there’s still alpha or beta versions of external secrets in the clusters, using this command:

k get externalsecret -A -o jsonpath='{range .items[*]}{.apiVersion}{"\n"}{end}' |rg -i '(alpha|beta)'

A bit testing, eg. deleting a few external secrets and re-creating them, is necessary too since I skipped recommended versions.

Then I repeated the above steps for v1.1.1 then v2.9.0 which was the latest at the moment. After the upgrade, I noticed that a few things have changed though these versions:

1. It becomes more strict when processing JSON from upstream secrets, eg. I had an extra comma in my Google Secret Manager(GSM) secret which is invalid for JSON but it did work with older version of ESO. Obviously it’s a good thing to have valid JSON in upstream in the first place.
2. Labels and annotations for an external secret are used to be copied over to the managed Kubernetes secret, not anymore. If some label is needed for managed k8s secret, it needs to be done via template, eg.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-secret
  labels:
    app: podinfo          # stays on the ExternalSecret only
spec:
  target:
    name: my-secret
    template:
      mergePolicy: Merge   # keep existing labels on the Secret rather than replacing
      metadata:
        labels:
          app: podinfo     # this is what lands on the Secret
        annotations:
          reloader.stakater.com/match: "true"

That’s all the hassle, not bad for 4 years of changes 🙂