In a Kubernetes cluster, A Kubernetes Secret is a resource type to hold sensitive data for the apps to use, such as an API key or database password.

Secrets are namespaced so if I have RBAC access to a namespace, I can pretty much see all the secrets there, as the secrets are only base64 encoded, not encrypted by any sort of keys. This is Ok if I own the whole Kubernetes cluster or in a multi-tenant cluster with carefully designed RBAC rules.

Sealed Secrets

As an engineer, I still need to put the secret in some place safe, and encrypted. There comes SealedSecrets. In a nut shell, SealedSecrets enabled us to check-in encrypted secrets into github.com or other code repositories – even the public ones. Here’s an old page of notes I wrote on running SealedSecrets on a Raspberry Pi.

The pros of SealedSecrets:

• Works flawlessly with gitops, such as ArgoCD
• A SealedSecret will be decrypted into a normal Kubernetes Secret, which is a no brainer and works well with existing pods

And cons of course(debatable):

• The data need to be encrypted first with kubeseal command line tool and there might be trails of sensitive information left in command history, ie. k create secret generic test --from-literal=password=1234 ... |kubeseal -o yaml
• Also it becomes a toil if the secrets are rotated routinely
• If the private key needs to be safeguarded and installed to multiple clusters if those clusters will have the same set of SealedSecrets

External Secrets

ExternalSecrets, like SealedSecrets, is an extension to Kubernetes with the capabilities to import secrets from secret manage services such as AWS Secret Manager, Google Secret Manager, Vault and so on… And here are some pros and cons based on my quick experiment:

Pros:

• Support multiple sources of secrets
• Map external secrets into native Kubernetes Secrets
• No longer need to worry about secret encryption as it’s handled by external providers
• Easy to encrypt a secret, ie. copy and paste, with a GUI of AWS or GCP, etc.
• Secret rotation supported
• Support IAM integration such as Workload Identity in GCP which suggests that a Secret Store’s permission can be locked down to a single secrets, a group of secrets or all secrets in a project.

Cons:

• The need to install and configure ExternalSecrets in an existing Kubernetes cluster and take responsibility to keep it running
• If used outside of a provider’s cloud, ie. IAM not available, credentials for the service accounts need to be installed first

For installation, it’s super easy with a bit of Helm and ArgoCD. Here’s the code for my experiment in github. And here’s real ExternalSecret I now use for my cert-manager. You can see an ExternalSecret can be safely checked-in and pushed to a public repository as the only thing exposed is the reference to a secret name in an external secret manager, which is GSM in my case.

Also an official guide to import secrets from Google Secret Manager.

🙂