Use Variables with Kustomize, Part 2

I was looking at the Kustomize variable trick I did a year ago and I think I’ve learned some new tricks worth noting down.

Variables are very handy most of the times, here’s a pattern to define a variable to be used in Kustomize templates and set its value via annotations.

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    # the domain name will be set by the variable
    - $(DOMAIN)
    tls:
      mode: SIMPLE
      credentialName: $(CERT)
    port:
      name: https
      number: 443
      protocol: HTTPS

# base/virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: wordpress-vs
spec:
  gateways:
    - wordpress-gateway
  hosts:
    # same domain here so no need to repeat the domain name, also the virtual service will always match the gateway
    - $(DOMAIN)
  http:
    - route:
      - destination:
          host: wordpress

# base/config.yaml, ensure variables are enabled for Istio resources
varReference:
  - path: spec/hosts
    kind: VirtualService
  - path: spec/servers/hosts
    kind: Gateway
  - path: spec/servers/tls/credentialName
    kind: Gateway

# at last the base/kustomize.yaml which defines 2 variables: DOMAIN and CERT
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configurations:
  - config.yaml
resources:
  - gateway.yaml
  - virtual-service.yaml
  # the content of deployment isn't given here, any valid deployment resource should do
  - deployment.yaml 
vars:
  - name: DOMAIN
    objref:
      apiVersion: apps/v1
      kind: Deployment
      name: wordpress
    fieldref:
      fieldpath: metadata.annotations.domain
  - name: CERT
    objref:
      apiVersion: apps/v1
      kind: Deployment
      name: wordpress
    fieldref:
      fieldpath: metadata.annotations.cert

# the above templates form a base Kustomize template and any Kustomize template extending the above base can have the variables set for real values
# site1/kustomize.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: wordpress-1
bases:
  - ../base

commonAnnotations:
  domain: site1.blog
  cert: site1-blog-cert

There you have it: When building the overlay template site1, variable DOMAIN will have the value of site1.blog; and CERT variable will be set to site1-blog-cert.

🙂

Automate SSL/TLS Certificates for Kubernetes and Istio with Cert Manager

It’s been a lot easier nowadays to turn on full site SSL/TLS encryption with an ACME issuer such as the popular non-profit Let’s Encrypt which I’ve started using it a few months ago.

However the free certificates from Let’s Encrypt are only valid for 90 days and I have been notified to renew them already. I could run the cert-bot again to semi-automatically renew the certificates in my Kubernetes cluster, but there’s Cert Manager which claims to be able to fully automate the whole process. I have to give it a try.

The Cert Manager essentially is a bundle of a Kubernetes Operator + CRDs(Custom Resource Definition), I followed the installation steps and installed Cert Manager easily with default settings. I chose to install with plain manifests because I tried the Helm charts first but it had some issues with my FluxCD pipeline. The plain manifests include the Cert Manager operator, CRDs and relevant Kubernetes resources such as Namespace and Service Account.

To use the same CloudFlare DNS challenge I used with cert-bot, I created a secret containing my CF API token:

$ k create secret generic cloudflare-api-token -n cert-manager --from-literal=api-key=eQ9Ls6...

Note: it will take a while for the Cert Manager to self-signed certificate for itself and get ready for requests.

Then I created a ClusterIssuer using CloudFlare DNS challenge to verify domain authority:

$ cat <<EOF |k apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: <my email>
    server: https://acme-v02.api.letsencrypt.org/directory
    # this is the new CA chain
    preferredChain: "ISRG Root X1"
    # a secret name to save the private key
    privateKeySecretRef:
      name: letsencrypt-issuer
    solvers:
    - dns01:
        cloudflare:
          email: <my cloudflare email>
          # the secret to provide CloudFlare API Token
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-key
EOF

Then I checked if the ClusterIssuer is up:

$ k get clusterissuer
NAME                 READY   AGE
letsencrypt-issuer   True    1m

Almost done! The Cert Manager is now ready to manage certificates. I then created a Certificate resource to request and manage a new certificate:

$ cat <<EOF |k apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wordpress-raynix
  # the cert will be in istio-system namespace to be used by istio
  namespace: istio-system
spec:
  # the secret to hold this cert
  secretName: wordpress-raynix-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - raynix.info
  isCA: false
  privateKey:
    algorithm: RSA
    size: 2048
  usages:
    - server auth
    - client auth
  # put all DNS names in the cert here
  dnsNames:
    - raynix.info
  # refer to the ClusterIssuer
  issuerRef:
    name: letsencrypt-issuer
    kind: ClusterIssuer
EOF

Then it usually takes a minute or longer to actually see the secret in the newly created secret. The DNS challenge process is now run by the Cert Manager pod instead of the cert-bot.

As a last step, the new cert can be used in an Istio Gateway:

# this is the existing gateway
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress-raynix
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      # use the new cert in the new secret!
      credentialName: wordpress-raynix-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

That’s it! Cert Manager claims that it can manage that cert and renew it accordingly, let’s see if that’s true 🙂

Grant a Service Account an IAM Role in AWS/GCP

How to grant a pod running in a Kubernetes cluster necessary permissions to access cloud resources such as S3 buckets? The most straight forward approach is to save some API key in the pod and use it to authenticate against cloud APIs. If the cluster is running inside the cloud, an IAM role can then be bound to a service account in the cluster, which is both convenient and safe.

I’ll compare the ways IAM role and service account bind in AWS/EKS and GCP/GKE.

AWS/EKS

The EKS is the managed Kubernetes service in AWS. To bind an EKS service account to an AWS IAM role:

  1. Create an IAM OIDC provider for your cluster
  2. Create an IAM role which can be assumed by the EKS service account
  3. Annotate the EKS service account to assume the IAM role

GCP/GKE

The GKE is the managed Kubernetes service in GCP. In GCP this is called Workload Identity(WLI), in a nut shell it binds a GKE service account to a GCP IAM service account, so it’s a bit different than the one above. The full instruction is here but in short:

  1. Enable WLI for the GKE cluster
  2. Create or update node-pool to enable WLI
  3. Create IAM service account and assign roles with necessary permissions
  4. Allow IAM service account to be impersonated by a GKE service account
  5. Annotate the GKE service account to impersonate the GCP service account

🙂

TLS Full Site Encryption with Istio and Let’s Encrypt

These are steps to easily install TLS certs to a Kubernetes cluster with Istio service mesh as ingress controller, provided by Let’s Encrypt‘s awesome certbot.

Installation of the certbot (on Ubuntu Linux 20.04LTS)

The certbot can be install via snap on Ubuntu Linux

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/local/bin/certbot
certbot --version
certbot 1.15.0

By default certbot needs to write to system directories which I thought unnecessary. I use this alias to run certbot as a normal user

mkdir ~/.certbot
alias certbot="certbot --config-dir ~/.certbot/ --work-dir ~/.certbot/ --logs-dir ~/.certbot"

Generate a new cert

Here’s an example to use certbot’s plugin to create certificate for domains hosted at CloudFlare. Here for more info on the plugin.

# install the plugin first
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

# save a cloudflare API token
echo "dns_cloudflare_api_token = xxxx" > ~/.cloudflare.ini

# generate the cert
# cert and key will be in ~/.certbot/live/raynix.info
certbot certonly --dns-cloudflare -d raynix.info -d '*.raynix.info' --dns-cloudflare-credentials ~/.cloudflare.ini
ls ~/.certbot/live/raynix.info/ -lht
total 4.0K
-rw-rw-r-- 1 ray ray 692 May 10 11:52 README
lrwxrwxrwx 1 ray ray  35 May 10 11:52 cert.pem -> ../../archive/raynix.info/cert1.pem
lrwxrwxrwx 1 ray ray  36 May 10 11:52 chain.pem -> ../../archive/raynix.info/chain1.pem
lrwxrwxrwx 1 ray ray  40 May 10 11:52 fullchain.pem -> ../../archive/raynix.info/fullchain1.pem
lrwxrwxrwx 1 ray ray  38 May 10 11:52 privkey.pem -> ../../archive/raynix.info/privkey1.pem

Install the cert to an Istio gateway

The cert and the key will be put into a Kubernetes secret in istio-system namespace

# assuming kubectl is installed and configured
kubectl create secret -n istio-system tls wild-cert --key ~/.certbot/live/raynix.info/privkey.pem --cert ~/.certbot/live/raynix.info/fullchain.pem

Now the Istio gateway object needs to use this secret as TLS credential

cat <<EOF >gw.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress
spec:
  selector:
    # default istio ingress gateway
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: wild-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

Then this can be locally tested with curl

curl -v -HHost:raynix.info --resolve "raynix.info:<TLS node port>:<node IP>" "https://raynix.info:<TLS node port>"

🙂