TLS Full Site Encryption with Istio and Let’s Encrypt

These are steps to easily install TLS certs to a Kubernetes cluster with Istio service mesh as ingress controller, provided by Let’s Encrypt‘s awesome certbot.

Installation of the certbot (on Ubuntu Linux 20.04LTS)

The certbot can be install via snap on Ubuntu Linux

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/local/bin/certbot
certbot --version
certbot 1.15.0

By default certbot needs to write to system directories which I thought unnecessary. I use this alias to run certbot as a normal user

mkdir ~/.certbot
alias certbot="certbot --config-dir ~/.certbot/ --work-dir ~/.certbot/ --logs-dir ~/.certbot"

Generate a new cert

Here’s an example to use certbot’s plugin to create certificate for domains hosted at CloudFlare. Here for more info on the plugin.

# install the plugin first
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

# save a cloudflare API token
echo "dns_cloudflare_api_token = xxxx" > ~/.cloudflare.ini

# generate the cert
# cert and key will be in ~/.certbot/live/
certbot certonly --dns-cloudflare -d -d '*' --dns-cloudflare-credentials ~/.cloudflare.ini
ls ~/.certbot/live/ -lht
total 4.0K
-rw-rw-r-- 1 ray ray 692 May 10 11:52 README
lrwxrwxrwx 1 ray ray  35 May 10 11:52 cert.pem -> ../../archive/
lrwxrwxrwx 1 ray ray  36 May 10 11:52 chain.pem -> ../../archive/
lrwxrwxrwx 1 ray ray  40 May 10 11:52 fullchain.pem -> ../../archive/
lrwxrwxrwx 1 ray ray  38 May 10 11:52 privkey.pem -> ../../archive/

Install the cert to an Istio gateway

The cert and the key will be put into a Kubernetes secret in istio-system namespace

# assuming kubectl is installed and configured
kubectl create secret -n istio-system tls wild-cert --key ~/.certbot/live/ --cert ~/.certbot/live/

Now the Istio gateway object needs to use this secret as TLS credential

cat <<EOF >gw.yaml
kind: Gateway
  name: wordpress-gateway
  namespace: wordpress
    # default istio ingress gateway
    istio: ingressgateway
  - hosts:
      name: https
      number: 443
      protocol: HTTPS
      credentialName: wild-cert
      mode: SIMPLE
  - hosts:
      name: http
      number: 80
      protocol: HTTP
      httpsRedirect: true

Then this can be locally tested with curl

curl -v --resolve "<TLS node port>:<node IP>" "<TLS node port>"


OpenSSL Commands to Verify TLS Certs in Kubernetes Secrets

Sometimes a TLS cert deployed into a Kubernetes cluster in a Secret doesn’t work as expected. Here are some handy commands to verify the certs. The sample commands work for Istio Ingressgateway, but should be adapted to other CNIs without huge efforts.

Commands to verify the cert served by your web-app

# Use openssl to retrieve cert and decrypt and print it out
# This can be used to verify that the correct cert is in use in an gateway
# use ctrl-c to end it
openssl s_client  -connect -showcerts -servername |openssl x509 -noout -text

# Print out dates of the cert
openssl s_client  -connect -showcerts -servername |openssl x509 -noout -dates

# Print out the subject/CN of the cert
openssl s_client  -connect -showcerts -servername |openssl x509 -noout -subject

# Print out the subjectAltName/SAN of the cert
openssl s_client  -connect -showcerts -servername |openssl x509 -noout -text |grep 'Subject Alternative Name' -A1

Commands to verify the cert installed in a secret

# This needs access to secrets so the cert secret can be downloaded and verified
kubectl get secret -n istio-system my-namespace-cert-secret -o yaml

# one-liner to print md5 hash of the X509 modulus from the cert
kubectl get secret -n istio-system my-namespace-cert-secret -o jsonpath={'.data.cert'} |base64 -d | openssl x509 -noout -modulus |openssl md5
# example output

# one-liner to print md5 hash of the RSA modulus from the key
# this output has to match the previous one.
kubectl get secret -n istio-system my-namespace-cert-secret -o jsonpath={'.data.key'} |base64 -d | openssl rsa -noout -modulus |openssl md5
# example output