Automate SSL/TLS Certificates for Kubernetes and Istio with Cert Manager

It’s been a lot easier nowadays to turn on full site SSL/TLS encryption with an ACME issuer such as the popular non-profit Let’s Encrypt which I’ve started using it a few months ago.

However the free certificates from Let’s Encrypt are only valid for 90 days and I have been notified to renew them already. I could run the cert-bot again to semi-automatically renew the certificates in my Kubernetes cluster, but there’s Cert Manager which claims to be able to fully automate the whole process. I have to give it a try.

The Cert Manager essentially is a bundle of a Kubernetes Operator + CRDs(Custom Resource Definition), I followed the installation steps and installed Cert Manager easily with default settings. I chose to install with plain manifests because I tried the Helm charts first but it had some issues with my FluxCD pipeline. The plain manifests include the Cert Manager operator, CRDs and relevant Kubernetes resources such as Namespace and Service Account.

To use the same CloudFlare DNS challenge I used with cert-bot, I created a secret containing my CF API token:

$ k create secret generic cloudflare-api-token -n cert-manager --from-literal=api-key=eQ9Ls6...

Note: it will take a while for the Cert Manager to self-signed certificate for itself and get ready for requests.

Then I created a ClusterIssuer using CloudFlare DNS challenge to verify domain authority:

$ cat <<EOF |k apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: <my email>
    server: https://acme-v02.api.letsencrypt.org/directory
    # this is the new CA chain
    preferredChain: "ISRG Root X1"
    # a secret name to save the private key
    privateKeySecretRef:
      name: letsencrypt-issuer
    solvers:
    - dns01:
        cloudflare:
          email: <my cloudflare email>
          # the secret to provide CloudFlare API Token
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-key
EOF

Then I checked if the ClusterIssuer is up:

$ k get clusterissuer
NAME                 READY   AGE
letsencrypt-issuer   True    1m

Almost done! The Cert Manager is now ready to manage certificates. I then created a Certificate resource to request and manage a new certificate:

$ cat <<EOF |k apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wordpress-raynix
  # the cert will be in istio-system namespace to be used by istio
  namespace: istio-system
spec:
  # the secret to hold this cert
  secretName: wordpress-raynix-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - raynix.info
  isCA: false
  privateKey:
    algorithm: RSA
    size: 2048
  usages:
    - server auth
    - client auth
  # put all DNS names in the cert here
  dnsNames:
    - raynix.info
  # refer to the ClusterIssuer
  issuerRef:
    name: letsencrypt-issuer
    kind: ClusterIssuer
EOF

Then it usually takes a minute or longer to actually see the secret in the newly created secret. The DNS challenge process is now run by the Cert Manager pod instead of the cert-bot.

As a last step, the new cert can be used in an Istio Gateway:

# this is the existing gateway
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress-raynix
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      # use the new cert in the new secret!
      credentialName: wordpress-raynix-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

That’s it! Cert Manager claims that it can manage that cert and renew it accordingly, let’s see if that’s true 🙂

TLS Full Site Encryption with Istio and Let’s Encrypt

These are steps to easily install TLS certs to a Kubernetes cluster with Istio service mesh as ingress controller, provided by Let’s Encrypt‘s awesome certbot.

Installation of the certbot (on Ubuntu Linux 20.04LTS)

The certbot can be install via snap on Ubuntu Linux

sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/local/bin/certbot
certbot --version
certbot 1.15.0

By default certbot needs to write to system directories which I thought unnecessary. I use this alias to run certbot as a normal user

mkdir ~/.certbot
alias certbot="certbot --config-dir ~/.certbot/ --work-dir ~/.certbot/ --logs-dir ~/.certbot"

Generate a new cert

Here’s an example to use certbot’s plugin to create certificate for domains hosted at CloudFlare. Here for more info on the plugin.

# install the plugin first
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

# save a cloudflare API token
echo "dns_cloudflare_api_token = xxxx" > ~/.cloudflare.ini

# generate the cert
# cert and key will be in ~/.certbot/live/raynix.info
certbot certonly --dns-cloudflare -d raynix.info -d '*.raynix.info' --dns-cloudflare-credentials ~/.cloudflare.ini
ls ~/.certbot/live/raynix.info/ -lht
total 4.0K
-rw-rw-r-- 1 ray ray 692 May 10 11:52 README
lrwxrwxrwx 1 ray ray  35 May 10 11:52 cert.pem -> ../../archive/raynix.info/cert1.pem
lrwxrwxrwx 1 ray ray  36 May 10 11:52 chain.pem -> ../../archive/raynix.info/chain1.pem
lrwxrwxrwx 1 ray ray  40 May 10 11:52 fullchain.pem -> ../../archive/raynix.info/fullchain1.pem
lrwxrwxrwx 1 ray ray  38 May 10 11:52 privkey.pem -> ../../archive/raynix.info/privkey1.pem

Install the cert to an Istio gateway

The cert and the key will be put into a Kubernetes secret in istio-system namespace

# assuming kubectl is installed and configured
kubectl create secret -n istio-system tls wild-cert --key ~/.certbot/live/raynix.info/privkey.pem --cert ~/.certbot/live/raynix.info/fullchain.pem

Now the Istio gateway object needs to use this secret as TLS credential

cat <<EOF >gw.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: wordpress-gateway
  namespace: wordpress
spec:
  selector:
    # default istio ingress gateway
    istio: ingressgateway
  servers:
  - hosts:
    - raynix.info
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: wild-cert
      mode: SIMPLE
  - hosts:
    - raynix.info
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

Then this can be locally tested with curl

curl -v -HHost:raynix.info --resolve "raynix.info:<TLS node port>:<node IP>" "https://raynix.info:<TLS node port>"

🙂

Remove GPS Coordinates From Images Before Uploading to SNS

The recent events(I meant around NYD 2021) in US are quite shocking to me.

  • Trump and his supporters were banned from major SNS platforms
  • The one which didn’t ban them, Parler, got banned from the cloud platforms which Parler runs on, eg. AWS, iOS, Android
  • Parler’s data got scraped by a group of hackers
  • All images Trump supporters uploaded to Parler still have GPS coordinates in them. This could be the best set of evidence in the history

I probably should not post anything meaningful to an SNS but when I still do, here are something I can do to improve protection for privacy.

First the GPS coordinates in photos are added by smart phones and this can usually be turned off. Search for location tags or similar term in camera settings.

For existing images, GPS data can be deleted using following commands in Ubuntu Linux

# install exiftool
sudo apt install libimage-exiftool-perl
# remove GPS data from 1 photo
exiftool -gps:*= my-photo.jpg
# remove GPS data from all .jpg in current directory
find . -name \*.jpg -exec exiftool -gps:*= {} \;

🙂

利用 Ubuntu + VNC 做远程遥控

最近给 EPS-China 公司做好了 IT 基础设施, 但后期维护该怎么办呢? 毕竟我要离开北京了. 于是我想到了家里闲置的 Atom 330 的微型电脑, 俗称下载机. 我叫它 Xpatom, 因为要用迅雷下载而特意装了一套 Windows XP SP2. 将 Xpatom 安置到 EPS 的机柜后, 配置了端口转发 3389, 回来一看远程桌面, 嗯, 挺好的. 但隔天就变成这样了.

也难怪, 我的 Windows XP SP2 系统是盗版的, 也没怎么打补丁. 不过居然这么快就被嗅到了… 于是我将 Xpatom 重装了 Ubuntu 并改名为 Atomuntu  😀 本来 Ubuntu 开箱支持远程桌面的, 但内建的需要一个 Unity/Gnome session, 也就是说本地有用户登录了才能远程连接. 留下一个 session 没人管, 那我会失眠的吧…

搜索了一下, 恰好发现 Taskiller 的这篇 Ubuntu下配置VNC, 于是照着葫芦画瓢吧. 首先安装 VNC4Server:

$sudo apt-get install vnc4server

之后第一次启动 vnc4server 时会索要一个口令/password, 这个和用户账号无关, 是 VNC 连接时需要验证的. 遗憾的是最高只支持8位. 我以为所有基于口令验证的机制都是不保险的. 于是我在 Atomuntu 上又开启了 sshd, 这样我的远程遥控系统就又多了一层保护:

  1. 通过 public-key 登录 sshd, 这个本身是有 SSL 保护的, 所以目前依旧是安全的.
  2. 登录到 shell 后, 开启一个 vnc4server
    $vnc4server
  3. 在远程电脑上, 使用 VNC 兼容的客户端连接到 vnc4server, 由于只开了一个远程桌面, 所以连接是 xxx.xxx.xxx.xxx:1
  4. 使用这个桌面, 基本就可以利用浏览器管理公司内的其它设备了, 例如路由器/打印机/电话交换机等
  5. 完事后, 记得通过 shell 关掉 vnc4server:
    $vnc4server -kill :1
  6. $logout

这回黑客估计会放过我了吧 😀