Posted on

Rebuild a Kubernetes Node Without Downtime

When I built the in-house Kubernetes cluster with Raspberry PIs, I followed the kubeadm instructions and installed Raspberry PI OS on the PIs. It was all good except the RPI OS is 32-bit. Now I want to install a Ubuntu 20.04 Server ARM64 on this PI, below are steps with which I rebuilt the node with Ubuntu and without disrupting the workloads running in my cluster.

First, I didn’t need to shutdown the running node because I’ve got a spare MicroSD card to prepare the Ubuntu image. The instruction for writing the image to the MicroSD card is here. When the card is prepared by the Imager, I kept it in the card reader because I wanted to set the IP address instead of the automatic IP by default. A fixed IP makes more sense if I want to connect to it, right?

To set a static IP in the Ubuntu MicroSD card, open system-boot/network-config file with a text editor and put in something like this:

version: 2
ethernets:
  eth0:
    # Rename the built-in ethernet device to "eth0"
    match:
      driver: bcmgenet smsc95xx lan78xx
    set-name: eth0
    addresses: [192.168.1.82/24]
    gateway4: 192.168.1.1
    nameservers:
      addresses: [192.168.1.1]
    optional: true

Now the new OS is ready. To gracefully shutdown the node, drain it with

kubectl drain node-name
# wait until it finishes
# the pods on this node will be evicted and re-deployed into other nodes
kubectl delete node node-name

Then I powered down the PI and replaced the MicroSD card with the one I just prepared, then I powered it back on. After a minute or 2, I was able to ssh into the node with 

# wipe the previous trusted server signature
ssh-keygen -R 192.168.1.82
# login, default password is ubuntu and will be changed upon first login
ssh [email protected]
# install ssh key, login with updated password
ssh-copy-id [email protected]

The node needs to be prepared for kubeadm, I used my good old ansible playbook for this task. The ansible-playbook command looks like

ansible-playbook -i inventory/cluster -l node-name kubeadm.yaml

At the moment I have to install less recent versions of docker and kubeadm to keep it compatible with the existing cluster.

When running kubeadm join command I encountered an error message saying CGROUPS_MEMORY: missing. This can be fixed with this. And one more thing is to create a new token from the master node with command:

kubeadm token create

At last the new node can be joined into the cluster with command:

kubeadm join 192.168.1.80:6443 --token xxx     --discovery-token-ca-cert-hash sha256:xxx

The node will then be bootstrapped in a few minutes. I can tell it’s now ARM64

k get node node-name -o yaml |rg arch
    beta.kubernetes.io/arch: arm64
    kubernetes.io/arch: arm64
...

🙂

Posted on

OpenSSL Commands to Verify TLS Certs in Kubernetes Secrets

Sometimes a TLS cert deployed into a Kubernetes cluster in a Secret doesn’t work as expected. Here are some handy commands to verify the certs. The sample commands work for Istio Ingressgateway, but should be adapted to other CNIs without huge efforts.

Commands to verify the cert served by your web-app

# Use openssl to retrieve cert and decrypt and print it out
# This can be used to verify that the correct cert is in use in an gateway
# use ctrl-c to end it
openssl s_client  -connect my.example.com:443 -showcerts -servername my.example.com |openssl x509 -noout -text

# Print out dates of the cert
openssl s_client  -connect my.example.com:443 -showcerts -servername my.example.com |openssl x509 -noout -dates

# Print out the subject/CN of the cert
openssl s_client  -connect my.example.com:443 -showcerts -servername my.example.com |openssl x509 -noout -subject

# Print out the subjectAltName/SAN of the cert
openssl s_client  -connect my.example.com:443 -showcerts -servername my.example.com |openssl x509 -noout -text |grep 'Subject Alternative Name' -A1

Commands to verify the cert installed in a secret

# This needs access to secrets so the cert secret can be downloaded and verified
kubectl get secret -n istio-system my-namespace-cert-secret -o yaml

# one-liner to print md5 hash of the X509 modulus from the cert
kubectl get secret -n istio-system my-namespace-cert-secret -o jsonpath={'.data.cert'} |base64 -d | openssl x509 -noout -modulus |openssl md5
# example output
c17642...

# one-liner to print md5 hash of the RSA modulus from the key
# this output has to match the previous one.
kubectl get secret -n istio-system my-namespace-cert-secret -o jsonpath={'.data.key'} |base64 -d | openssl rsa -noout -modulus |openssl md5
# example output
c17642...

🙂

Posted on

Fixed: Duplicated Cluster IPs in a Kubernetes Cluster

A Cluster IP is an IP address allocated in the cluster’s virtual LAN, usually allocated to Kubernetes Services. As a user nobody should care about which cluster IP a service would get because we will use the service-name.namespace-name DNS name which will map to that IP automatically.

In my case, some services in a GKE cluster got identical cluster IPs which caused them unreachable. With help from Google’s GCP support, the result is a bit funny:

The cluster was built with a /23 sub-net, which can only hosts about 510 IPs. At the time when the issue occurred, there were about 540 services in the cluster, so the last few services got duplicated IPs. It’s unexpected to me because I would rather get some error when Cluster IPs run out.

There’s no real fix to this situation as the sub-net is immutable after the cluster is built. So I migrated some workloads out of this cluster to get the service count back down.

Verdict: A Kubernetes cluster should be built with at least a /16(65534 IPs) sub-net to have a large enough IP pool for services in a busy cluster.

Posted on

The Upgrade of Kubernetes Ingress Nginx

The ingress-nginx container image I’ve been using was v0.25, and that’s more than 1 year old. The recent release is v0.44 but it’s a big leap from 25 to 44 and I’ve found some major differences between the 2 versions.

Version 0.25 implemented API version of networking.k8s.io/v1beta1 while version 0.44 has networking.k8s.io/v1. Here are samples for both versions:

# v1beta1
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: wordpress
  annotations:
    kubernetes.io/ingress.class: prod
spec:
  rules:
    - host: raynix.info
      http:
        paths:
          - path: /
            backend:
              serviceName: wordpress
              servicePort: 8080
# v1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress
spec:
  ingressClassName: prod
  rules:
    - host: raynix.info
      http:
        paths:
          - path: /
            pathType: prefix
            backend:
              service:
                name: wordpress
                port: 
                  number: 8080

For more info on networking.k8s.io/v1 please see this page

🙂