Category: Systems Admin

  • Hello World, Grafana Tanka

    I liked YAML a lot, until it gets longer and longer, and even longer. There are tools to make YAML ‘DRY’, the popular ones are Helm and Kustomize. But none of them can say it got the job done. To be honest, I didn’t like Helm much from the start. Helm uses templating syntax similar […]

  • Rebuild a Kubernetes Node Without Downtime

    When I built the in-house Kubernetes cluster with Raspberry PIs, I followed the kubeadm instructions and installed Raspberry PI OS on the PIs. It was all good except the RPI OS is 32-bit. Now I want to install a Ubuntu 20.04 Server ARM64 on this PI, below are steps with which I rebuilt the node […]

  • OpenSSL Commands to Verify TLS Certs in Kubernetes Secrets

    Sometimes a TLS cert deployed into a Kubernetes cluster in a Secret doesn’t work as expected. Here are some handy commands to verify the certs. The sample commands work for Istio Ingressgateway, but should be adapted to other CNIs without huge efforts. Commands to verify the cert served by your web-app Commands to verify the […]

  • Fixed: Duplicated Cluster IPs in a Kubernetes Cluster

    A Cluster IP is an IP address allocated in the cluster’s virtual LAN, usually allocated to Kubernetes Services. As a user nobody should care about which cluster IP a service would get because we will use the service-name.namespace-name DNS name which will map to that IP automatically. In my case, some services in a GKE […]

  • The Upgrade of Kubernetes Ingress Nginx

    The ingress-nginx container image I’ve been using was v0.25, and that’s more than 1 year old. The recent release is v0.44 but it’s a big leap from 25 to 44 and I’ve found some major differences between the 2 versions. Version 0.25 implemented API version of while version 0.44 has Here are samples […]

  • Fixed gsutil OSError: Permission Denied

    When trying to run gsutil in a kubernetes Job as nobody to backup stuff to Google Cloud Storage(GCS), I encountered simple error messages like But it didn’t say where or how the permission was denied! It worked fine if the container was run as root user so the problem is not with Google Cloud. I […]

  • Kubernetes: How to Use Affinity

    Affinity is a great feature in Kubernetes to assign pods to nodes based on labels. In my case, I have a hybrid Kubernetes cluster with half nodes are of X86 architecture and other half of ARM architecture, and I need to deploy the X86 only containers to the X86 nodes. Of course I can build […]

  • Kubernetes and GitOps with Flux CD V2.0

    GitOps could be the next big thing in cloud automation so I’d give it a try with my in house hybrid Kubernetes cluster. I was recommended to try Flux CD and there’s a good reference project initiated by my colleage: k8s-gitops. However, in order to fully understand how to use Flux CD, I chose to […]