-
Solved: Kustomize Unable to Parse SM or JSON Patch
Recently I noticed that my ArgoCD manifests I used to patch using kustomize don’t work with latest version of kustomize anymore. The error I got looks like: My patch.yaml had something like this: There’s no syntax error in my files as this worked before. I tried to remove sections from the patch file and see…
-
How to Use Dynamic Storage Provisioning in Kubernetes
Here are the notes on how I enabled dynamic provisioning for persistent volumes in my garage Kubernetes cluster, using a combination of: NFS server I used a typical installation of NFS server on Ubuntu Linux. Here’s the documentation from Ubuntu. CSI driver for NFS From the Github repo there are Helm templates provided to install…
-
How to Do Kubernetes Cronjobs with Sidecars Properly in 2024
TL; DR: Since Kubernetes v1.28, there’s no need to do fancy bash scripts to kill your sidecar when the main container finishes. So using sidecars in Kubernetes Jobs or Cronjobs are now much simpler and more elegant too. For example, I wanted to run a database backup job in a GKE + Cloud SQL environment,…
-
How to Use Pod Anti-Affinity in Kubernetes
By default the Kubernetes scheduler distributes pods of a replica set evenly to all nodes, if no taints are present of course. So why or when do we need pod anti-affinity? 1 scenario I can think of is like this: Without any advanced tuning such as pod anti-affinity, the replicas can possibly be scheduled like…
-
How to Upgrade a Kubernetes Cluster with `kubeadm` in 2024
TL; DR: I upgraded my Garage Kubernetes Lab cluster from 1.28 to 1.29 recently. Here’s how I did it. Upgrading the Control Plane First, the Linux package repository needs to be updated to include kubeadm 1.29. This can be done like this(My cluster is built with Ubuntu 22.04, for other Linux distributions please refer to…
-
How to Upgrade Istio the Git-Ops Way
Istio 1.17 has been running really well in my Garage Kubernetes lab, but I thought it’s time to give it an upgrade. It was installed and configured using istioctl utility last time when I bootstrapped the cluster. Since I’ve automated a lot of Kubernetes resources using ArgoCD I would like to upgrade Istio the git-ops…
-
A Simple and Interactive Decoder for Kubernetes Secrets
TL; DR: Here’s a simple shell function which can decode Kubernetes secrets interactively and should work in any Bash and compatible environments. Requirements: Here’s the code. This can be chained together with other commands too, eg. on a Mac, I can do Then the decoded content of selected secret key will be put into clipboard.…
-
A GKE Issue: Metadata Service Is Unreachable In Nodes
Recently I encountered some errors in a GKE cluster where a lot of pods were stuck at crash loop backoff state, which means the pods couldn’t recover on their own. When taking a closer look, I saw errors like: It’s a DNS issue then. However when I created a pod and ran some DNS tests…