-
How to Prune Only Selected Fields in Jsonnet
In Jsonnet’s standard libraries, there’s an std.prune function which will recursively go through an object and remove any empty field in the object which is quite handy, usually. In my case I wanted to prune all empty fields in a container object but spare the legit empty emptyDir field so the std.prune may cause some…
-
How to Do Kubernetes Cronjobs with Sidecars Properly in 2024
TL; DR: Since Kubernetes v1.28, there’s no need to do fancy bash scripts to kill your sidecar when the main container finishes. So using sidecars in Kubernetes Jobs or Cronjobs are now much simpler and more elegant too. For example, I wanted to run a database backup job in a GKE + Cloud SQL environment,…
-
Solved: Google Managed Prometheus Kept Crashing
Context: I use Google Cloud Managed Service for Prometheus(GMP I call it for convenience) as the central piece of my observability stack. In a nut shell it’s Prometheus managed by Google. GMP runs a collector Prometheus pod in each node, as a DaemonSet. The collector Prometheus scrapes metrics within the node and forwards them to…
-
How to Use Pod Anti-Affinity in Kubernetes
By default the Kubernetes scheduler distributes pods of a replica set evenly to all nodes, if no taints are present of course. So why or when do we need pod anti-affinity? 1 scenario I can think of is like this: Without any advanced tuning such as pod anti-affinity, the replicas can possibly be scheduled like…
-
Monitoring MySQL with mysqld-exporter
TL; DR: I use mysqld-exporter with Prometheus to monitor my MySQL server. The mysqld-exporter runs as a container in my Kubernetes cluster. A sample manifest looks like: 🙂
-
How to Upgrade a Kubernetes Cluster with `kubeadm` in 2024
TL; DR: I upgraded my Garage Kubernetes Lab cluster from 1.28 to 1.29 recently. Here’s how I did it. Upgrading the Control Plane First, the Linux package repository needs to be updated to include kubeadm 1.29. This can be done like this(My cluster is built with Ubuntu 22.04, for other Linux distributions please refer to…
-
How to Upgrade Istio the Git-Ops Way
Istio 1.17 has been running really well in my Garage Kubernetes lab, but I thought it’s time to give it an upgrade. It was installed and configured using istioctl utility last time when I bootstrapped the cluster. Since I’ve automated a lot of Kubernetes resources using ArgoCD I would like to upgrade Istio the git-ops…
-
How to Calculate Proportions Using 2 Metrics in Prometheus
Recently I needed to improve an alert defined using Prometheus Alert Manager to trigger when less than half of the minimum replicas are up, eg. I had a look at existing metrics collected by kube-state-metrics in my Kubernetes cluster, the number of replicas running well could be queried with the kube_pod_status_ready metric and the number…