Category: Systems Admin

  • [ Solved ] Service kubelet Can’t Start After Upgraded to 1.35

    When upgrading my home-lab Kubernetes cluster from 1.32 to 1.35, I encountered some error that the first node upgraded became not ready, eg. It should show 1.35 but it didn’t. I guess there should be some error in the logs of the kubelet service. So this –pod-infra-container-image flag needs to be removed. Easy. I probably…

  • [ Solved ] kube-apiserver Kept Crashing

    I planned to upgrade my home-lab Kubernetes cluster from 1.32 to 1.35, using the same shortcut I used last time. Unfortunately when I took a look at the cluster which I didn’t touch for a while, I couldn’t connect to it anymore. It’s not convenient at all – I can’t use kubectl to troubleshoot the…

  • [ Solved ] `33554536:system library:OPENSSL_internal:Connection reset by peer` Error with GCLB + Gateway API + OSS Istio

    I was trying to get a PoC of Google global external load-balancer to work with Gateway API and Istio as gateway class running in a standard GKE cluster. Everything was working as expected except I got My setup looks like below First I tried to test if the request can be served by the gateway:…

  • How to Test a Terraform Provider Locally

    In a .tf file, in a required_providers block all the Terraform providers will be listed there. When terraform plan is executed the providers will be pull from a registry or cache if defined. But how to test a provider that’s not published to a registry yet? Say the provider is called raynix so normally we’d…

  • Google SSO and ArgoCD Localhost

    TL; DR: I needed to enable Google SSO for ArgoCD at localhost because I planned to upgrade Istio using ArgoCD however the process will make ArgoCD unavailable as the ingress gateway ArgoCD depends on will be replaced I was doing an Istio upgrade for yet another GKE cluster but this time it’s a bit special…

  • How to Connect to a CloudSQL MySQL Instance with Private IP

    Not sure about everyone else, I wouldn’t run my MySQL server like a website with a public IP attached. Making the instance private is good as a security approach it is a bit inconvenient when I need to connect to it from my laptop. With Google Cloud CLI gcloud command, it’s very easy to connect…

  • Solved: HTTP 525/526 CloudFlare Errors

    HTTP 525/526 errors are unofficial HTTP errors specific to CloudFlare. In a nut shell they mean HTTP TLS cert related errors when communicating with origin web servers. I got one of these from my sites so I took a look into this. I use cert-manager’s integration with Let’s Encrypt to automatically renew my TLS certs.…

  • MySQL Online DDL Operations Made Simple with gh-ost

    What’s an online DDL operation? In a nut shell, a DDL(Data Definition Language) operation is something like alter table … which will change the data structure of a table. Some changes only touch metadata, eg. index, etc. so those can be updated in-place. But if a field definition needs to change, eg. UTF8mb3 –> UTF8mb4,…