-
A GKE Issue: Metadata Service Is Unreachable In Nodes
Recently I encountered some errors in a GKE cluster where a lot of pods were stuck at crash loop backoff state, which means the pods couldn’t recover on their own. When taking a closer look, I saw errors like: It’s a DNS issue then. However when I created a pod and ran some DNS tests…
-
How to Mount an Existing Google Persistent Disk to a Pod in GKE
TL; DR: Below are steps to mount an existing Google Persistent Disk to a pod in a GKE cluster in the same region. Most of the time I’d prefer to run stateless pods in Kubernetes clusters – they come, they go, as if nothing happened. When there are things to be persisted, a Persistent Volume…
-
Github Hosted Runners and Their Access to GCP VPC
TL; DR: Given plenty of free time of Github hosted runners, I’m tempted to use it instead of running self-hosted ones in my GCP environment. Here are some options to grant network access to my GCP VPC. Prerequisites: Google Workload Identity Federation for Github Runners Option #1, grab the public IP of the runner on-the-fly…
-
How to Setup Google Workload Identity Federation for Github Actions Runners
When using GHA(Github Actions) for CI/CD purposes, the Github-hosted runners are free unless you use it quite a lot. But if I want to use GHA to build a docker container image and save it to my private repository in Google Artifact Registry, I have to grant the permission to some VMs I don’t even…
-
Solved: Error 403: The caller does not have permission
Recently I got a very confusing error when setting up a PubSub subscription to write to a Big Query table via terraform. Here’s the partial terraform code snippet: And the input variable subscriptions looks like: It doesn’t show any error when doing terraform plan however it gives 403 error when applying. After some googling I…
-
How to Connect to Cloud SQL via SSH
TL;DR: here’s a handy bash script which can connect to a private Cloud SQL instance(MySQL in this case) via a bastion host. Prerequisites: Google Cloud SQL instance with mTLS certs provisioned Google Cloud SDK installed(the gcloud command) a bash shell(better be V5.0+) OpenSSH and MySQL CLI installed Google Cloud SQL is an RDB as a…
-
A Load Balancer for Cloud SQL Replicas
TL;DR: As of Aug 2022, there’s no out of box load balancing for Google Cloud SQL read replicas. So I built one for MySQL replicas with HAProxy with the ability to dynamically reload when number of replicas has changed. It will be quite straight forward to run a few HAProxy pods in Kubernetes as a…
-
Some Field Test with Google Cloud Run
Recently I got a chance to migrate on-premise applications to GCP(Google Cloud Platform), and ran the apps in containers via Cloud Run. Here are some pros and cons that I think about the fully managed Cloud Run. Pros: Very easy to get started. As long as the app can run in a container, it can…