-
How to Provision Google Log-Based Metrics with Terraform and YAML
TL; DR: here are my notes on provisioning Google log-based metrics using Terraform and YAML. The metrics are defined in YAML in a clean fashion and they will be created via terraform apply The HCL part for Terraform Ref.1, https://cloud.google.com/logging/docs/logs-based-metrics2, https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_metric 🙂
-
Indexes for Structured Logs in Google Cloud
TL; DR: Google Cloud accepts structured logs ingested from different sources but will not index fields automatically. So by default the ingested jsonPayload will not be optimised for query. Here’s how to add indexes to it using Terraform. 🙂
-
Solved: Google Managed Prometheus Kept Crashing
Context: I use Google Cloud Managed Service for Prometheus(GMP I call it for convenience) as the central piece of my observability stack. In a nut shell it’s Prometheus managed by Google. GMP runs a collector Prometheus pod in each node, as a DaemonSet. The collector Prometheus scrapes metrics within the node and forwards them to…
-
A GKE Issue: Metadata Service Is Unreachable In Nodes
Recently I encountered some errors in a GKE cluster where a lot of pods were stuck at crash loop backoff state, which means the pods couldn’t recover on their own. When taking a closer look, I saw errors like: It’s a DNS issue then. However when I created a pod and ran some DNS tests…
-
How to Mount an Existing Google Persistent Disk to a Pod in GKE
TL; DR: Below are steps to mount an existing Google Persistent Disk to a pod in a GKE cluster in the same region. Most of the time I’d prefer to run stateless pods in Kubernetes clusters – they come, they go, as if nothing happened. When there are things to be persisted, a Persistent Volume…
-
Github Hosted Runners and Their Access to GCP VPC
TL; DR: Given plenty of free time of Github hosted runners, I’m tempted to use it instead of running self-hosted ones in my GCP environment. Here are some options to grant network access to my GCP VPC. Prerequisites: Google Workload Identity Federation for Github Runners Option #1, grab the public IP of the runner on-the-fly…
-
How to Setup Google Workload Identity Federation for Github Actions Runners
When using GHA(Github Actions) for CI/CD purposes, the Github-hosted runners are free unless you use it quite a lot. But if I want to use GHA to build a docker container image and save it to my private repository in Google Artifact Registry, I have to grant the permission to some VMs I don’t even…
-
Solved: Error 403: The caller does not have permission
Recently I got a very confusing error when setting up a PubSub subscription to write to a Big Query table via terraform. Here’s the partial terraform code snippet: And the input variable subscriptions looks like: It doesn’t show any error when doing terraform plan however it gives 403 error when applying. After some googling I…