-
MySQL Online DDL Operations Made Simple with gh-ost
What’s an online DDL operation? In a nut shell, a DDL(Data Definition Language) operation is something like alter table … which will change the data structure of a table. Some changes only touch metadata, eg. index, etc. so those can be updated in-place. But if a field definition needs to change, eg. UTF8mb3 –> UTF8mb4,…
-
Monitoring MySQL with mysqld-exporter
TL; DR: I use mysqld-exporter with Prometheus to monitor my MySQL server. The mysqld-exporter runs as a container in my Kubernetes cluster. A sample manifest looks like: 🙂
-
Cloud SQL MySQL: Best Password is No Password
Security for web applications has come a long way. In the case of database passwords, there is a lot of approaches to keep them safe: But isn’t it the best if there’s no need for a password at all? With IAM authentication for Cloud SQL, the password can be avoided after all. Here’s how to…
-
How to Connect to Cloud SQL via SSH
TL;DR: here’s a handy bash script which can connect to a private Cloud SQL instance(MySQL in this case) via a bastion host. Prerequisites: Google Cloud SQL instance with mTLS certs provisioned Google Cloud SDK installed(the gcloud command) a bash shell(better be V5.0+) OpenSSH and MySQL CLI installed Google Cloud SQL is an RDB as a…
-
A Load Balancer for Cloud SQL Replicas
TL;DR: As of Aug 2022, there’s no out of box load balancing for Google Cloud SQL read replicas. So I built one for MySQL replicas with HAProxy with the ability to dynamically reload when number of replicas has changed. It will be quite straight forward to run a few HAProxy pods in Kubernetes as a…
-
Flyway Container, MySQL and SSL/mTLS
Flyway is a handy utility to manage database schema migrations. Very similar to the schema migration mechanism in Ruby on Rails or Django, but Flyway is a standalone tool. So it’s best suited for some project which doesn’t have database schema management yet. I needed to use Flyway for a project I worked with, the…
-
Use MySQL Trigger To Do Incremental ETL
There’s a huge MySQL table that I need to ETL to Google BigQuery daily, about 1 billion rows. The rows are updated in a random fashion all the time so I can’t do incremental ETL by the recording the max primary key. Then my colleague brought up the trigger idea, which I believe is the…
-
MySQL/ Aurora to Google BigQuery
Google BigQuery(BQ) is Google’s column storage, it’s super fast when you need to query over many gigabytes of data. BQ defaults to utf8, so it makes sense the tables in MySQL/ Aurora are also utf8 encoded. I use Python and SQL Alchemy to load data to BQ. It’s a waste for SA’s ORM features but…