Recently I got a very confusing error when setting up a PubSub subscription to write to a Big Query table via terraform
. Here’s the partial terraform
code snippet:
resource "google_pubsub_subscription" "subscriptions" { for_each = var.subscriptions ... dynamic "bigquery_config" { for_each = can(each.value.bigquery_config) ? [each.value.bigquery_config] : [] content { table = bigquery_config.value.table write_metadata = true } } }
And the input variable subscriptions
looks like:
subscriptions = { "my-test-sub" = { topic = "my-test-topic" ... bigquery_config = { table = "my-gcp-project.my-dataset-id.my-table-id" } }
It doesn’t show any error when doing terraform plan
however it gives 403 error when applying.
googleapi: Error 403: The caller does not have permission
After some googling I found this issue very relevant, because in Google’s doc(Example Usage – Pubsub Subscription Push Bq as of 6th Sept 2023) project.dataset.table
is the format of the table name however the format that actually works is project:dataset.table
. So the variable looks like this now:
subscriptions = { "my-test-sub" = { topic = "my-test-topic" ... bigquery_config = { table = "my-gcp-project:my-dataset-id.my-table-id" } }
But I still got that 403 when applying, which makes 0 sense to me because I’m the owner of my GCP project obviously. Retrying a few more times with terraform apply
won’t change anything, I decided to try the same thing via the Cloud Console – maybe it works there then I can see what should be changed in my code.
In the Cloud Console GUI, I tried to create the subscription the click-ops way, and voila! Here’s the real reason of the 403s:
Service account [email protected] is missing permissions required to write to the BigQuery table: bigquery.tables.get, bigquery.tables.updateData.
So in a nutshell the 403 wasn’t for me but for this built-in service account of PubSub. And the best role supplying bigquery.tables.get
and bigquery.tables.updateData
permissions is roles/bigquery.dataEditor
. After this role binding was added, terraform apply
finally succeeded 🙂