How to Provision Google Log-Based Metrics with Terraform and YAML


TL; DR: here are my notes on provisioning Google log-based metrics using Terraform and YAML. The metrics are defined in YAML in a clean fashion and they will be created via terraform apply

# my-metrics.yaml
metrics:
  my-custom-metric-from-logs:
    bucket_name: my-logging-bucket-name
    filter: |
      logName="projects/my-gcp-project/logs/my-logging-bucket-name"
      resource.type="global"
      jsonPayload.filter.criteria="block"
    labels:
      ip:
        type: STRING
        field: jsonPayload.client_ip
        regex: ([0-9]+\.[0-9]+\.[0-9]+) # only grab the first 3 numbers to reduce the amount of labels
# some optional defaults
defaults:
  type: INT64
  unit: "1"
  disabled: false        

The HCL part for Terraform

locals {
  manifest = yamldecode(file("./my-metrics.yaml"))
  metrics = {
    for k, v in local.manifest.metrics : k => merge(v, {
      // bucket_name is optional in YAML as global logs won't need this
      bucket_name = can(v.bucket_name) ? format(
        "projects/%s/locations/global/buckets/%s",
        var.project,
        v.bucket_name
      ) : null
      // check and use REGEXP_EXTRACT when regex is present
      label_extractors = {
        for lk, lv in try(v.labels, {}) : lk =>
        can(lv.regex) ? "REGEXP_EXTRACT(${lv.field}, \"${lv.regex}\")" : "EXTRACT(${lv.field})"
      }
    })
  }
  defaults = local.manifest.defaults
}

resource "google_logging_metric" "default" {
  for_each    = local.metrics
  project     = var.project
  name        = each.key
  filter      = each.value.filter
  bucket_name = each.value.bucket_name
  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = try(each.value.type, local.defaults.type, "INT64")
    unit        = try(each.value.unit, local.defaults.unit, "1")

    dynamic "labels" {
      for_each = try(each.value.labels, {})

      content {
        key         = labels.key
        value_type  = labels.value.type
        description = try(labels.value.description, "")
      }
    }

    display_name = try(each.value.name, each.key)
  }

  value_extractor  = try(each.value.value_extractor, "")
  label_extractors = each.value.label_extractors
}

Ref.
1, https://cloud.google.com/logging/docs/logs-based-metrics
2, https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_metric

🙂