Building Dynamic CI Pipeline with BuildKite


I was inspired by this BuildKite pipeline sample given by the support team:

# .buildkite/pipeline.yml
steps:
  - command: echo building a thing
  - block: Test the thing?
  - command: echo testing a thing
  - wait
  - command buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml

# .buildkite/pipeline.deploy.yml
steps:
  - block: Deploy the thing?
  - command: echo deploy the thing

So in the above case, if the first 2 commands succeed, pipeline.deploy.yml will be loaded into the main CI pipeline. This implementation is just brilliant. I’m not sure if jenkinsfile can do dynamic pipeline like this, but at least jenkinsfile won’t look as elegant as yaml.

Since buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml is just another bash command, I can even use it in a script to put more logic in it, such as git flow implementation like:

#!/bin/bash
export CHOICE=$(buildkite-agent meta-data get "next-section")

case $CHOICE in
deploy)
  buildkite-agent pipeline upload .buildkite/pipeline.qa.yml
  ;;
signoff)
  # feature finish
  if [[ $BUILDKITE_BRANCH == feature* ]]; then
    python .buildkite/scripts/github_ci.py \
      --action pr \
      --repo flow-work \
      --head $BUILDKITE_BRANCH \
      --base develop

  # release start
  elif [[ $BUILDKITE_BRANCH == develop ]]; then
    git checkout -b release/$FULL_VERSION
    git push --set-upstream origin release/$BUILDKITE_BUILD_NUMBER

  # release finish
  elif [[ $BUILDKITE_BRANCH == release* ]]; then
    buildkite-agent pipeline upload .buildkite/pipeline.pass.yml
  fi
  ;;
reject)
  #mark build as failure
  exit -1
  ;;

FYI. example tested with BuildKite agent version 3.2.0.

🙂