Buildkite DSL

Expeditor’s Buildkite DSL is a custom YAML DSL written on top of Buildkite’s existing YAML DSL. The DSL is contained within the expeditor keyspace and is intended to provide shortcuts and additional Chef-specific functionality beyond what is available from Buildkite.

steps.#.expeditor

accounts

Automatically configure the CLI associated with one or more of the supported account types in the execution runtime of a single Buildkite job.

• aws Configure the aws CLI with the temporary STS credentials (as a profile) for the specified AWS account.

  steps:
    - label: a job that uses the aws CLI to interact with an AWS account
      commands:
        - aws --profile chef-example s3 ls
      expeditor:
        accounts:
          - aws/chef-example

• github Configure the git CLI to use a custom git credential helper that will fetch a temporary access token to the specified GitHub organization. This functionality currently only supports the chef GitHub organization.

  steps:
    - label: a job that uses the git CLI to fetch from a private GitHub repository
      commands:
        - git clone https://github.com/chef/private-repository.git
      expeditor:
        accounts:
          - github/chef

cache

Automatically configure the CLI associated with an S3 bucket to cache repository level directories before and after the execution runtime of a single Buildkite job step.

---
steps:
  - label: "default usage: run S3 cp to cache directories"
    expeditor:
      cached_folders:
      - foo/
  - label: "default usage: run S3 cp to cache directories"
    expeditor:
      cached_folders: [ "bar/" ]

secrets

Grant permission for a Buildkite step to access secrets stored in Chef’s internal Vault instance. Please check out our best practices for writing Bash scripts for guidance on which of the two patterns below you should use.

• Pattern 1: Only inject Vault credentials. This pattern will only inject the VAULT_ADDR, VAULT_NAMESPACE, and VAULT_TOKEN environment variables. This allows your step to make calls to Vault to fetch the secrets you need. If you are running in an executor where the vault CLI is not available to you, we recommend you use Pattern 2.

  ---
  steps:
    - label: "Bash script that needs access to Vault"
      command:
        - .expeditor/buildkite/my_script.sh
      expeditor:
        executor:
          docker:
        secrets: true

• Pattern 2: Pre-fetch the secrets and inject them as environment variables. This pattern will leverage Buildkite hooks to pre-fetch the secrets from Vault and make them available to your step as environment variables. For full details on how to reference secrets using this format, please check out the Secrets DSL reference documentation.

  ---
  steps:
    - label: "test that needs access to Vault"
      command:
        - echo \$MY_SECRET
      expeditor:
        secrets:
          GITHUB_TOKEN:
            account: github/chef
            field: token
          MY_SECRET:
            path: /secrets/my-app/my-secrets
            field: my-secrets

executor

For Buildkite pipelines managed with Expeditor, you select where to run your step by specifying an executor. The executor allows you to specify the where and how you want to run your step but how, combining the concept of Buildkite agent tags with plugins into a single entity.

Each executor has its own settings, but there are four settings that are available to all executors:

• host-os (string)
• os-version (string)
• privileged (boolean)
• single-use (boolean)

docker

The docker executor is a modified shim for the docker Buildkite Plugin. It supports all the settings that the plugin supports. We highly encourage that you use the docker executor whenever possible as it simplifies the management of the CI instances and makes it easier to limit and/or reproduce environment-specific errors.

---
steps:
  - label: "default usage: run the chefes/buildkite linux container on a linux host"
    expeditor:
      executor:
        docker:
  - label: "run the chefes/buildkite container in privileged mode on a linux host"
    expeditor:
      executor:
        docker:
          privileged: true
  - label: "run the chefes/buildkite-windows container on a windows host"
    expeditor:
      executor:
        docker:
          host-os: windows
  - label: "Run the chefes/buildkite-windows container in privileged mode on a windows host"
    expeditor:
      executor:
        docker:
          privileged: true
          host-os: windows
  - label: "default usage but specifying one of the docker-plugin settings"
    expeditor:
      executor:
        docker:
          environment:
            - FOO=bar

Unless an image setting is specified, the docker executor will use one of three Chef Release Engineering maintained Docker images which come pre-installed with the supported versions of all the programming languages used in Chef projects, as well as some other common utilities.

• chefes/buildkite (default)
• chefes/buildkite-windows (for host-os: windows)

Release Engineering does not manage tags for these images. If you’d like to control which iteration of the image you’d like to use we recommend you manage that using the image_sha256 setting in the expeditor.defaults.executor keyspace at the top of your pipeline definition, and update it automatically by executing a bash action in response to the docker_image_published workload for the appropriate image. You can see this pattern in use in the chef/automate GitHub repository.

subscriptions:
  - workload: docker_image_published:chefes/buildkite:*
    actions:
      - bash:.expeditor/update_docker_image_version_in_verify_pipeline.sh

#!/bin/bash

set -eou pipefail

# only bump the sha256 digest for the "latest" tag
if [[ "$EXPEDITOR_TAG" != "latest" ]]; then
  exit 0
fi

branch="expeditor/bump-chefes-buildkite"
git checkout -b "$branch"

sed -i -r "s|image_sha256: .+|image_sha256: ${EXPEDITOR_SHA256_DIGEST#"sha256:"}|" .expeditor/verify.pipeline.yml

git add .expeditor/verify.pipeline.yml

# give a friendly message for the commit and make sure it's noted for any future audit of our codebase that no
# DCO sign-off is needed for this sort of PR since it contains no intellectual property
dco_safe_git_commit "Bump chefes/buildkite version"

open_pull_request

# Get back to the release branch and cleanup the leftovers - any changed files left over at the end of this script will get committed to the release branch.
git checkout -
git branch -D "$branch"

expeditor:
  defaults:
    executor:
      docker:
        image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

steps:
  - label: "run a thing in docker"
    expeditor:
      executor:
        docker:

linux

The linux executor will simply run your step on an available Amazon Linux 2 machine.

steps:
  - label: "simple executor"
    expeditor:
      executor:
        linux:
  - label: "run on a privileged machine"
    expeditor:
      executor:
        linux:
          privileged: true
  - label: "run on a single use machine"
    expeditor:
      executor:
        linux:
          single-use: true
  - label: "run on a privileged, single-use machine"
    expeditor:
      executor:
        linux:
          privileged: true
          single-use: true

macos

The macos executor will run your step on an available Mac OS machine. Under the covers, we use our own Anka Buildkite plugin to launch ephemeral containers of either 10.15, 11, or 12. By default, we run your step in a Mac OS 10.15 container. You may pass any of the defined Anka Buildkite plugin settings to the macos executor.

steps:
  - label: "run command on latest macos"
    expeditor:
      executor:
        macos:
  - label: "run command on macos 10.15"
    expeditor:
      executor:
        macos:
          os-version: "10.15"

windows

The windows executor will run your step on an available Windows Server 2019 or Windows Server 2016 machine. By default, we run your step in a Windows Server 2019 machine.

steps:
  - label: "run on the default (windows 2019)"
    expeditor:
      executor:
        windows:
  - label: "run on windows 2016"
    expeditor:
      executor:
        windows:
          os-version: "2016"
  - label: "run on a privileged machine"
    expeditor:
      executor:
        windows:
          privileged: true
  - label: "run on a single use machine"
    expeditor:
      executor:
        windows:
          single-use: true
  - label: "run on a privileged, single-use machine"
    expeditor:
      executor:
        windows:
          privileged: true
          single-use: true

expeditor

accounts

Automatically configure the CLI associated with one or more of the supported account types in the execution runtime of a every Buildkite job in a pipeline.

---
expeditor:
  accounts:
    - github/chef

steps:
  - command: make test1
  - command: make test2
  - command: make test3

If there is one or more steps where you don’t need or want an account that is specified in expeditor.defaults.accounts, you can reject it at the step level using by prefixing it with a bang (!).

---
expeditor:
  accounts:
    - github/chef

steps:
  - command: make test1
  - command: make test2
  - command: make test3
    expeditor:
      accounts:
        - !github

cache

Automatically configure the CLI associated with an S3 bucket to cache repository level directories before and after the execution runtime of every Buildkite job step.

---
expeditor:
  cached_folders:
  - foo/

steps:
  - label: "default usage: run S3 sync to cache directories"
    command: test 1
    command: test 2

If there is one or more steps where you do not need to cache directories, you can reject it at the step level by adding adding a bang (!) in front of the directory you’d like skipped.

---
expeditor:
  cached_folders:
  - foo/

steps:
  - label: "default usage: run S3 cp to cache directories"
    command: test 1
    command: test 2

  - label: "default usage: skip S3 cp for directory foo/"
    command: test 1
    expeditor:
      cached_folders:
      - "!foo/"
      - bar/

defaults

The expeditor.defaults keyspace provides you a means to automatically inject certain step-level configuration across all your steps.

buildkite

Specify settings that are part of the native Buildkite DSL that should applied to every step in your pipeline definition file. If there is a conflict, Expeditor will default to the value specified in the step.

---
expeditor:
  defaults:
    buildkite:
      timeout_in_minutes: 15

steps:
  - command: make short-test
  - command: make another-short-test
  - command: make long-test
    timeout_in_minutes: 30

In the example above, the fully processed pipeline definition file would look like this:

---
steps:
  - command: make short-test
    timeout_in_minutes: 15
  - command: make another-short-test
    timeout_in_minutes: 15
  - command: make long-test
    timeout_in_minutes: 30

executor

Specify default settings for executors used in steps.

expeditor:
  defaults:
    executor:
      docker:
        image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

steps:
  - command: make short-test
    expeditor:
      executor:
        linux:
  - command: make another-short-test
    expeditor:
      executor:
        docker:
  - command: make long-test
    expeditor:
      executor:
        docker:

The example below is logically equivalent to one above, except that the image_sha256 value is specified once instead of N times.

steps:
  - command: make short-test
    expeditor:
      executor:
        linux:
  - command: make another-short-test
    expeditor:
      executor:
        docker:
          image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9
  - command: make long-test
    expeditor:
      executor:
        docker:
          image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

secrets

If you would like to have an environment variable secret injected into every step in your pipeline definition, you can specify the secret in the expeditor.secrets keyspace.

---
expeditor:
  secrets:
    GITHUB_TOKEN:
      account: github/chef
      field: token

steps:
  - command: make test1
  - command: make test2
  - command: make test3