Buildkite DSL

Buildkite pipelines are YAML (or JSON) files comprised of a number of different types of steps. The most common type of step being the command step which is used to execute one or more shell commands. However, the attributes that can be provided to these steps are limited. In this document, attributes supported by native Buildkite pipelines will be referred to as “core attributes.”

A powerful capability of the Buildkite Agent is to dynamically define the steps of a pipeline using the buildkite-agent pipeline upload command. At Chef, we’ve established a pattern where the first step for all pipelines is a trigger step which passes a pipeline definition file into this command. This offers us a ton of flexibility and consistency in how we manage our pipelines.

With the help of the Expeditor CLI, we’re able to duplicate this process with the added benefit of being able to add hooks that allow us to process these definition files before they are fed to the buildkite-agent pipeline upload command.

These hooks include, but are not limited to:

In this document we’ll use the term “keyspace” to refer to a specific key-value pair in a large hash of key-value pairs. For example, when we say “expeditor.defaults.buildkite keyspace”, we are referring to the following:

---
expeditor:
  defaults:
    # below is the expeditor.defaults.buildkite "keyspace"
    buildkite:
      ...

CLI Accounts

Some command steps in pipelines may require CLI utilities such as git and aws to be configured. More information about which accounts are available can be found in our Secrets DSL documentation.

Specifying an account does not inject any environment variables. The account setting is here to pre-configure the corresponding CLI. If you need an environment variable (GITHUB_TOKEN, AWS_ACCESS_KEY_ID, etc), you’ll want to use the secrets setting.
---
steps:
  - label: "test that requires github"
    expeditor:
      accounts:
        - github
  - label: "test that requires chef-cd aws account"
    expeditor:
      accounts:
        - aws/chef-cd
  - label: "test that requires the chef-engineering aws account"
    expeditor:
      accounts:
        - aws/chef-engineering

Supported CLI Account Types

The following are the types of accounts that are supported in the expeditor.accounts and expeditor.defaults.accounts keyspace. The full Secrets DSL has access to other types of accounts, but these are the only ones for which Expeditor will configure the corresponding CLI utility.

github

When github is specified, the Expeditor CLI will configure the expeditor git-credential-helper command as a git credential helper. This helper will fetch the necessary credentials from Vault allowing you easy access to protected GitHub repositories.

aws

When aws is specified, the Expeditor CLI will generate temporary credentials for the specified account and confingure the aws CLI with the specified profile. You can, if neccesary, specify more than one aws account on the same step.

For example, if you specify the aws/chef-cd account, you will be allowed to run the aws CLI with the --profile chef-cd options in commands like aws --profile chef-cd s3 list-buckets. The full list of supported AWS accounts can be found in Vault.

Environment Variable Secrets

The expeditor.secrets and expeditor.defaults.secrets keyspace allows you to inject environment variables populated with data from our Vault instance into a Buildkite step. The specifics of the Secrets DSL are covered here.

---
steps:
  - label: "test that needs github credentials"
    expeditor:
      secrets:
        GITHUB_TOKEN:
          account: github
          field: token

Enabling Vault access for step

If you want your Buildkite step to have access to Vault, but do not want to specify specific environment variables, you can set expeditor.secrets or expeditor.default.secrets to true.

---
steps:
  - label: "test that needs access to Vault"
    expeditor:
      secrets: true

Selecting an Execution Runtime

Buildkite provides very powerful tools to control where and how steps are executed. At Chef Software, we want to keep step executions as uniform as possible to ensure consistency and avoid drift or decay. To solve this, we provide a number of different executors that you can choose from, with settings that allow you to customize your execution runtime.

There is no default executor. If you do not specify an executor, we fall back to default Buildkite behavior of scheduling you on the default queue.

Global Settings

Each executor will have specific settings, but all executors (except macos) support the privileged and single-use settings which can be used together or separately.

Privileged

When privileged is set to true, Expeditor will schedule the build on a machine that has the following:

  • Linux
  • Windows
    • The buildkite-agent user is a member of the Administrators group.
---
steps:
  - label: "test that requires sudo access"
    expeditor:
      executor:
        linux:
          privileged: true

Single Use

When single-use is set to true, Expeditor will schedule the build on a machine that will be terminated after the job is complete, even if the job is unsuccessful. This setting should only be used in situations where your test cannot be run in a container and modifies the instance in some unrecoverable way.

The macos executor does not support the single-use setting. It uses ephemeral containers by default. See macos executor for more info.
---
steps:
  - label: "test that mangles the instance"
    expeditor:
      executor:
        linux:
          single-use: true

Executor Flavors

docker

We highly encourage that you use executor.docker whenever possible. It simplifies the management of the CI instances and makes it easier to limit and/or reproduce environment-specific errors.

The docker executor is a modified shim for the docker Buildkite Plugin. It supports all the settings that the plugin supports.

---
steps:
  - label: "default usage for docker executor"
    expeditor:
      executor:
        docker:
  - label: "run docker in privileged mode"
    expeditor:
      executor:
        docker:
          privileged: true
  - label: "using docker-plugin settings with executor"
    expeditor:
      executor:
        docker:
          environment:
            - FOO=bar

Specifying an OS and OS Version

We have images pre-built for Linux, Windows Server 2019 and Windows Server 2016.

steps:
  - label: "Linux command"
    expeditor:
      executor:
        docker               # uses chefes/buildkite Linux image
steps:
  - label: "Windows Server 2019 command"
    expeditor:
      executor:
        docker:
          host-os: windows   # uses chefes/buildkite-windows Windows Server 2019 image
steps:
  - label: "Windows Server 2016 command"
    expeditor:
      executor:
        docker:
          host-os: windows
          os-version: 2016   # uses chefes/buildkite-windows-2016 Windows Server 2016 image

Specifying an Image

Unless an image setting is specified, the docker executor will use one of three Chef Release Engineering maintained Docker images: chefes/buildkite for linux, chefes/buildkite-windows for Windows 2019, and chefes/buildkite-windows-2016 for Windows 2016 . These images come pre-installed with the supported versions of all the programming languages used in Chef projects, as well as some other common utilities.

The chefes/buildkite and chefes/buildkite-windows images do not have version tags. If you’d like to control which iteration of the image you’d like to use we recommend you manage that using the image_sha256 executor setting. This will configure the Docker plugin to pull that iteration of the image.

This value can be set as a default at the top of your pipeline definition, and updated automatically by running a simple bash script in response to the docker_image_published for this image.

.expeditor/config.yml

subscriptions:
  - workload: docker_image_published:chefes/buildkite:*
    actions:
      - bash:.expeditor/update_docker_image_version_in_verify_pipeline.sh

.expeditor/update_docker_image_version_in_verify_pipeline.sh

#!/bin/bash

set -eou pipefail

# only bump the sha256 digest for the "latest" tag
if [[ "$EXPEDITOR_TAG" != "latest" ]]; then
  exit 0
fi

branch="expeditor/bump-chefes/buildkite"
git checkout -b "$branch"

sed -i -r "s|image_sha256: .+|image_sha256: ${EXPEDITOR_SHA256_DIGEST#"sha256:"}|" .expeditor/verify.pipeline.yml

git add .expeditor/verify.pipeline.yml

# give a friendly message for the commit and make sure it's noted for any future audit of our codebase that no
# DCO sign-off is needed for this sort of PR since it contains no intellectual property
git commit --message "Bump chefes/buildkite version" --message "This pull request was triggered automatically via Expeditor." --message "This change falls under the obvious fix policy so no Developer Certificate of Origin (DCO) sign-off is required."

open_pull_request

# Get back to master and cleanup the leftovers - any changed files left over at the end of this script will get committed to master.
git checkout -
git branch -D "$branch"

.expeditor/verify.pipeline.yml

expeditor:
  defaults:
    expeditor:
      executor:
        docker:
          image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

steps:
  - label: "run a thing in docker"
    expeditor:
      executor:
        docker:

linux

The linux executor will simply run your step on an available Linux machine. It doesn’t do anything special other than allow you to leverage the single-use and privileged executor settings.

We highly recommend using the linux executor vs no executor at all. It allows us to manage default behavior more efficiently.
steps:
  - label: "simple executor"
    expeditor:
      executor:
        linux:

macos

The macos executor will run your step on an available Mac OS machine. Under the covers, we use the chef/anka Buildkite plugin to launch ephemeral containers of either Mac OS 10.12, 10.13, or 10.14. By default, we run your step in a Mac OS 10.14 container. You may pass any of the defined chef/anka buildkite plugins to the macos executor.

The macos executor is not currently supported in public Buildkite pipelines.
steps:
  - label: "mac command"
    expeditor:
      executor:
        macos:
Specifying an OS Version

We have images pre-built for Mac OS 10.12, 10.13, and 10.14.

steps:
  - label: "macos 10.13 command"
    expeditor:
      executor:
        macos:
          os-version: 10.13

windows

The windows executor will run your step on an available Windows Server 2019 or Windows Server 2016 machine. By default, we run your step in a Windows Server 2019 machine.

steps:
  - label: "simple executor"
    expeditor:
      executor:
        windows:
Specifying an OS Version
steps:
  - label: "Windows Server 2016 command"
    expeditor:
      executor:
        windows:
          os-version: 2016

Global Default Settings

While you can use things like YAML anchors to reduce duplication, the Expeditor Buildkite DSL offers an expeditor.defaults keyspace that allows you to specify default settings with a bit more nuance.

---
expeditor
  defaults: # the expeditor.defaults keyspace
    buildkite: # defaults for core attributes
      ...
    executor:  # default settings for executors
      ...

steps:
  ...

Default Buildkite Settings

The expeditor.defaults.buildkite keyspace gives you a place to provide default value for common core attributes. The attributes provided in this keyspace must be valid command step core attribute.

When Expeditor is processing a pipeline definition file with expeditor.defaults.buildkite settings, it performs a merge between the default attributes and the step attributes, deferring to the step attributes.

---
expeditor:
  defaults:
    buildkite:
      timeout_in_minutes: 15

steps:
  - command: make short-test
  - command: make another-short-test
  - command: make long-test
    timeout_in_minutes: 30

In the example above, the fully processed pipeline definition file would look like this:

---
steps:
  - command: make short-test
    timeout_in_minutes: 15
  - command: make another-short-test
    timeout_in_minutes: 15
  - command: make long-test
    timeout_in_minutes: 30

Default Executor Settings

Settings specified in the expeditor.defaults.executor keyspace are used as defaults when those executors are reference in a step. Unlike the expeditor.defaults.buildkite keyspace, which applies the default setting to every step, the expeditor.defaults.executor keyspace applies the default only when the executor is referenced.

Below is one common example of using expeditor.defaults.executor, which we reference above in Specifying an Image.

expeditor:
  defaults:
    executor:
      docker:
        image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

steps:
  - command: make short-test
    expeditor:
      executor:
        linux:
  - command: make another-short-test
    expeditor:
      executor:
        docker:
  - command: make long-test
    expeditor:
      executor:
        docker:

The example below is logically equivalent to one above, except that the image_sha256 value is specified once instead of N times.

steps:
  - command: make short-test
    expeditor:
      executor:
        linux:
  - command: make another-short-test
    expeditor:
      executor:
        docker:
          image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9
  - command: make long-test
    expeditor:
      executor:
        docker:
          image_sha256: a7e1fa7fc3e1d9b91b6d025c5a099b9975c22be24b54275c4857ccb7bfdb89a9

For more information on executors, please read the documentation below

Global Accounts and Secrets

If there is an account or secret that you would like injected in every test, you can use the expeditor.defaults keyspace to do so.

Global Accounts

See above for more information on accounts.

When an account is specified in the expeditor.defaults.accounts keyspace, that account will automatically be associated with every step in the file.

---
expeditor:
  defaults:
    accounts:
      - github

steps:
  - command: make test1
  - command: make test2
  - command: make test3

If there are a few steps where do you don’t need or want an account that is specified in expeditor.defaults.accounts, you can reject it at the step level.

---
expeditor:
  defaults:
    accounts:
      - github

steps:
  - command: make test1
  - command: make test2
  - command: make test3
    expeditor:
      accounts:
        - !github

Global Secrets

If you would like to have an environment variable secret injected into every step in your pipeline definition, you can specify the secret in the expeditor.defaults.secrets keyspace.

---
expeditor:
  defaults:
    secrets:
      GITHUB_TOKEN:
        account: github
        field: token

steps:
  - command: make test1
  - command: make test2
  - command: make test3

Pipeline Protection

With the introduction of public Buildkite pipelines, it became necessary to ensure a level of security for our infrastructure. This document outlines some of the various protections we utilize.

Enforce Timeouts

By default, a default 10 minute timeout is enforced for all pipelines in both the public and private Buildkite organizations. You can override this value by specifying a value for timeout_in_minutes on the impacted step. There is a hard limit of 60 minutes. If you require more than 60 minutes, please reach out to Release Engineering.

Public vs Private Pipelines

Chef maintains two Buildkite organizations: one for public use and another for private use. Only Chef employees and other trusted contributors (here on out refered to as “registered contributors”) have access to the private Buildkite organization.

Limited Secrets Access

Pipelines in the public Buildkite organization do not have access to Vault. There is a GITHUB_TOKEN environment variable available to you with read-only access to public repositories should you need to avoid GitHub rate limiting.

Pull Requests and Private Pipelines

When a pull request is opened by a non-registered contributor on a project that has a verify pipeline in the private Buildkite organization, the execution of that pipeline will be blocked until a registered contributor reviews the pull requests and unblocks it in the Buildkite UI. Logs and output for private pipelines are not publically available, even if the project’s source code is.