Buildkite Pipelines

While Expeditor has the ability to execute tasks using actions, there are some tasks that should be run in Buildkite pipelines instead. There are a variety of reasons for this delineation, a number of which we’ll go over in a moment, but the main reason boils down to complexity. Expeditor actions should be used to run short, simple tasks while longer, more complex tasks should be run in Buildkite pipelines.

Expeditor agents only run one action set at a time, so action sets that take a long time to execute can clog up the system. For example, if you have a very long-running bash script that runs in response to a pull_request_merged workload, your agent will be blocked from running any other action sets.

The recommended pattern (covered in more detail in our general purpose pipeline documentation) is to:

1. Extract these long-running processes into Buildkite pipelines
2. Use the trigger_pipeline action to trigger the pipeline
3. React to the success or failure of these pipelines by subscribing to the associated buildkite_build workload (e.g. buildkite_build_passed).

Public vs Private Pipelines

At Chef, we use three Buildkite organizations to manage all our CI/CD pipelines:

• chef, which is used to manage all our private pipelines
• chef-canary, which is used to manage all our private canary pipelines
• chef-oss, which is used to manage all our public pipelines

The chef-oss organization dashboard is publicly viewable, making it great for verify pipelines for open source projects. The chef organization dashboard is accessible only to Chef employees via our Okta integration, and is the default location for all external pipelines. The chef-canary organization dashboard is accessible only to Chef employees via our Okta integration, and is part of the CI pipeline for our Buildkite queues. Please reach out to helpdesk.chef.io if you require access to the chef Buildkite organization.

Using Expeditor to create the pipelines

Expeditor’s configuration file is the source of truth for all our release processes. As such, we use Expeditor to manage the life-cycle of the Buildkite pipelines in our chef, chef-canary and chef-oss organizations. For guidance on how to create new Buildkite pipelines using Expeditor, check out one of the following guides:

• General purpose pipelines
• GitHub pull request verify pipelines
• Chef Habitat build pipelines
• Omnibus build pipelines
• Docker build pipelines

One of the reasons Chef initially chose Buildkite was its ability to dynamically define the steps in a pipeline. The Buildkite pipelines that are created using Expeditor are automatically configured with a pipeline trigger step that allows us to layer in additional functionality on top of the functionality already provided by Buildkite. The end result of this process is that we’re able to create a consistent execution environment across all our pipelines as well as provide helpers and shortcuts that make the Buildkite pipeline YAML definition files easier to manage.

Best Practices & Naming Conventions

• Use snake_case for pipeline names. This allows them to be cohesively used in variety of situations.

  ---
  pipelines:
    - my_pipeline   # ok
    - my-pipeline   # bad
    - my pipeline   # bad
    - myPipeline    # bad

• Keep your pipeline definition files in the .expeditor/ directory. This keeps all your pipelines together and creates an implicit association of those pipelines with Expeditor.

  .
  ├── .buildkite/
  │   ├── my_pipeline.pipeline.yml      # bad
  ├── .expeditor/
  │   ├── buildkite/
  │   │   └── my_pipeline.pipeline.yml  # bad
  │   ├── my_pipeline.pipeline.yml      # ok

• Use the Expeditor Buildkite DSL. This ensures that your pipelines will be able to automatically take advantage to improvements to the underlying Buildkite infrastructure.

Why use Buildkite when we could use X?

There are a number of alternatives to Buildkite when it comes to CI/CD pipelines. As such, there are inevitably going to be situations where Release Engineering is going to recommend we do something in Buildkite that we could do somewhere else, such as:

• Using the Habitat Build Service to build Chef Habitat package builds
• Using Docker Hub’s automated builds to build Docker images
• Using TravisCI, AppVeyor, or Azure DevOps for GitHub pull request verification pipelines

These are all good services — one of which is a Chef product. So why do we recommend using Buildkite?

• Visibility. With everyone using the same tools, we get increased visibility into how teams are using pipelines. This makes it easier to identify usage patterns that we can extract into re-usable plugins.
• Consistency. The push towards re-usable plugins, along with the visibility, helps ensure a measure of consistency across the hundreds of pipelines that we manage. It also makes it easier to understand where a certain process might occur. A release process either happens in an Expeditor action set, or is executed as part of a Buildkite pipeline.
• Functionality. Buildkite provides a wonderful execution ecosystem that allows us to layer on functionality where it is missing from their core service. Often times we are able to roll out functionality to our Buildkite pipelines before it becomes available on the public service. For example, teams building their Docker images in our pipelines are able to build Windows Docker images — functionality that is not currently available in Docker’s official automated builds service.
• Security. Secrets management is already a complex problem, but secrets management across an indeterminate number of third party services is even more complex. By using private Buildkite pipelines, you’ll get access to our Hashicorp Vault instance which has pretty much any secret you would need. By consolidating all our secrets in Vault, we’re able to improve our security footprint by dynamically generate temporary credentials and more easily rotating static credentials.

Pipeline Image Groups

The Release Engineering team categorizes the functionality provided inside Buildkite pipelines into image groups. These image groups are then assigned to the various queues which power our pipelines. Our goal is to ensure that the functionality available inside these image groups remains as consistent as possible.

Core

The core pipeline image group is what powers the executors used in our general purpose and verify pipelines. This group is currently comprised of default Buildkite AMIs provided by Buildkite’s Elastic CI Stack for AWS. As such, they do not have any additional software languages or tooling installed.

Core Docker

The core Docker image group include the chefes/buildkite and chefes/buildkite-windows Docker images. These are the default Docker images used by the docker executor.

Chef Habitat Build

The Chef Habitat Build image group is used to power our Chef Habitat Buildkite pipelines. They are specially built AWS images intended explicitly for the purpose of building Chef Habitat packages.

Omnibus Build

The Omnibus Build image group is used to power the Omnibus pipelines. They are specially built AWS images intended explicitly for the purpose of building and testing Omnibus package.