Using Buildkite to build Habitat Packages

Expeditor leverages Buildkite pipelines to build Chef Habitat packages. It does this because the Chef Habitat Build Service does not currently support notifications for external services.

Getting Started

To get started building Chef Habitat packages, you’ll need to do the following:

  1. Add the necessary settings, subscriptions, and actions to your .expeditor/config.yml
  2. Add a .bldr.toml with the necessary settings
  3. Add the pipeline definition file (.expeditor/build.habitat.yml) with the necessary settings.
.expeditor/config.yml
---
pipelines:
  - habitat/build:
      definition: .expeditor/build.habitat.yml

promote:
  channels:
    - unstable
    - dev
    - stable

subscriptions:
  - workload: pull_request_merged:{{agent_id}}:*
    actions:
      - trigger_pipeline:habitat/build
  - workloads: project_promoted:{{agent_id}}:*
    actions:
      - built_in:promote_habitat_packages
.bldr.toml
[your_package_name]
... your settings

Click Here for more details about the configuration for you .bldr.toml file.

.expeditor/build.habitat.yml
---
origin: chef
smart_build: true

Click Here for more details about the configuration for your .expeditor/build.habitat.yml file.

Usage

In Expeditor, naming a pipeline habitat/* indicates that the pipeline is used to build Chef Habitat packages. Like all Buildkite pipelines that Expeditor creates, the first step is a trigger step that reads in the pipeline definition file and populates the remainder of the pipeline. In the case of Chef Habitat pipelines, this trigger knows how to read the particular schema used for the .expeditor/build.habitat.yml files.

Populating the Job

The first thing that the trigger step does is process the .bldr.toml file (if one exists), the same way that the Chef Habitat Build Service does (with a few additions, which are covered here). If a .bldr.toml file does not exist then Expeditor falls back to default behavior. It assumes that there is only one package defined in the habitat/ directory that should always get built with no exports.

The trigger step processes the .bldr.toml file and identifies which packages it needs to build, the order in which to build them, and whether or not to export the packages as Docker images. Once it has calculated all of the individual steps it needs to take, it populates the remainder of the pipeline with those steps.

Building the packages

The first thing the pipeline does is build all the packages. It does this in parallel when possible by creating build groups that respect intra-project pkg_deps and pkg_build_deps to ensure that we always build against the correct version of packages. To ensure we build packages against the appropriate dependencies, we leverage a unique HAB_BLDR_CHANNEL: expeditor-<ORG>-<REPO>-<BRANCH>.

<ORG> and <REPO> are your GitHub organization and repository name. <BRANCH> is the name of your release branch (typically master).

After we finish building the packages, we export any of the packages you’ve specified in your .bldr.toml.

The final step sends a buildkite_hab_build_group event back to Expeditor to let it know that the builds have completed successfully.

.bldr.toml

The basis of the configuration for Expeditor’s Habitat builds is Chef Habitat’s .bldr.toml. Please see Habitat’s documentation for more information on .bldr.toml.

Custom Settings

Expeditor recognizes the following package settings in your .bldr.toml.

The Habitat Build Service will not honor these settings.
export_targets

A list of hab pkg export actions that you wish to take once the habitat packages have completed building. Defaults to []. Currently only supports docker.

[my-package]
export_targets = ["docker"]
private

Whether or not the package should be visible on the public Depot. Defaults to false.

[my-package]
private = true

.expeditor/build.habitat.yml

Where Habitat’s Builder service stores build configuration within the UI, Expeditor stores the same information in the pipeline definition file (.expeditor/build.habitat.yml).

bldr_toml

Provide the path to your .bldr.toml, if you have one. If no value is specified, we assume .bldr.toml. If we cannot find the file, we proceed with the assumption that you have either a ./plan.sh or ./habitat/plan.sh file.

In 99% of cases, you are safe leaving this value unspecified.
---
bldr_toml: .bldr.toml

origin

Provide the name of the origin where you would like the packages in your repository uploaded.

---
origin: chef

studio_secrets

You have access to the Secrets DSL via the studio_secrets setting. For simplicity, Expeditor will automatically prefix all your secrets with the requisite HAB_STUDIO_SECRET_.

---
studio_secrets:
 FEAT_IGNORE_LOCAL:
 value: "true"
 GITHUB_TOKEN:
 account: github/chef-ci
 field: token

smart_build

When smart_build is set to true in your .expeditor/build.habitat.yml, Expeditor will—when a pull request is merged—only rebuilds modified packages.

---
smart_build: true

To do this Expeditor maintains git tags in your repository corresponding with the last commit against which each package was last successfully built. The naming scheme for these tags is hab-pkg-<PKG_NAME>. For builds triggered by the merging of a pull request, Expeditor performs a git diff between the HEAD of your release branch and the commit associated with the last successful build for each package. It compares the results of this diff against the paths specified in the .bldr.toml for that package and—if it finds matching files—marks that package and all of its reverse dependencies (rdeps) in the repository as “needing to be built.”

It also ensures all the modified packages, and their local reverse dependencies, are rebuilt in the correct order.

If you have enabled smart builds, and trigger a build manually through the Buildkite UI, Expeditor rebuilds all packages in the repository, updating the corresponding git tags as a result.