Using Buildkite to build Omnibus Packages

Chef Software currently uses Omnibus to package up a number of our client and server software packages for distribution as traditional software binaries (e.g. .deb, .rpm, etc). These packages are built in Buildkite using a shim pipeline definition (e.g. .expeditor/release.omnibus.yml) which allows maintainers to easily define on which of our supported platforms they wish to build and/or test their product.

Creating your Omnibus pipelines

Buildkite Omnibus pipelines are an example of a named pipeline that requires the following configuration:

1. Two Omnibus Buildkite pipelines: omnibus/release and omnibus/adhoc.
2. A shim pipeline definition file named .expeditor/release.omnibus.yml (which is shared by both pipelines).
3. A valid Omnibus software definition.

We use the term “release” instead of “build” (as we do with other pipelines) to maintain continuity with our old Jenkins pipelines which were named “release” and to reinforce this pipeline is used to create releasable artifacts, rather than one-off builds. If you prefer, you can replace the usage of “release” with “build” (or just about anything else); Expeditor makes no hard-coded assumptions that your omnibus pipeline is named “release.”

As with other named pipelines, prefixing your pipeline name with omnibus/ informs Expeditor that this Buildkite pipeline is different than a general purpose pipeline and tells it to expect a shim pipeline definition rather than a traditional pipeline definition. Remember, shim pipeline definition files are not supported by the native Buildkite DSL.

---
pipelines:
  - omnibus/release:
      definition: .expeditor/release.omnibus.yml
  - omnibus/adhoc:
      definition: .expeditor/release.omnibus.yml
      env:
        - ADHOC: true

• The omnibus/release pipeline is your main release pipeline that, when triggered, builds omnibus artifacts that are promoted to the current channel when complete.
• The omnibus/adhoc pipeline, by setting the ADHOC environment variable to true, tells our automation to include a timestamp with the version and does not promote the build to the current channel (indicating it is not suitable for external consumption).

Artifact Publish Locations

Omnibus pipelines upload and make available the artifacts they create to the following locations.

1. You can override this behavior by setting PUBLISH_TO_ARTIFACTORY=true as an environment variable when you trigger an adhoc build.

Triggering release builds

There are three ways we recommend to trigger your omnibus/release pipeline.

1. Via the trigger_pipeline:omnibus/release action. The most common pattern is to trigger this action in response to a pull_request_merged workload. Projects that have Omnibus pipelines are also good candidates for staging areas as Omnibus builds are sometimes slow and are susceptible to build storms.

   subscriptions:
     - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
       actions:
         - ... # pre-commit actions like built_in:bump_version
         - trigger_pipeline:omnibus/release

2. Via the Buildkite UI. Triggering a release build via the Buildkite UI is useful when you need to trigger a fresh build out of band of a code change to your project.
3. Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the bk build create command.

   bk build create --pipeline=chef/chef-example-main-omnibus-release

Triggering adhoc builds

What makes an adhoc build different from a release build is the presence of the ADHOC environment variable, which we’ve configured to present in all our builds by default as part of pipelines configuration above.

There are three ways we recommend to trigger you omnibus/adhoc pipeline.

1. Via the trigger_pipeline:omnibus/adhoc action. The most common pattern is to trigger this action in response to an external workload. One common use case is triggering an adhoc build when an chef/omnibus-software definition is updated. Another common example is running the build on a schedule.

   subscriptions:
     - workload: pull_request_merged:chef/omnibus-software:master:*
       actions:
         - trigger_pipeline:omnibus/adhoc:
             only_if_modified:
               - config/software/*

2. Via the Buildkite UI. Triggering a release build via the Buildkite UI is useful when you need to trigger a fresh build out of band of a code change to your project.
3. Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the bk build create command.

   bk build create --pipeline=chef/chef-example-main-omnibus-adhoc

Building subset of platforms using OMNIBUS_BUILD_FILTER

By specifying a comma-separated list of globs in the OMNIBUS_BUILD_FILTER environment variable, you can build only a subset of the platforms you have specified without needing to modify your shim pipeline definition file.

---
# config omitted for brevity
builder-to-testers-map:
  el-6-x86_64:
    - el-6-x86_64
  el-7-x86_64:
    - el-7-x86_64
  mac_os_x-10.15-x86_64:
    - mac_os_x-10.15-x86_64
    - mac_os_x-11-x86_64
    - mac_os_x-12-x86_64
  mac_os_x-11-arm64:
    - mac_os_x-11-arm64
  sles-12-x86_64:
    - sles-12-x86_64
  ubuntu-16.04-x86_64:
    - ubuntu-16.04-x86_64
    - ubuntu-18.04-x86_64
  windows-2012r2-x86_64:
    - windows-2012-x86_64
    - windows-2012r2-x86_64
    - windows-2016-x86_64
    - windows-2019-x86_64

Building and/or testing subset of platforms using OMNIBUS_FILTER

By specifying a comma-separated list of globs in the OMNIBUS_FILTER environment variable, you can build and/or test only a subset of the platforms you have specified without needing to modify your shim pipeline definition file.

---
# config omitted for brevity
builder-to-testers-map:
  el-6-x86_64:
    - el-6-x86_64
  el-7-x86_64:
    - el-7-x86_64
  mac_os_x-10.15-x86_64:
    - mac_os_x-10.15-x86_64
    - mac_os_x-11-x86_64
  mac_os_x-11-arm64:
    - mac_os_x-11-arm64
  sles-12-x86_64:
    - sles-12-x86_64
  ubuntu-16.04-x86_64:
    - ubuntu-16.04-x86_64
    - ubuntu-18.04-x86_64
  windows-2012r2-x86_64:
    - windows-2012-x86_64
    - windows-2012r2-x86_64
    - windows-2016-x86_64
    - windows-2019-x86_64

Triggering a build with an expired cache

You may wish to trigger a build on your omnibus/release or omnibus/adhoc pipelines with an expired cached. This is handled by setting the EXPIRE_CACHE environment variable to true. Expiring the cache ensures that the entire Omnibus cache directory is deleted from the build systems. This can be especially useful when you’ve built the latest major version N of an omnibus project and now you need to clear the cache to successfully build the N-1 version of the project.

There is no way to pass an environment variable in as part of the trigger_pipeline action, and we do not recommend permanently configuring your pipelines to expire the cache on every build, so the only way to trigger a build with an expired cache is manually.

1. Via the Buildkite UI. You can specify the EXPIRE_CACHE=true environment variable when triggering a manual build via the Buildkite UI by clicking on the “Options” link in the “New Build” window.
2. Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the bk build create command.

   bk build create --pipeline=chef/chef-example-main-omnibus-adhoc --env="EXPIRE_CACHE=true"

Triggering a build with MAC_NOTARIZATION disabled

You may wish to trigger a build on your omnibus/release or omnibus/adhoc pipelines without notarizing the macOS package. This is handled by setting the MAC_NOTARIZE environment variable to false. By default expeditor cli will enable macOS Notarization, which means the package created during the build will be uploaded as a buildkite artifact which will be run through “Notarize macOS Package” phase where the artifact will be notarized from apple and the ticket will be stapled to the package before it is published to artifactory. Apple is going to make notarizing all build macOS packages mandatory starting February 3rd 2020. If your pipeline is not ready for notarizing the package during the build you may disable notarization for your pipeline by passing the environment variable “MAC_NOTARIZE” in your build settings and setting it to “false”

There is no way to pass an environment variable in as part of the trigger_pipeline action, and we do not recommend permanently configuring your pipelines to skip macOS notarization on every build, so the only way to trigger a build macOS without notarization manually. This value does not affect pipelines that do not contain macOS builds.

1. Via the Buildkite UI. You can specify the MAC_NOTARIZE=false environment variable when triggering a manual build via the Buildkite UI by clicking on the “Options” link in the “New Build” window.
2. Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the bk build create command.

   bk build create --pipeline=chef/chef-example-main-omnibus-adhoc --env="MAC_NOTARIZE=false"

Walk-through of what happens in the pipeline

Let’s break down the steps of an Omnibus Buildkite pipeline build and walk through all the processes.

1. Parse the shim pipeline definition. Our trigger step reads in the .expeditor/release.omnibus.yml shim pipeline definition file and performs the following sub-steps.
   1. Determine on which platforms we need to build. Expeditor reads through your builder-to-testers-map setting and determines on which platform it needs to build. It also looks at the fips-platforms setting to determine whether or not to build with FIPS compatibility.
   2. Determine on which platforms we need to test. While reading through your builder-to-testers-map setting, Expeditor also determines on which platforms certain builds should be tested. It configures the jobs accordingly, using the test-path or test-path-windows setting for each job as appropriate.
   3. Determine whether or not to promote to current at the end. Based on whether or not the ADHOC environment variable is set to true, Expeditor will tack a final step which will promote all the builds to the current channel in Artifactory.
2. Build all the artifacts and publish them to the unstable channel. An individual Buildkite job is created for each Omnibus build so that all the builds can occur in parallel. If the Omnibus build succeeds, the resulting artifact is published to the unstable channel of Chef’s internal Artifactory.
3. Create the Build Record. Upon the completion of all the builds, a build record representing all the artifacts is created inside Artifactory.
4. Test all of the artifacts. Once the build record has been completed it is time to test the artifacts using the test script specified in either test-path or test-path-windows. Each test platform downloads the appropriate artifact (as determined by our mixlib-install logic) and tested.
5. Promote to the current channel (release only). Once all of the tests scripts have passed successfully, our builds are promoted to the current channel, where they can be consumed by early adopters and other beta/QA testers.

release.omnibus.yml

The release.omnibus.yml file is a shim pipeline definition file with a particular schema that informs the pipeline how to build, test, and upload to Artifactory an Omnibus package.

---
project-name: <PROJECT_NAME>
config: omnibus/omnibus.rb
test-path: omnibus/omnibus-test.sh
test-path-windows: omnibus/omnibus-test.ps1
manual-promote-release: true
skip-artifactory-platforms:
  - sles-12-x86_64
  - el-*-x86_64
fips-platforms:
  - el-*-x86_64
  - windows-*
builder-to-testers-map:
  el-6-x86_64:
    - el-6-x86_64
  el-7-x86_64:
    - el-7-x86_64
  mac_os_x-10.15-x86_64:
    - mac_os_x-10.15-x86_64
    - mac_os_x-11-x86_64
  sles-12-x86_64:
    - sles-12-x86_64
  ubuntu-16.04-x86_64:
    - ubuntu-16.04-x86_64
    - ubuntu-18.04-x86_64
  windows-2012r2-x86_64:
    - windows-2008r2-x86_64
    - windows-2012-x86_64
    - windows-2012r2-x86_64
    - windows-2016-x86_64
    - windows-2019-x86_64

Placeholder Definitions

PROJECT_NAME

The name of the project as defined in our internal Artifactory instance: typically the same as the product_key with few exceptions.

build-options

Additional parameters to pass into the omnibus CLI at build time.

builder-to-testers-map

A hash that maps each builder platform used for building a package to its corresponding tester platforms.

You can find names of the currently supported platforms in omnibus-toolchain’s release.omnibus.yml file.

config

The relative path to the omnibus configuration file in the local repository. Defaults to omnibus.rb.

fips-platforms

A list of platforms that should have OMNIBUS_FIPS_MODE environment variable set to true. This list can include glob patterns such as el-*-x86_64.

install-dir

Install path of the omnibus project. Defaults to /opt/<project-name>.

install-dir-windows

Install path of the omnibus project on a Windows platform. Defaults to C:/opscode/<project-name>.

project-name

The name of the omnibus project.

skip-artifactory-platforms

A list of platforms that will not upload to artifactory and will not fail the entire pipeline if their specific step fails. Can be used to target multiple platforms with the * character such as el-*-x86_64. This will match BOTH builders and testers.

test-channel

The channel from which to download the omnibus package for testing. Defaults to unstable. If set, we skip the build step.

test-dep-channel

The channel from which to download dependent omnibus packages for testing. Defaults to current.

test-path

The relative path to the script used to test the omnibus package. If unset, we skip the testing step.

test-path-windows

The relative path to the Windows batch or PowerShell script used to test the omnibus package on a Windows platform. If unset, we skip the testing step for Windows.

manual-promote-release

Optional. Defaults to false if not specified. The buildkite step changes from a wait step to a block step. This means that someone will have to click a button to proceed to the promote. This is useful if the need to bypass a complete successful build in all OS’s is needed.

test-version

The omnibus package version to be downloaded for testing. Defaults to the value stored in Buildkite metadata by the build stage. If unset, we skip the build step.

windows-64-msystem

The msys2 toolchain environment to use when building 64 bit Windows packages. Acceptable values are MINGW64 and UCRT64. Defaults to MINGW64.