Using Buildkite to build Omnibus Packages

Chef Software currently uses Omnibus to package up a number of our client and server software packages for distribution.

Getting Started

To get started building Omnibus packages for your project, you’ll need to do the following:

  1. Add the necessary settings, subscriptions, and actions to your .expeditor/config.yml
  2. Add the pipeline definition file (.expeditor/release.omnibus.yml) with the necessary settings.
.expeditor/config.yml
---
pipelines:
  - omnibus/release     # The value of "definition" defaults to the pipeline name minus "omnibus/" (e.g. .expeditor/release.omnibus.yml).
  - omnibus/adhoc:
      definition: .expeditor/release.omnibus.yml
      env:
        - ADHOC: true

product_key: <PRODUCT_KEY>

promote:
  channels:
    - unstable
    - current
    - stable

subscriptions:
  - workload: pull_request_merged:{{agent_id}}:*
    actions:
      - trigger_pipeline:omnibus/release
  - workloads: project_promoted:{{agent_id}}:*
    actions:
      - built_in:promote_artifactory_artifact
  # Subscription to workload action of pull request merge to master branch on
  # omnibus-software project will allow us to trigger an uncached omnibus build
  # for chef-server when the omnibus-software has modified software config files.
  - workload: pull_request_merged:chef/omnibus-software:master:*
    actions:
      - trigger_pipeline:omnibus/adhoc:
          only_if_modified:
            - config/software/*
.expeditor/release.omnibus.yml
---
project-name: <PROJECT_NAME>
config: omnibus/omnibus.rb
test-path: omnibus/omnibus-test.sh
test-path-windows: omnibus/omnibus-test.ps1
fips-platforms:
  - el-*-x86_64
  - windows-*
builder-to-testers-map:
  el-6-x86_64:
    - el-6-x86_64
  el-7-x86_64:
    - el-7-x86_64
  mac_os_x-10.12-x86_64:
    - mac_os_x-10.12-x86_64
    - mac_os_x-10.13-x86_64
    - mac_os_x-10.14-x86_64
  sles-11-x86_64:
    - sles-11-x86_64
  sles-12-x86_64:
    - sles-12-x86_64
  ubuntu-16.04-x86_64:
    - ubuntu-16.04-x86_64
    - ubuntu-18.04-x86_64
  windows-2012r2-x86_64:
    - windows-2008r2-x86_64
    - windows-2012-x86_64
    - windows-2012r2-x86_64
    - windows-2016-x86_64
    - windows-2019-x86_64

Click Here for more details about the configuration for your .expeditor/release.omnibus.yml file.

Placeholder Definitions

PRODUCT_KEY
The appropriate value from the PRODUCT_MATRIX
PROJECT_NAME
The name of the project as defined in our internal Artifactory instance: typically the same as the PRODUCT_KEY with few exceptions.

Usage

In Expeditor, naming a pipeline omnibus/* indicates to Expeditor that the pipeline is used to build omnibus packages. Like all Buildkite pipelines that Expeditor creates, the first step is a trigger step that reads in the pipeline definition file and populates the remainder of the pipeline. In the case of omnibus pipelines, this trigger knows how to read the particular schema used for the .expeditor/release.omnibus.yml files.

It is not uncommon for projects to have at least two omnibus pipelines: release and adhoc. The release pipeline is used to build the releases that may eventually be consumed by customers, while adhoc builds are used to test the omnibus build process itself. Developers frequently run adhoc builds when attempting to debug the build process itself. Some teams that manage projects that are no longer under active development may also choose to run adhoc builds on a regular cadence to track the health of the build.

The difference between the release and adhoc pipelines is the ADHOC=true environment variable passed into the pipeline. This variable tells our automation to include a timestamp with the version and does not promote the build to the current channel (indicating it is not suitable for external consumption).

Advanced usage of omnibus pipelines can also incorporate more than one pipeline definition file. For example, you could have .expeditor/release.omnibus.yml as your “main” build file, but also have .expeditor/linux.omnibus.yml which only has Linux platforms defined. This document, however, only covers the use case where you have a single definition file: .expeditor/release.omnibus.yml.

Optional Pipeline Environment Variables

ADHOC - Setting this to true tells our automation to include a timestamp with the version and does not promote the build to the current channel (indicating it is not suitable for external consumption)

EXPIRE_CACHE - Setting this to true ensures that the entire Omnibus cache directory is deleted from the build systems. This can be especially useful when you’ve built the latest major version N of an omnibus project and now you need to clear the cache to successfully build the N-1 version of the project.

release.omnibus.yml

The release.omnibus.yml file is a pipeline definition file with a particular schema that informs the pipeline how to build, test, and upload to Artifactory an omnibus package.

build-options

Additional omnibus build options.

builder-to-testers-map

A hash that maps each builder platform used for building a package to its corresponding tester platforms.

You can find names of the currently supported platforms in omnibus-toolchain’s release.omnibus.yml file.

config

The relative path to the omnibus configuration file in the local repository. Defaults to omnibus.rb.

fips-platforms

A list of platforms that should have OMNIBUS_FIPS_MODE environment variable set to true. This list can include glob patterns such as el-*-x86_64.

install-dir

Install path of the omnibus project. Defaults to /opt/<project-name>.

install-dir-windows

Install path of the omnibus project on a Windows platform. Defaults to C:/opscode/<project-name>.

project-name

The name of the omnibus project.

test-channel

The channel from which to download the omnibus package for testing. Defaults to unstable. If set, we skip the build step.

test-dep-channel

The channel from which to download dependent omnibus packages for testing. Defaults to current.

test-path

The relative path to the script used to test the omnibus package. If unset, we skip the testing step.

test-path-windows

The relative path to the Windows batch or PowerShell script used to test the omnibus package on a Windows platform. If unset, we skip the testing step for Windows.

test-version

The omnibus package version to be downloaded for testing. Defaults to the value stored in Buildkite metadata by the build stage. If unset, we skip the build step.