Project Promotion

At the core of promotion inside Expeditor is the concept of channels. Channels in Expeditor emulate the software release channels with which you may already be familiar. What channels your project has is controlled by the artifact_channels configuration setting in your .expeditor/config.yml configuration file.

When you promote a project within Expeditor, you are indicating that a version of your project should be moved to its next available channel. When Expeditor was first written, its promotion logic was migrated from another utility that operated under the assumption that you were promoting only once from the current channel to the stable channel. As such, the only identifying elements required to trigger a promotion were the package name (now replaced with the agent ID or alias) and the version number. This interface was kept to ensure a smooth transition for people from this old utility to Expeditor.

/expeditor promote AGENT_ID_OR_ALIAS VERSION_REFERENCE

However, the introduction of Chef Habitat and its undefined channel structure introduced the need for a new promotion paradigm. Projects like chef/automate — that do not ship as an omnibus package and do not follow the standard unstable/current/stable channel paradigm — broke the assumptions that Expeditor made about how projects were promoted. In response, the existing Expeditor promotion logic was tweaked slightly to allow a wider variety of promotions.

Today, all Expeditor projects fall into one of two categories:

1. Projects that are promoted once using a semantic version
2. Projects that are promoted more than once using a source channel, also referred to as “head of channel” promotion

How many channels a project has does not impact which of the two categories the project falls under; all that matters is how many times you need to run the /expeditor promote Slack command before your project reaches it’s final channel (typically stable).

Promoting projects by semantic version

While it is not enforced by Expeditor, the assumption it will make when you promote a project using a semantic version is that you are promoting that project to its final channel, most commonly the stable channel. To use this pattern, you must select a primary artifact type through which all your other promotion actions are funneled. Common examples of primary artifact types include an Omnibus package, a RubyGem, and in certain cases a Habitat package.

Omnibus projects

Most Omnibus projects are software projects defined in Mixlib::Install that existed prior to the creation of more modern tooling such as Docker or Habitat. To tell Expeditor that your project is an Omnibus project, you’ll need to configure the product_key configuration variable in the .expeditor/config.yml file. Since Expeditor’s promotion logic was taken from the utility initially responsible for promotion Omnibus packages, Omnibus projects are the most straight-forward to document.

Omnibus projects traditionally have three channels: unstable, current, and stable. As part of the omnibus/build pipeline, packages are built and published to the unstable channel and — after passing some automated testing — automatically promoted to the current channel. The final promotion, from current to stable, is handled by the built_in:promote_artifactory_artifact in response to the project_promoted workload.

# product key from https://github.com/chef/mixlib-install/blob/master/PRODUCT_MATRIX.md
product_key: PRODUCT_KEY

pipelines:
  - omnibus/release

artifact_channels:
  - unstable
  - current
  - stable

subscriptions:
  # other subscriptions omitted for brevity..

  # Trigger a build on the Omnibus pipeline when a pull request is merged
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - ... # various pre-commit actions
      - trigger_pipeline:omnibus/release
      - ... # various other post-commit actions

  # Perform operations when the Omnibus package is published to the current channel
  - workload: artifact_published:current:PRODUCT_KEY:{{version_constraint}}
    actions:
      - ... # optional actions such as building docker images

  # Subscribe to the promotion from the current channel to stable
  - workload: project_promoted:{{agent_id}}:*
    actions:
      - built_in:promote_artifactory_artifact

  # Trigger the bulk of the promotion activity once the artifact has been successfully promoted
  - workload: artifact_published:stable:PRODUCT_KEY:{{version_constraint}}
    actions:
      - ... # other actions such as built_in:rollover_changelog

If your omnibus project also provides alternate artifact formats such as a Ruby gem, Chef Habitat Package, or Docker image, we suggest coordinating activities for those artifacts based on the life-cycle of the “primary” omnibus package via the artifact_published workload.

Ruby gem projects

Projects which are stand-alone Ruby gems (e.g. chef/mixlib-install) can be promoted by version because they only need to be promoted once. The RubyGems API has no the concept of channels so Expeditor has faked them by managing two separate RubyGems repositories:

• A dedicated RubyGems repository on Chef’s internal Artifactory, which is treated as the unstable channel
• RubyGems.org, which is treated as the stable channel

As part of the ruby gems build process the gem is automatically published to our internal Artifactory RubyGems repository. In response to the project_promoted workload, the built_in:publish_rubygems action can be executed which will download the specified version of the gem and re-upload it to the rubygems.org.

artifact_channels:
  - unstable
  - stable

rubygems:
  - YOUR_GEM

subscriptions:
  # other subscriptions omitted for brevity..

  # Build the ruby gem and publish it to Artifactory when a pull request is merged
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - ... # various pre-commit actions
      - built_in:build_gem
      - ... # various other post-commit actions

  # Subscribe to the promotion from the unstable channel to stable
  - workload: project_promoted:{{agent_id}}:*
    actions:
      - built_in:publish_rubygems

  # Trigger the rest of the promotion activities once the artifact has been successfully promoted
  - workload: ruby_gem_published:YOUR_GEM:{{version_constraint}}
    actions:
      - ... # other actions such as built_in:rollover_changelog

Single-package Habitat projects

More often than not you’ll likely be managing your Habitat packages as a downstream of another artifact type (e.g. building a Habitat package for your project whose primary artifact type is an Omnibus package). However, if your project is configured correctly it is possible to handle the promotion of a standalone Habitat package using its semantic version.

There are, however, a few limitations:

1. Your project can only contain a single package definition. This pattern only works if there is a single Habitat package your project is managing. Having multiple package targets (e.g. x86_64-linux and x86_64-windows) is fine.
2. You should manage your version using Expeditor’s version management. While it may work without it, this pattern isn’t technically supported unless you’re using it.
3. Your Habitat package should be build using Expeditor’s Chef Habitat build pipelines. The Chef Habitat build service does not support webhooks. Expeditor’s Habitat support only works if builds and promotions are handled by Expeditor.

As part of Expeditor’s Chef Habitat build pipeline, the Chef Habitat package(s) are automatically published to the unstable channel of the public Chef Habitat Builder. In response to the project_promoted workload, you’ll want to run the built_in:promote_habitat_package action to publish the specified version to the next channel (which given the configuration would be stable). Chef Habitat packages that are promoted using Expeditor actions trigger the hab_package_published workload. You can subscribe to this workload to complete the remainder of your promotion activities.

artifact_channels:
  - unstable
  - stable

pipelines:
  - habitat/build

subscriptions:
  # other subscriptions omitted for brevity..

  # Trigger the build of the Chef Habitat package when a pull request is merged
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - ... # various pre-commit actions
      - trigger_pipeline:habitat/build
      - ... # various other post-commit actions

  # Subscribe to the promotion from the unstable channel to stable
  - workload: project_promoted:{{agent_id}}:*
    actions:
      - built_in:promote_habitat_packages

  # Trigger the rest of your promotion activities once the artifact has been successfully promoted
  - workload: hab_package_published:stable:PKG_ORIGIN/PKG_NAME/{{version_constraint}}
    actions:
      - ... # other actions such as built_in:rollover_changelog

Promoting projects by “head of channel”

With the current implementation of promotion in Expeditor, if you need to promote your project more than once than you’ll need to promote it by “head of channel.” This pattern is a workaround that was introduced to allow for promotion of projects like chef/automate without breaking backwards compatibility with the existing implementation of the project_promoted workload.

A common element of a project that indicates a clear decision to use “head of channel” promotion is the lack of a single primary artifact. Usually projects that require the use of the “head of channel” approach have multiple artifacts, usually consisting of a combination of compiled CLIs and Habitat packages, that need to be shipped together. Two examples of projects that use “head of channel” promotion are habitat-sh/habitat and chef/automate.

To promote a project that is promoted by “head of channel,” rather than specifying a version in your promote command you specify the channel from which you want to promote.

/expeditor promote AGENT_ID_OR_ALIAS SOURCE_CHANNEL

Expeditor has no concept as to whether or not you’re promoting a channel or version. The identifying marker in the workload is the “thing” you want to promote, referred to internally as the “promotable.” This is where most people get tripped up with this pattern. When thinking about channels, most people think in terms of the target channel (i.e. “I want to promote to stable.") Because “head of channel” is a workaround, the name of the channel you’re specifying in your promotion command is the channel you’re promoting from, or the source channel. If you need to use the “head of channel” promotion pattern, its important that you internalize this. Otherwise, you may end up promoting the wrong channel.

When it comes to designing your release automation, there are a couple things you’ll need to remember about “head of channel” projects:

1. While Expeditor knows the order of your channels, it does not know how to move things from one channel to the next. Projects that use the “head of channel” promotion method typically don’t have a single artifact that Expeditor is able to understand how to promote. Most of your promotion actions will therefore be bash actions that understand the following:
   1. What artifacts need to be promoted and where they are stored.
   2. How to promote those artifacts from the source channel to the target channel.
2. Many built-ins designed to work with semantically versioned projects will not work with “head of channel” projects. We will indicate whether or not a particular action is supported for use in “head of channel” promotions in our actions reference documentation.
3. There are no primary artifacts with “head of channel” projects. Most “head of channel” projects don’t have a single promotable artifact that Expeditor can recognize. Unlike other projects, which chain their promotion actions around the successful promotion of the primary artifact, “head of channel” projects execute all their promotion actions in response to the project_promoted workload.

# talk with Release Engineering about which channels are right for you
artifact_channels:
  - dev
  - acceptance
  - stable

pipelines:
  - habitat/build

subscriptions:
  # other subscriptions omitted for brevity..

  # Trigger the build of the Chef Habitat packages when a pull request is merged
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - ... # various pre-commit actions
      - trigger_pipeline:habitat/build
      - ... # various other post-commit actions

  # Automatically promote the Habitat packages from unstable to dev upon successful build
  - workload: buildkite_hab_build_group_published:{{agent_id}}:*
    actions:
      - built_in:promote_habitat_packages

  # Subscribe to the promotion of the dev channel to acceptance
  - workload: project_promoted:{{agent_id}}:dev:*
    actions:
      - built_in:promote_habitat_packages
      - ... # other promotion actions

  # Subscribe to the promotion of the acceptance channel to stable
  - workload: project_promoted:{{agent_id}}:acceptance:*
    actions:
      - built_in:promote_habitat_packages
      - ... # other promotion actions