Workloads & Subscriptions

Expeditor uses the concept of subscriptions to allow us to express what actions we want to take in response to certain events in terms of “when, then.”

• When a GitHub pull request is merged, then bump the version of our software, update the change log, and trigger a new artifact build.
• When an artifact is successfully uploaded to Artifactory, then trigger a job in our deploy pipeline to deploy that artifact to our dev/test environment.
• When a new version of an upstream dependency is released, then trigger a new build of our software that automatically pulls in that new version.

This design differs from the more traditional pipeline models used by tools like Jenkins in that it is designed to support workflows with large numbers of individual, yet highly interconnected, software projects — all of which likely have one or more release life-cycle dependencies on one another. For example:

1. When we release a new version of Ohai, we need to update the version pinning in Chef Infra Client.
2. When we release a new version of Chef Infra Client, we need to update the pinnings in Chef Workstation and Chef Infra Server.
3. When we release a new version of InSpec, we need to update the pinnings in Chef Workstation and Chef Automate.

If you look at the dependency graph of all of Chef Software’s software projects, down to utility libraries like mixlib-shellout, it is a complex web worthy of Shelob herself. Trying to understand the steps required to build, test, and release all of these software projects necessitated the creation of several release documents. These word documents lived in Google Drive, quickly fell out of date, and like the Isla de Muerta, could not be found except by those who already knew where they were. This is to say nothing of managing the variety of permissions and accesses that allowed engineers to publish these associated release artifacts to wherever they needed to go. With Expeditor subscriptions, all of this release information is kept (versioned) with the relevant code.

Composition of a Subscription

Now that we have discussed the motivation behind subscriptions, let’s take a look at an example. To reiterate, a subscription is the codification of the concept of “when, then.” Your “when” is captured by workload, and the “then” is represented by a list of one or more actions.

subscriptions:
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - noop:action_one
      - noop:action_two

We’ll cover the “then” portion of subscriptions when we talk about action sets. For right now, let’s focus on the “when” component of our subscription: the workload.

Workloads are pieces of identifying information joined by colons. You can read pull_request_merged:{{github_repo}}:{{release_branch}}:* as “when any pull request is merged on my project.” We say project here, even though {{agent_id}} is in the workload, because the action is being taken against the GitHub repository and not the Expeditor agent. We’ll discuss the use of {{agent_id}} more in a moment.

The first piece in our example, pull_request_merged, is referred to as the event. The types of actions you can take depend on the event and are documented here, along with all of the workloads and events we support.

The second piece, {{agent_id}}:*, is referred to as the event ID glob. In this section, {{agent_id}} is a subscription template variable used to stand in for the full agent ID while * is a glob that will match on any remaining identifying data.

There are a number of ways we could have written our pull_request_merged workload subscription based on the available subscription template variables. The way we’ve chosen — using {{agent_id}} — is simply the most flexible and concise. Assuming this subscription was for the main branch of the fictional chef/example GitHub repository (which would have the agent ID chef/example:main) all of the following examples are ways we could subscribe to “when any pull request is merged on my project” using subscription template variables.

• pull_request_merged:chef/example:main:*
• pull_request_merged:{{github_repo}}:{{github_branch}}:*
• pull_request_merged:{{agent_id}}*

Subscriptions are Global

The power of the subscriptions keyword is that you can subscribe to any workload published within Expeditor — even workloads not associated with your project or agent. Let’s take a look at an expansion of our example from above.

subscriptions:
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - noop:action_one
      - noop:action_two
  - workload: pull_request_merged:inspec/example:main:*
    actions:
      - noop:action_three

We now have two “when, then” subscriptions:

1. When any pull request is merged into our release branch (the main branch of our fictional chef/example GitHub repository), then execute actions noop:action_one and noop:action_two.
2. When any pull request is merged into the main branch of (the fictional) inspec/example GitHub repository, then execute the noop:action_three action.

The “when, then” model of Expeditor really begins to shine when we start using of cross-project subscriptions. Cross-project subscriptions allow us to chain together release activities that span multiple projects. A project that utilizes this pattern heavily is Chef Workstation, which needs to keep track of pinnings of a number of different other software projects. If you take a look at their .expeditor/config.yml, you’ll see a number of subscriptions that look something like this.

subscriptions:
  - workload: chef/another_example:main_completed:pull_request_merged:chef/another_example:main:*
    actions:
      - bash:.expeditor/update_chef_another_example_to_latest.sh

There are a two things in this example subscription that we have not touched on yet: the use of completed workloads and the bash action. Those items are both covered in more detail in their respective sections, but let’s take a moment to dissect what this subscription is saying using our “when, then” model:

When the chef/another_example:main Expeditor agent has successfully processed the pull_request_merged:chef/another_example:main:* workload, then I would like my agent (chef/example:main) to run the .expeditor/update-chef_another_example_to_latest.sh script.