Staging Areas

Given the highly asynchronous nature of many release life-cycle activities you can quickly run into a problem with race conditions. This is especially true with projects with a high release velocity, where processes are being triggered faster than they can complete. To address this issue Expeditor has the concept of staging areas. A staging area is a place where workloads, rather than being immediately acted upon, are staged until they are ready to be worked on.

Let’s say you want to trigger a pipeline when a pull request is merged. If we write this in our “when, then” format, it would look something like this:

• When a pull request is merged, then trigger my pipeline.

There are several situations in which this configuration might warrant the use of staging areas. Maybe you only want one build of your pipeline running at a time, or you want to minimize the number of times you trigger the pipeline to prevent “build storms” that can clog up your pipeline for hours. Either way, by introducing the concept of staging areas, you are splitting your original “when, then” statements into two new statements.

• When a pull request is merged, then add that workload to my staging area.
• When my pull request workload is released from my staging area, then trigger my pipeline.

Adding a workload to the staging area

As alluded to above, a staging area is mapped one-to-one with a workload subscription.

staging_areas:
  - post_merge:
      workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*

By specifying that a workload is associated with a staging area, you are telling Expeditor that when your agent receives that workload it should immediately be placed in the staging area, making it unavailable for normal subscriptions. This means that if you create a staging area for pull_request_merged:{{github_repo}}:{{release_branch}}:* workloads, you can not also create a subscription for it.

Subscribing to a workload that has been released from a staging area

Since the staging area has monopolized your workload subscription, to trigger your action set you must now subscribe to the staged_workload_released workload — special wrapper around the original workload that has all the same variables as the original. If your staging area is for the pull_request_merged workloads, then the corresponding staged_workload_released workload will have all the same metadata.

staging_areas:
  - post_merge:
      workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*

pipelines:
  - my_pipeline

subscriptions:
  - workload: staged_workload_released:{{agent_id}}:post_merge:*
    actions:
      - trigger_pipeline:my_pipeline

What happens while a workload is in the staging area

Now that we’ve covered how workloads get into the staging area and how to subscribe to them when they leave, let’s dig into what happens when a workload enters a staging area and how Expeditor determines when it should leave, resulting in the staged_workload_released workload.

A staging area exists in one of two states: locked or unlocked. A staging area is locked after a workload leaves the staging area and it publishes the staged_workload_released workload. The staging area is unlocked when the agent executes the unlock_staging_area action, or when a human manually unlocks it via a Slack command.

staging_areas:
  - post_merge:
      workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*

pipelines:
  - my_pipeline

subscriptions:
  - workload: staged_workload_released:{{agent_id}}:post_merge:*
    actions:
      - trigger_pipeline:my_pipeline
  - workload: buildkite_build_passed:{{agent_id}}:my_pipeline:*
    actions:
      - unlock_staging_area:post_merge

Locking the staging area

When a staging area is first created it is unlocked. This means that the staging area, upon receiving its first workload, will do the following:

1. Translate the workload into a staged_workload_released workload
2. Release the new workload, triggering the subsequent subscriptions
3. Lock the staging area until it is unlocked (either manually or via the unlock_staging_area action)

While the staging area is locked, any new workloads it received will be combined with the workload that already exists in the staging area. We refer to workloads that represent multiple workloads as composite workloads.

Composite Workloads

Here is a little secret: all workloads are technically composite workloads. “Normal” workloads are technically just composite workloads that have not superseded any workloads. We refer to composite workloads as such only as a short-hand to describe workloads that have superseded other workloads.

Lets walk through a quick example to exemplify how our staging area above configured to receive pull_request_merged workloads creates composite workloads while it is locked.

When our staging area is unlocked, workload #12 is re-branded as a staged_workload_released workload and acted upon by the associated subscription. When actions in the action set act upon workload #12, they will do so in one of two ways:

1. Most recent workload. The action will act only the latest workload that was received by staging area while it was locked (workload #12)
2. All workloads. The action will act on all workloads that were received by the staging area while it was locked, in the order they were received (e.g. workloads #10, #11, then #12).

Which way an action acts upon a composite workload will vary depending on the action and is in the action reference guide.

Managing your staging area

Expeditor handles the locking of a staging area, but unlocking a staging area user requires action in of three forms:

1. Automatically, using the unlock_staging_area action.
2. Manually, using the “Unlock” button in the Expeditor Dashboard UI.
3. Manually, using the /expeditor staging-area Slack command.

Automatically unlocking your staging area

The primary method of unlocking your staging area will be through the use of the unlock_staging_area action as part of an action set that responds to a workload indicating your staging area is ready to be unlocked. Here are some common scenarios which might indicate that a staging area is ready to be unlocked.

1. The successful completion of a Buildkite pipeline. Why successful and not any completion? People often want to re-try failed pipelines. If Expeditor unlocks the staging area after a pipeline failed and triggers another build, but then a human comes along and retries the failed pipeline, the promise of the staging area has been violated. You either have two pipeline builds running (the retry by the human and the automated build from Expeditor), or you’re running builds out of order.
2. The availability of a build artifact. Since most build processes are asynchronous, using an event like artifact_published or buildkite_hab_build_group_published can serve as an indication that your staging area is ready to be unlocked. However, if you’re project creates multiple types of artifacts, this pattern will not work for you.

By default, staging areas will unlock after 24 hours. This setting can be controlled by modifying the timeout value in your Expeditor configuration file by specifying a value in a format similar to 10m (10 minutes), 12h (12 hours), or 2d (2 days).

staging_areas:
  - post_merge:
      workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
      timeout: 12h

Manually unlocking your staging area

Staging areas are useful to help coordinate and control the flow of work in your release life-cycle, but their asynchronous nature exposes them to a certain degree of unpredictability. Messages are delayed or dropped and outages can occur, which might leave your staging area in a locked state when it would otherwise be unlocked.

To determine if your staging area is locked you need to visit the Expeditor Dashboard. There, there are two methods to determining if your staging area is locked:

1. There are workloads on the main page with a button available to unlock your staging area.
2. Visit your agent page by clicking the “View All” link at the top of the table and clicking on the link associated with your agent. On that page you’ll see a table containing all of your staging areas and buttons to either lock or unlock them based on their status.

Once you have determined the status of your staging area, you can use the button in the UI to unlock your staging area. Alternatively, if you want to run a command through Slack to unlock your staging area, you can do so via the /expeditor staging-area Slack command.