Best practices for writing Bash scripts

The goal of Expeditor is to capture as many patterns and practices as we can into reusable components in one of two ways:

1. Create a built-in action for the process, like we did for things like built_in:bump_version.
2. Create a Buildkite pipeline plugin, like we did for omnibus and Chef Habitat builds.

As your project’s release life-cycle matures, it’s likely you’ll run into a situation where there is no reusable component you can use to accomplish your task. To do this you’ll want to automate your process using a Bash script and run it in one of two ways:

1. A bash action that runs as part of an action set in response to a workload.
2. A general purpose Buildkite pipeline.

The presence of a Bash script is an indication that there is a Expeditor built-in or Buildkite plugin that could be developed. When you find yourself writing a Bash script inside an Expeditor bash action or Buildkite pipeline, approach it as if you’re using that script as a way to explore and experiment with how a future built-in or plugin should operate.

The ultimate goal of every Bash script should be to be replaced by a built-in or plugin that can be shared across the engineering organization.

If this is not the case — if you’re looking at your bash script and thinking “this really only applies to me” — we encourage you to reach out and have a conversation with Release Engineering. It’s possible we have some insight that might allow you to make that script a bit more generic and usable by the larger Chef Engineering organization.

The differences between bash actions and Buildkite pipelines

Consider the following when deciding whether to run your Bash script as a bash action or in a general purpose Buildkite pipeline.

• If the script takes less than 30 seconds to run, you should run it as a bash. Otherwise run it as a step in a general purpose Buildkite pipeline.
• If the script makes pushes a commit to GitHub, you should run it as a bash action.
• If you want a human to be able to trigger the execution of the script manually, you should run it as a step in a general purpose Buildkite pipeline.
• If the script has high I/O requirements, you should run it in a general purpose pipeline.

Because bash actions and Buildkite steps are intended to be used for different purposes, they have different libraries and optimizations that are (or are not) available to you.

• The bash action helper functions are not available to Bash scripts in Buildkite pipelines.
• The logging functionality of bash actions is no where near as robust as those available to Buildkite pipelines.
• Buildkite pipelines have a whole host of useful functionality such as automatic retries.
• Buildkite pipelines allow you to run scripts in a huge variety of different ecosystems — bash actions only run on unprivileged Linux Docker containers.
• Buildkite pipelines can be triggered manually via the Buildkite API or UI — bash actions can only be triggered in response to workloads.

Three patterns for using Bash scripts in a bash action

In general Bash scripts that get run in bash actions are accomplishing a task in of one of these three categories:

1. Modifying some files and push the changes directly to the release branch. If you recall from our action sets documentation, any files modified as part of the pre-commit action set phase are committed and pushed directly to the release branch. This makes pre-commit actions incredibly useful for automating otherwise mundane code maintenance tasks like version bumping and changelog management. In fact, one of the most common examples of a Bash script making modifications that are committed and pushed directly to the release branch is the .expeditor/update_version.sh script that is commonly paired with the built_in:bump_version action. Click here to read more about that pattern.
2. Modifying some files and open up a pull request with those changes for further review. This pattern of script can be run in either the pre-commit or post-commit action set phase because any file changes are committed to a branch which is subsequently pushed to GitHub and opened as a pull request. This pattern of scripting is very common for routine maintenance tasks such as automated dependency updates. The script runs some code which updates your dependencies, whatever they might be, and opens a pull request which a developer then reviews and decides whether or not to merge. Check out the open_pull_request helper function documentation for directions on how to leverage this pattern.
3. Communicate with an external API to accomplish some task. Sometimes your project needs to interact with an external service in a way that is not supported by Expeditor, either because you’re use cases does not fit inside the supported workflow or the functionality you need does not exist in Expeditor at all. These sorts of scripts are incredibly common in projects that are exploring new release patterns.

Where to keep your scripts

We recommend that you use the .expeditor/ folder to house all of your scripts that are used within the context of bash actions and Buildkite pipelines. Scripts written for Expeditor often leverage things like the helper functions or call out to utilities available in the Buildkite pipelines like buildkite-agent. This means that they can’t be run outside of those environments easily. Storing those scripts in the .expeditor/ folder provides inherent context about where they are intended to be run.

Accessing secrets in Bash scripts

In the context of a Bash script, we require that you pull whatever secrets you need directly from Vault using the CLI, whether you’re using bash actions or general purpose Buildkite pipelines. For bash actions, the VAULT_ADDR, VAULT_NAMESPACE, and VAULT_TOKEN environment variables are automatically populated by Expeditor. For Buildkite pipelines, you’ll need to set the expeditor.secrets to true in your pipeline definition file..

#/bin/bash

set -eou pipefail

MY_VARIABLE=$(vault kv get -field MY_FIELD PATH/TO/MY/SECRET)

echo "$MY_VARIABLE"

steps:
  - label: "A step that calls a secret from a bash script"
    command: .expeditor/buildkite/my_script.sh
    expeditor:
      secrets: true

  - label: "A step that needs a secret in the command"
    command:
      - echo "\$MY_VARIABLE"
    expeditor:
      secrets:
        MY_VARIABLE:
          path: YOUR_VAULT_PATH
          field: YOUR_VAULT_FIELD

You should defer to referencing your environment variables this way. However, if in your Buildkite pipeline you’re calling a command line utility directly from the commands field and need a secret to exist as an environment variable, you should leverage the Secrets DSL.

Expeditor-provided Environment Variables

When a Bash script is executed through Expeditor, either as a bash action or through a Buildkite pipeline via trigger_pipeline, a number of Expeditor-specific environment variables are made available to you. You can identify Expeditor environment variables as they are prefixed with EXPEDITOR_ for workload metadata and EXPEDITOR_AGENT_CONFIG_ for agent configuration.

Expeditor flattens both the workload metadata and and agent configuration into key-value environment variables, which makes complex data objects like arrays and hashes difficult to grok at first. The algorithm that flattens the metadata follows the following function.

def flatten(env_var_name, value, all_env_vars = {})
  case value
  when Array
    # Append index onto the current environment variable name
    value.each_index do |i|
      flatten("#{env_var_name}_#{i}", value[i], results, joiner)
    end
  when Hash
    # Append key onto the current environment variable name
    value.each do |k, v|
      flatten("#{env_var_name}_#{k}", v, results, joiner)
    end
  when String
    # Finalize environment variable by removing all non-word characters
    results[env_var_name.gsub(/\W/, "")] = value
  end
end

Developing and troubleshooting Bash scripts locally

The goal of any Bash script written for use in a Release Engineering pipeline is that they should be written to run locally first, and Expeditor/Buildkite second. You should avoid writing a Bash script that can only be run within the context of a bash action or Buildkite pipeline. To accomplish this, we recommend you adhere to the following guidelines.

1. Never assume that environment variables are present. When writing a Bash script that uses environment variables provided by either Expeditor or Buildkite, make sure you do the following.
   1. Use “set -eou pipefail” at the top of every Bash script. Check out Buildkite’s documentation for a more detailed breakdown of this pattern.

      #!/bin/bash
      set -eou pipefail

   2. Set a default value for the missing environment variable (if appropriate). If you want to allow your script to run in a default mode outside of Expeditor, one option is to provide a default for that environment variable.

      #!/bin/bash
      set -eou pipefail
      
      # Fall back to the unstable channel if one is not provided by Expeditor
      channel="${EXPEDITOR_CHANNEL:-unstable}"
      echo "Do some things against the ${channel} channel."

   3. Control the error message when a required environment variable is not provided. If you can not specify a default value for a missing environment variable, then you should make sure the script exits non-zero. However, you’ll want to provide useful feedback about why the script failed.

      #!/bin/bash
      set -eou pipefail
      
      version="${EXPEDITOR_VERSION:?You must manually set the EXPEDITOR_VERSION environment variable to an existing semantic version.}"
      echo "We're about to do something that requires ${version} to exist"

2. Control behavior that should only occur in Expeditor/Buildkite using the CI environment variable. You should write your Bash scripts with the intention that they should be easily runnable from a local workstation. Using the CI environment variable is an indication of a larger issue that should be solved and should only be used for short-term remediation of issues in your script. If you find that you need to use CI in your script, please start a conversation with Release Engineering.

   #!/bin/bash
   set -eou pipefail
   
   if [[ "${CI:-false}" == "true" ]]; then
     # We need to perform some steps to make our script work in Expeditor/Buildkite
     # ...
   fi
   
   # ... The rest of the script that works as expected when run locally

3. Use the chefes/lita-worker:latest image to locally debug scripts with helpers. If you’re using one of the helper functions that is available within bash actions, then you’ll need to run the script in question inside the chefes/lita-worker Docker image. Expeditor uses this image to run all bash actions, and it is based on the same chefes/releng-base Docker image used to power the chefes/buildkite image used for the docker executor.

   docker run --rm --volume $(PWD):/workspace --workdir /workspace chefes/lita-worker .expeditor/my-script.sh

   Make sure that you know what required environment setup each helper function requires (which is documented with each helper function). There are a few common types of environment setup that you’ll need to pass in to the Docker container in order for a helper function to work correctly.
   1. git CLI credentials. The easiest way to pass these in reliably is to maintain a netrc file with the required credentials and mounting it into the container.

      docker run --rm --volume $(PWD):/workspace --volume $(HOME)/.netrc:/root/.netrc --workdir /workspace chefes/lita-worker .expeditor/my-script.sh

   2. Environment Variables. To pass in environment variables you need only to specify the --env parameter in your docker CLI command. The example below is an example for how to inject the VAULT_ADDR, VAULT_NAMESPACE, and VAULT_TOKEN environment variables required by a number of helpers. You’ll need to make sure you’re properly logged in to Vault before you can pass in these credentials. Please reach out to Release Engineering if you have not previously logged in to our Vault instance.

      docker run --rm --volume $(PWD):/worksapce --env VAULT_ADDR="$VAULT_ADDR" --env VAULT_NAMESPACE="$VAULT_NAMESPACE" --env VAULT_TOKEN=$(cat ~/.vault-token) --workdir /workspace chefes/lita-worker .expeditor/my-script.sh