Inject secrets from Vault into Buildkite

Secret data used in Expeditor is stored in our Hashicorp Vault instance, which is only accessible to private pipelines. There are two methods of access secrets from Vault:

Using the vault CLI to directly fetch secrets as part of a Bash script.
Injecting secrets as environment variables using the secrets DSL when the vault CLI is not accessible.

This guide will cover the latter.

The secrets DSL is available in the following contexts:

General purpose Buildkite pipelines. The secrets DSL is available under the steps.#.expeditor.secrets or expeditor.secrets keyspace in your pipeline definition file..
Chef Habitat Buildkite build pipelines. The secrets DSL is available under the studio_secrets keyspace in your shim pipeline definition file.

The structure of the secrets DSL is a hash where the key is the name of your environment variable and the value is another hash containing secret location data.

---
ENV_VAR_NAME:
  ... # secret location data

There are three categories of secret location data that the secrets DSL supports:

account. Secret data associated with dynamically generated account credentials for things like AWS and GitHub.
path. Referencing static secret data stored at a specific path in Vault.
value. A way to inject “plain text secrets” into an environment. This is commonly used with the studio_secrets implementation to take advantage of the automatic injection (and prefixing with HAB_STUDIO_SECRET_) of environment variables into the Chef Habitat studio.

In this document we will refer to actions that Expeditor takes, however the implementation within the Buildkite pipelines themselves is handled by the vault-util helper. This helper utility is installed by default on the Core, Core Docker and Chef Habitat Build pipeline images. Rather than injecting the environment variables and their secrets as plain text into the Buildkite environment, Expeditor passes the content of the secrets DSL as the VAULT_UTIL_SECRETS environment variable to the Buildkite build. Inside the appropriate Buildkite step, the vault-util fetch-secret-env helper runs as part of the Buildkite environment hook to fetch and inject the secrets at runtime, helping ensure that no secrets are stored in plain text in the logs.

account

An account secret is one associated with dynamically generated credentials for one of the following services:

aws
azure
github
google

What differentiates account secrets from path or value secrets is that Expeditor ensures that multiple secrets from the same account work together. For example, let’s take a look at a pretty standard usage of the aws account type in the secrets DSL.

AWS_ACCESS_KEY_ID:
  account: aws/chef-example
  field: access_key_id
AWS_SECRET_ACCESS_KEY:
  account: aws/chef-example
  field: secret_access_key
AWS_SESSION_TOKEN:
  account: aws/chef-example
  field: session_token
AWS_REGION:
  account: aws/chef-example
  field: region

In the code block above you can see we have four individual secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, and AWS_REGION. All of those secrets are pulling specific fields for the fictional aws/chef-example account. What Expeditor does is ensure that all of those individual secrets are actually related to a single temporary STS credential set.

Expeditor does not enforce any special naming requirements for environment variables. The intent of the account secrets functionality is to provide a mechanism by which to easily inject environment variables into execution environments so that native tooling can detect their presence (e.g. AWS_ACCESS_KEY_ID). You are free to name the environment variables whatever you please.

aws

When you specify an AWS account, you’ll need to provide the name of the account for which you want to generate the temporary STS credentials. To determine which account is best for your situation, please consult the Release Engineering team.

Available Fields

The following fields are available for an AWS account secret.

Field	Description
`access_key_id`	The Access Key ID for the temporary IAM credentials
`secret_access_key`	The Secret Access Key for the temporary IAM credentials
`session_token`	The Session Token for the temporary IAM credentials
`region`	The AWS region for which the temporary credentials have access.

Sample Usage

Inject the three environment variables required by the AWS to communicate with AWS API using temporary STS credentials for the fictional chef-example account.

AWS_ACCESS_KEY_ID:
  account: aws/chef-example
  field: access_key_id
AWS_SECRET_ACCESS_KEY:
  account: aws/chef-example
  field: secret_access_key
AWS_SESSION_TOKEN:
  account: aws/chef-example
  field: session_token

Let’s take a moment to walk through how this code block is processed.

AWS_ACCESS_KEY_ID. Expeditor generates a temporary STS credentials, fetches the access_key_id field from the Vault secret, and assigns it to the AWS_ACCESS_KEY_ID environment variable.
AWS_SECRET_ACCESS_KEY. Expeditor sees that is has already generated temporary STS credentials for the chef-example AWS account, so it fetches the secret_access_key from that cached Vault secret and assigns it to the AWS_SECRET_ACCESS_KEY environment variable.
AWS_SESSION_TOKEN. Expeditor repeats the same process again, fetching the session_token from the cached Vault secret and assigning it to the AWS_SESSION_TOKEN environment variable.

azure

When you specify an Azure account, you’ll need to provide the nickname for the tenant for which you want to generate temporary credentials. To determine which tenant is best for your situation, please consult the Release Engineering team.

Available Fields

The following fields are available for an Azure account secret.

Field	Description
`tenant_id`	The unique identifier for the Azure Tenant
`subscription_id`	The unique identifier for the subscription within the Azure Tenant
`client_id`	The ID for the Azure Application
`client_secret`	The secret associated with the Azure Application

Sample Usage

Inject the three environment variables required by Azure to communicate with the Azure API using temporary service principal credentials for the fictional example Azure subscription.

AZURE_TENANT_ID:
  account: azure/example
  field: tenant_id
AZURE_CLIENT_ID:
  account: azure/example
  field: client_id
AZURE_CLIENT_SECRET:
  account: azure/example
  field: client_secret

Let’s take a moment to walk through how this code block is processed.

AZURE_TENANT_ID. Expeditor generates temporary service principal credentials for the example subscription, fetches the tenant_id field from the Vault secret, and assigns it to the AZURE_TENANT_ID environment variable.
AZURE_CLIENT_ID. Expeditor sees that is has already generated the temporary service principal credentials for the example Azure subscription, so it fetches the client_id from that cached Vault secret and assigns it to the AZURE_CLIENT_ID environment variable.
AZURE_CLIENT_SECRET. Expeditor repeats the same process again, fetching the session_token from the cached Vault secret and assigning it to the AZURE_CLIENT_SECRET environment variable.

github

Generate a temporary access token for one our our GitHub organizations. See our projects page for a list of supported organizations. If no organization is specified, Expeditor will default to the chef organization.

Available Fields

The following fields are available for a GitHub account secret.

Field	Description
`token`	A temporary access token for the specified GitHub organization.

Sample Usage

Inject the GITHUB_TOKEN environment variable that can be used for interactions with the GitHub API for the inspec GitHub organization.

GITHUB_TOKEN:
  account: github/inspec
  field: token

google

Generate a temporary Google Cloud Principal with access to the specified Application. To determine which application is best for your situation, please reach out to Release Engineering.

Available Fields

The following fields are available to your Google Cloud account secret.

Field	Description
`token`	The API KEY that can be used for communicating with the Google Cloud API.
`json`	The service account keys JSON document that can be used for authenticating server applications.

Sample Usage: API Key

Inject the GOOGLE_API_KEY environment variable with access to the fictional example application to use with the Google Cloud API.

GOOGLE_API_KEY:
  account: google/example
  field: token

Sample Usage: Service Account Key

Inject the GOOGLE_APPLICATION_CREDENTIALS environment variable to access the fictional example application using the Google Cloud SDK.

GOOGLE_APPLICATION_CREDENTIALS:
  account: google/example
  field: json

path

A path secret is one associated with a specific secret stored in Vault. Path secrets have their path and field values fed directly into the vault read API. Expeditor does not do any additional processing like it does with account secrets. If you need assistance in accessing a specific secret, please reach out to the Release Engineering team.

Sample Usage

Inject the MY_ENVIRONMENT_VARIABLE environment variable with the value of the foobar field of the secret located at secret/myapp/custom_secret in Chef’s Hashicorp Vault instance.

MY_ENVIRONMENT_VARIABLE:
  path: secret/myapp/custom_secret
  field: foobar

value

There are some situations, like specifying studio_secrets in the Habitat pipeline definition file, where you simply want to pass in a plain text value into the DSL. To do this, you can simply provide a value.

Sample Usage

Inject the PLAINTEXT_SECRET environment variable with the plaintext value of “foo”.

PLAINTEXT_SECRET:
  value: foo