Using Expeditor to build Docker images

Docker images are a common standard for container images, and the Chef Release Engineering team’s preferred packaging mechanism for services running in Kubernetes.

Creating your Docker image build pipelines

Docker build Buildkite pipelines are another example of a named pipeline. To create the pipeline, you’ll need the following:

1. A single docker/build pipeline defined in your .expeditor/config.yml file.
2. A dobi.yaml (located in the root of your repository) that defines all the Docker images you wish to build.
3. The .expeditor/build.docker.yml shim pipeline definition file.

As with other named pipelines, prefixing your pipeline name with docker/ informs Expeditor that this Buildkite pipeline is different than a general purpose pipeline and tells it to expect a shim pipeline definition rather than a traditional pipeline definition. Remember, shim pipeline definition files are not supported by the native Buildkite DSL.

---
pipelines:
  - docker/build:
      definition: .expeditor/build.docker.yml

Triggering builds of your Docker images

There are three ways we recommend to trigger your docker/build pipeline.

1. Via the trigger_pipeline:docker/build action. The most common pattern is to trigger this action in response to a pull_request_merged workload.

subscriptions:
  - workload: pull_request_merged:{{github_repo}}:{{release_branch}}:*
    actions:
      - ... # pre-commit actions like built_in:bump_version
      - trigger_pipeline:docker/build

1. Via the Buildkite UI. Triggering a release build via the Buildkite UI is useful when you need to trigger a fresh build out of band of a code change to your project.
2. Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the bk build create command.

bk build create --pipeline=chef/chef-example-main-docker-build

dobi.yaml

The .expeditor/build.docker.yml shim pipeline definition file defers to Dobi’s dobi.yaml file to specify which Docker images it needs to build.

Per the DSL, you can enumerate the Docker images by specify a image=IMAGE_NAME key for each. Nested under each key is the configuration for that image. It’s important that the value for IMAGE_NAME matches the name of your image. The specific values are enumerated below.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .
  tags:
    - '{env.VERSION}'

image

The full name of your container image, including the container registry. You can use {env.IMAGE_REGISTRY} to refer to the image registry you specified in your .expeditor/build.docker.yml shim pipeline definition file. Keeping this value as an environment value makes it easy for you to specify an alternate value for local development (e.g. localhost:5000) using something like direnv.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'

image=image-two:
  image: '{env.IMAGE_REGISTRY}/image-two'

context

The path, relative to the root of the repository, to the Docker context. More often than not, this is the path to where your Dockerfile is stored. This value is required. You can specify . to refer to the root of the repository.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .

depends

An optional list of image keys which this image depends on (e.g. using a FROM instruction). Expeditor will ensure the dependent images are built prior to the image in question. Please note, you will need to ensure that the image name includes the correct tag (e.g. version). To ensure this happens, you can use a sed command in your update-version.sh script.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: ./image-one

image=image-two:
  image: '{env.IMAGE_REGISTRY}/image-two
  context ./image-two
  depends:
    - image-one

tags

A list of tags to be applied to the image when it is built. We strongly encourage you to include {env.VERSION} in this list, especially if you wish to use the built_in:promote_docker_images action.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .
  tags:
    - '{env.VERSION}`

If you are not using the built_in:promote_docker_images workload, you can include other tags like latest.

annotations.tags

An optional list of tags that can control Expeditor-specific behavior in our pipelines.

expeditor:host-os

By default, Expeditor will build Linux Docker images. However, you can control whether or not you want to build a Windows-based Docker image using the host-os tag.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: ./image-one

image=image-two:
  image: '{env.IMAGE_REGISTRY}/image-two'
  context: ./image-two
  annotations:
    tags:
      - expeditor:host-os=windows

expeditor:ignore

You can specify this tag if you have a Dobi image definition in your dobi.yaml file that you do not want to build as part of your docker/build pipeline. This is common if you are also using your Dobi file to build images in your verify pipeline.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .

image=image-one-build:
  image: '{env.IMAGE_REGISTRY}/image-one-build'
  context: .
  annotations:
    tags:
      - expeditor:ignore

expeditor:default-tags

An optional comma-separated list of tags that you wish to apply to images when built_in:promote_docker_images executes. This value supports our template tag variables.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .
  tags:
    - '{env.VERSION}`
  annotations:
    tags:
      - expeditor:default-tags={{channel}},{{channel}}-{{git_sha}}

expeditor:final-channel-tags

An optional comma-separated list of tags that you wish to apply to images when built_in:promote_docker_images promotes an image to its final artifact_channels. This value supports our template tag variables.

---
image=image-one:
  image: '{env.IMAGE_REGISTRY}/image-one'
  context: .
  tags:
    - '{env.VERSION}`
  annotations:
    tags:
      - expeditor:final-channel-tags={{major}},{{major}}.{{minor}},latest

Template Tag Variables

.expeditor/build.docker.yml

The final component of our docker/build pipeline is our build.docker.yaml. This file will look a lot like the Habitat build definition in that we’ll want to do the following:

1. Provide a means to specify a Dobi file name.
2. Provide a means to specify which Docker Hub organization you would like your image uploaded.
3. Provide a means to inject secrets and environment variables to use at build time.

---
env:
  VERSION: "1.0.0"
  VAULT_SECRET:
    path: "foo"
    field: "bar"

image_registry: chefes
dobi_file: dobi.yaml

image_registry

The name of the image registry where you wish to upload the Docker image upon build completion. There are two chocies that we support currently. The first is interacting with Docker Hub and the other is our internal Artifactory instance. We recommend that any container that does not need to be pulic facing to be stored in artifactory. At this time we only support pushing all containers to one registry so either push all to docker hub or push all to artifactory. You cannot mix and match. The default value points to docker hub chef repository.

Docker Hub Registries

You can specify the name of one of the following Docker Hub organizations:

• chef (for official Chef products only)
• chefes
• devchef (playground for containers that are not deployed to a specific environment)

Artifactory Registries

You can specify the name of one of the following repositories:

• artifactory.chef.co/chef (for official Chef products only)
• artifactory.chef.co/chefes
• artifactory.chef.co/devchef (playground for containers that are not deployed to a specific environment)

env

A hash of environment variables you wish to inject into the docker/build pipeline. It can support traditional key/value pairs (e.g. injecting plain text version) but it can also inject values from Vault using the Secrets DSL.

Managing your VERSION

Expeditor will not automatically inject the contents of your VERSION file into the {env.VERSION} environment variable. You will need to manually configure the version number in your .expeditor/docker.build.yml file and keep it up to date using your update_version.sh file (per the version management guidelines).

dobi_file

The path to the location of the Dobi configuration file. Defaults to dobi.yaml.