Skip to main content

Terraform HTTP Backend

Expeditor supports using the Terraform HTTP backend with either state locking or without. State locking prevents any two people from executing the same terraform at the same time to provision some resource. To use the backend, you need to be able to generate a vault token to authorize you with the API.

HTTP Call FLow

Terraform Init

Terraform issues a GET request to the API to get the current state. If the state exists then the API will return the current state. If it does not exist, then it will send a JSON object back with a version key to make terraform happy.

Terraform Plan without state locking

Terraform issues a GET request like terraform init.

Terraform Apply without state locking

Terraform issues a GET request like terraform init and after it done with the provision then it issues a POST request to save the current state.

Terraform Plan with state locking

Terraform issues a POST to the lock address to lock the state for planning. Then terraform issues a GET request like terraform init. It will then issue a DELETE request to the unlock request.

Terraform Apply with state locking

Terraform issues a POST to the lock address to lock the state for applying. Then terraform issues a GET requeset like terraform init. Terraform at this point will start provisioning your resources to where you want them. Once down, terraform will issue a POST to the API to save the current state and will finally issue the DELETE to the unlock address to free the resource to be used by someone else.

Prerequisite

In whatever terminal you are using the terraform CLI commands, you need to set this enviroment variable

export TF_HTTP_USERNAME="<progress_username>"
export TF_HTTP_PASSWORD="<vault-token>"

Without State locking

To use the backend without state locking, configure your backend as such:

terraform {
  backend "http" {
    address = "https://expeditor.chef.io/api/terraform/repo/<repo>/resource/<resource_id>"
  }
}
  • <repo> replace this variable with the repo you are storing the terraform in
  • <resource_id> replace this variable with some unique name to what your terraform is doing

Then just issue your terraform commands as normal, i.e. terraform init, terraform plan, etc

With State locking

To use the backend with state locking, configure your backend as such:

terraform {
  backend "http" {
    address = "https://expeditor.chef.io/api/terraform/repo/<repo>/resource/<resource_id>"
    lock_address = "https://expeditor.chef.io/api/terraform/repo/<repo>/resource/<resource_id>/lock"
    unlock_address = "https://expeditor.chef.io/api/terraform/repo/<repo>/resource/<resource_id>/lock"
 
    lock_method = "POST"
    unlock_method = "DELETE"
  }
}

Note the two extra addresses for lock/unlock and the type of REST method to invoke them.

  • <repo> replace this variable with the repo you are storing the terraform in
  • <resource_id> replace this variable with some unique name to what your terraform is doing

Then just issue your terraform commands as normal, i.e. terraform init, terraform plan, etc

Common Errors

  • Invalid Auth
terraform init
Initializing modules...
Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 2.15.0 for vault_snapshots_s3_bucket...
- vault_snapshots_s3_bucket in .terraform/modules/vault_snapshots_s3_bucket

Initializing the backend...

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint invalid auth

Either set TF_HTTP_PASSWORD or create a new Vault token and set the environment variable.

  • Cannot Acquire State Lock
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: HTTP remote state already locked, failed to unmarshal body
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Either wait a few minutes for the other person to release the lock or contact Release Engineering to manually remove the lock.

  • Failed to Unlock ressource
Releasing state lock. This may take a few moments...
╷
│ Error: Error releasing the state lock
│
│ Error message: Failed to unlock: DELETE http://expeditor.local/api/terraform/repo/expeditor/resource/test-s3/lock giving up after 3
│ attempt(s)
│
│ Terraform acquires a lock when accessing your state to prevent others
│ running Terraform to potentially modify the state at the same time. An
│ error occurred while releasing this lock. This could mean that the lock
│ did or did not release properly. If the lock didn't release properly,
│ Terraform may not be able to run future commands since it'll appear as if
│ the lock is held.
│
│ In this scenario, please call the "force-unlock" command to unlock the
│ state manually. This is a very dangerous operation since if it is done
│ erroneously it could result in two people modifying state at the same time.
│ Only call this command if you're certain that the unlock above failed and
│ that no one else is holding a lock.

Contact Release Engineering to manually unlock the resource.