hcp vault Use Vault to manage API tokens for the Terraform Cloud Operator

[Hashicorp]

The HashiCorp Terraform Cloud Operator for Kubernetes continuously reconciles infrastructure resources using Terraform Cloud. When you use the operator to create a Terraform Cloud workspace, you must reference a Terraform Cloud API token stored in a Kubernetes secret. One way to better secure these secrets instead of hard-coding them involves storing and managing secrets in a centralized secrets manager, like HashiCorp Vault. In this approach, you need to synchronize secrets revoked and created by Vault into Kubernetes. An operator like the Vault Secrets Operator (VSO) can retrieve secrets from an external secrets manager and store them in a Kubernetes secret for workloads to use.

This post demonstrates how to use the Vault Secrets Operator (VSO) to retrieve dynamic secrets from Vault and write them to a Kubernetes secret for the Terraform Cloud Operator to reference when creating a workspace. While the example focuses on Terraform Cloud API tokens, you can extend this workflow to any Kubernetes workload or custom resource that requires a secret from Vault.

Install Vault and operators

The Terraform Cloud Operator requires a user or team API token with permissions to manage workspaces, plan and apply runs, and upload configurations. While you can manually generate a token in the Terraform Cloud UI, configure Vault to issue API tokens for Terraform Cloud. The Terraform Cloud secrets engine for Vault handles the issuance and revocation of different kinds of API tokens in Terraform Cloud. Vault manages the token’s lifecycle and audits its usage and distribution once you reference it in the Terraform Cloud Operator.

The demo repository for this post sets up the required infrastructure resources, including a:

• Vault cluster on HCP Vault
• Kubernetes cluster on AWS

After provisioning infrastructure resources, the demo repo installs Helm charts for Vault, Terraform Cloud Operator, and Vault Secrets Operator in their own namespaces using Terraform. If you do not use Terraform, install each Helm chart by CLI.

First, install the Vault Helm chart. If applicable, update the values to reference an external Vault cluster:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

$ helm install vault hashicorp/vault

Install the Helm chart for the Terraform Cloud Operator with its default values:

$ helm install terraform-cloud-operator hashicorp/terraform-cloud-operator

Install the Helm chart for VSO with a default Vault connection to your Vault cluster:

$ helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --set defaultVaultConnection.enabled=true \
  --set defaultVaultConnection.address=$VAULT_ADDR

Any custom resources created by VSO will use the default Vault connection. If you have different Vault clusters, you can define a VaultConnection custom resource and reference it in upstream dependencies.

After installing Vault and the operators, configure the Kubernetes authentication method in Vault. This ensures VSO can use Kubernetes service accounts to authenticate to Vault.

Set up secrets in Vault

After installing a Vault cluster and operators into Kubernetes, set up the secrets engines for your Kubernetes application. The Terraform Cloud Operator needs a Terraform Cloud API token with permissions to create projects and workspaces and upload Terraform configuration. On the Terraform Cloud Free tier, you can generate a user token with administrative permissions or a team token for the “owners” team to create workspaces and apply runs.

To further secure the operator’s access to Terraform Cloud, upgrade to a plan that supports teams to secure the Terraform Cloud Operator’s access to Terraform Cloud. Then, create a team, associate a team token with it, and scope the token’s access to a Terraform Cloud project. This ensures that the Terraform Cloud Operator has sufficient access to create workspaces and upload configuration in a given project without giving it access to an entire organization.

Configure the Terraform Cloud secrets engine for Vault to handle the lifecycle of the Terraform Cloud team API token. The demo repo uses Terraform to enable the backend. Pass in an organization or user token with permissions to create other API tokens.

resource "vault_terraform_cloud_secret_backend" "apps" {
  backend     = "terraform"
  description = "Manages the Terraform Cloud backend"
  token       = var.terraform_cloud_root_token
}

Create a role for each Terraform Cloud team that needs to use the Terraform Cloud Operator. Then pass the team ID to the role to configure the secrets engine to generate team tokens:

resource "vault_terraform_cloud_secret_role" "apps" {
  backend      = vault_terraform_cloud_secret_backend.apps.backend
  name         = "payments-app"
  organization = var.terraform_cloud_organization
  team_id      = "team-*******"
}

Build a Vault policy that allows read access to the secrets engine credentials endpoint and role:

resource "vault_policy" "terraform_cloud_secrets_engine" {
  name     = "terraform_cloud-secrets-engine-payments-app"

  policy = <

The Terraform Cloud Operator needs the Terraform Cloud team token to create workspaces, upload configurations, and start runs. However, you may also want to pass secrets to workspace variables. For example, a Terraform module may need a username and password to configure HCP Boundary. You can store the credentials in Vault’s key-value secrets engine and configure a Vault policy to read the static secrets.

After setting up policies to read the required secrets, create a Vault role for the Kubernetes authentication method, which allows the terraform-cloud service account to authenticate to Vault and retrieve the Terraform Cloud token:

resource "vault_kubernetes_auth_backend_role" "terraform_cloud_token" {
 backend                          = "kubernetes"
 role_name                        = "payments-app"
 bound_service_account_names      = ["terraform-cloud"]
 bound_service_account_namespaces = ["payments-app"]
 token_ttl                        = 86400
 token_policies = [
   vault_policy.terraform_cloud_secrets_engine.name,
 ]
}

Refer to the complete repo to configure the Terraform Cloud secrets engine and store static secrets for the Terraform Cloud workspace variables.

Sync secrets from Vault to Kubernetes

The Terraform Cloud Operator includes a custom resource to create workspaces and define workspace variables. However, dynamic variables refer to values stored in a Kubernetes Secret or ConfigMap. Use VSO to synchronize secrets from Vault into native Kubernetes secrets. The demo repo for this post retrieves the Terraform Cloud team token and static credentials and stores them as a Kubernetes secret.

VSO uses a Kubernetes service account linked to the Kubernetes authentication method role in Vault. First, deploy a service account and service account token for terraform-cloud to the payments-app namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: terraform-cloud
 namespace: payments-app
---
apiVersion: v1
kind: Secret
metadata:
 name: terraform-cloud-token
 namespace: payments-app
type: kubernetes.io/service-account-token

Then, configure a VaultAuth resource for VSO to use the terraform-cloud service account and authenticate to Vault using the kubernetes mount path and payments-app role defined for the authentication method. The configuration shown here sets Vault namespace to admin for your HCP Vault cluster:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
 name: terraform-cloud
 namespace: payments-app
spec:
 method: kubernetes
 mount: kubernetes
 namespace: admin
 kubernetes:
   role: payments-app
   serviceAccount: terraform-cloud
   audiences:
     - vault

To sync the Terraform Cloud team token required by the Terraform Cloud Operator to a Kubernetes secret, define a VaultDynamicSecret resource to retrieve the credentials. VSO uses this resource to retrieve credentials from the terraform/creds/payments-app path in Vault and creates a Kubernetes secret named terraform-cloud-team-token with the token value. The resource refers to VaultAuth for authentication to Vault:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
 name: terraform-cloud-team-token
 namespace: payments-app
spec:
 mount: terraform
 path: creds/payments-app
 destination:
   create: true
   name: terraform-cloud-team-token
   type: Opaque
 vaultAuthRef: terraform-cloud

When you apply these manifests to your Kubernetes cluster, VSO retrieves the Terraform Cloud team token and stores it in a Kubernetes secret. The Operator’s logs indicate the handling of the VaultAuth resource and synchronization of the VaultDynamicSecret:

$ kubectl logs -n vault-secrets-operator $(kubectl get pods \
   -n vault-secrets-operator \
   -l app.kubernetes.io/instance=vault-secrets-operator -o name)

2024-03-14T16:38:47Z    DEBUG   events  Successfully handled VaultAuth resource request {"type": "Normal", "object": {"kind":"VaultAuth","namespace":"payments-app","name":"terraform-cloud","uid":"e7c0464e-9ce8-4f3f-953a-f8eb10853001","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"331817"}, "reason": "Accepted"}

2024-03-14T16:38:47Z    DEBUG   events  Secret synced, lease_id="", horizon=0s  {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"payments-app","name":"terraform-cloud-team-token","uid":"d1563879-41ee-4817-a00b-51fe6cff7e6e","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"331826"}, "reason": "SecretSynced"}

Verify that the Kubernetes secret terraform-cloud-team-token contains the Terraform Cloud team token:

$ kubectl get secrets -n payments-app \
  terraform-cloud-team-token -o jsonpath='{.data.token}' | base64 -d

******.****.*****

Create a Terraform Cloud workspace using secrets

You can now configure other Kubernetes resources to reference the secret synchronized by VSO. For the Terraform Cloud Operator, deploy a Workspace resource that references the Kubernetes secret with the team token:

apiVersion: app.terraform.io/v1alpha2
kind: Workspace
metadata:
 name: payments-app-database
 namespace: payments-app
spec:
 organization: hashicorp-stack-demoapp
 project:
   name: payments-app
 token:
   secretKeyRef:
     name: terraform-cloud-team-token
     key: token
 name: payments-app-database
 
## workspace variables omitted for clarity

The team token has administrator access to create and update workspaces in the “payments-app” project in Terraform Cloud. You can use a similar approach to passing Kubernetes secrets as workspace variables.

Deploy a Module resource to apply a Terraform configuration in a workspace. The resource references a module source, variables to pass to the module, and outputs to extract. The Terraform Cloud Operator uploads a Terraform configuration to the workspace defining the module.

apiVersion: app.terraform.io/v1alpha2
kind: Module
metadata:
 name: database
 namespace: payments-app
spec:
 organization: hashicorp-stack-demoapp
 token:
   secretKeyRef:
     name: terraform-cloud-team-token
     key: token
 destroyOnDeletion: true
 module:
   source:  "joatmon08/postgres/aws"
   version: "14.9.0"

## module variables omitted for clarity

Terraform Cloud will start a run to apply the configuration in the workspace.

Rotate the team API token

Terraform Cloud allows only one active team token at a time. As a result, the Terraform Cloud secrets engine does not assign leases to team tokens and requires manual rotation. However, Terraform Cloud does allow issuance of multiple user tokens. The secrets engine assigns leases to user API tokens and will rotate them dynamically.

To rotate a team token, run a Vault command to rotate the role for a team token in Terraform Cloud:

$ vault write -f terraform/rotate-role/payments-app

VSO must update the Kubernetes secret with the new token when the team token is rotated. Edit a field in the VaultDynamicSecret resource, such as renewalPercent, to force VSO to resynchronize:

$ kubectl edit VaultDynamicSecret terraform-cloud-team-token -n payments-app 

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  annotations:
    ## omitted
spec:
  ## omitted
  renewalPercent: 60
  vaultAuthRef: terraform-cloud

VSO recognizes the new team token in Vault and reconciles it with the Kubernetes secret:

$ kubectl logs -n vault-secrets-operator $(kubectl get pods \
   -n vault-secrets-operator \
   -l app.kubernetes.io/instance=vault-secrets-operator -o name)

2024-03-18T16:10:19Z    INFO    Vault secret does not support periodic renewal/refresh via reconciliation       {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "VaultDynamicSecret": {"name":"terraform-cloud-team-token","namespace":"payments-app"}, "namespace": "payments-app", "name": "terraform-cloud-team-token", "reconcileID": "3d0a15f1-0edf-450b-8be1-6319cd3b2d02", "podUID": "4eb7f16a-cfcb-484e-b3da-54ddbfc6a6a6", "requeue": false, "horizon": "0s"}
2024-03-18T16:10:19Z    DEBUG   events  Secret synced, lease_id="", horizon=0s  {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"payments-app","name":"terraform-cloud-team-token","uid":"f4f0483c-895d-4b05-894c-24fdb1518489","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1915673"}, "reason": "SecretRotated"

Note that this manual workflow for rotating tokens applies specifically to team and organization tokens generated by the Terraform Cloud secrets engine. User tokens have leases, which VSO handles automatically. VSO also supports the rotation of credentials for static roles in database secrets engines. Set the allowStaticCreds attribute in the VaultDynamicSecret resource for VSO to synchronize changes to static roles.

Learn more

As shown in this post, rather than store Terraform Cloud API tokens as secrets in Kubernetes, you can manage the tokens with Vault and use the Vault Secrets Operator to synchronize them to Kubernetes secrets for the Terraform Cloud Operator to use. By managing the Terraform Cloud API token in Vault, you can audit its usage and handle its lifecycle in one place.

In general, the pattern of synchronizing to a Kubernetes secret allows any permitted Kubernetes custom resource or workload to use the secret while Vault manages its lifecycle. As a result, you can track the usage of secrets across your Kubernetes workloads without refactoring applications already using Kubernetes secrets.

Learn more about the Vault Secrets Operator in our VSO documentation. If you want to further secure your secrets in Kubernetes, check out our blog post comparing three methods to inject secrets from Vault into Kubernetes workloads.

If you support a GitOps workflow in your organization and want to empower teams to deploy infrastructure resources using Kubernetes, review our documentation on the Terraform Cloud Operator to deploy and manage infrastructure resources through modules. Refer to GitHub for a complete example provisioning a database and other infrastructure resources.

View the full article