gitlab Cl/CD for Gitlab repositories with Cloud Build repositories Gen2

[Google Cloud Platform]

The trailhead for any path to production starts at source control. The integration of Cloud Build, as the automation tool, with the source repository is therefore crucial to increasing the delivery speed and ultimately becoming a high-performing organization.

In this article we look at two exciting new capabilities of the recently launched second generation of Cloud Build repositories. First, Cloud Build can now connect to source code repositories in Gitlab and Gitlab Enterprise. And second, repository connections can now be managed declaratively in Terraform. We will now dive into a simple end-to-end demonstration to experience both of these features.

Preparing the Gitlab Environment

For our demo we use a private repository that is hosted on Gitlab.com. If you already have a Gitlab repository with a Cloud Build configuration, you can set the GITLAB_REPO_URI variable to the HTTPS URI of your repository and continue with the next section.

To create a minimal Gitlab repository to experiment with Cloud Build you can perform the following steps:

1. Create a private repo in Gitlab.com. I am using cloud-build-demo as the name here.

2. Store the repo URI as the variable GITLAB_REPO_URI in my case this is: GITLAB_REPO_URI='https://gitlab.com/<USER>/cloud-build-demo.git' (you can use ssh for the local clone to the workstation but we'll use HTTPS for the Cloud Build Repository below

3. In a terminal (e.g. cloud shell or locally) initialize the repository with a cloudbuild.yaml config as follows:

code_block

[StructValue([(u'code', u'mkdir cloud-build-demo && cd cloud-build-demo\r\ncat <<EOF >cloudbuild.yaml\r\nsteps:\r\n - name: ubuntu\r\n id: just a demo\r\n args:\r\n - echo\r\n - Probably the world\'s simplest pipeline.\r\nEOF\r\ngit init --initial-branch=main\r\ngit remote add gitlab $GITLAB_REPO_URI\r\ngit add .\r\ngit commit -m "initial import"\r\ngit push -u gitlab main'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dc033610>)])]

You should now see your repository content (in our case just the cloudbuild.yaml file) in the Gitlab web UI.

Preparations in Google Cloud

To get started we need to ensure that our Google Cloud project has the necessary APIs for Cloud Build and Secret Manager enabled on the project. Our example requires us to add the two services to a Terraform configuration or to run the following commands in a terminal:

code_block

[StructValue([(u'code', u'export PROJECT_ID=<my-project-id-here>\r\ngcloud services enable cloudbuild.googleapis.com secretmanager.googleapis.com --project $PROJECT_ID'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dc0337d0>)])]

Cloud Build repositories are authenticated at the level of a so-called host connection. The authentication process of a host connection is specific to the source code repository and differs slightly between Gitlab and Github. For host connection to access repositories in Gitlab you need to issue personal access tokens for both the api and read-api scope as described in the document Connect to a GitLab host.

Once you have issued the tokens you can store them as environment variables GITLAB_API_TOKEN and GITLAB_READ_API_TOKEN and run the command below to create secrets in Secret Manager. If you look closely you’ll find that we will also create the required secret for Gitlab webhooks but we won’t use them for this demo. In a last step we also authorize the Cloud Build Service Agent to use the secrets that we just created such that it can establish the host connection.

Note: You could create these secrets in Terraform as well but the plain text values will be visible in your tf state.

code_block

[StructValue([(u'code', u'GITLAB_API_TOKEN=\'### SET TOKEN HERE ###\'\r\nGITLAB_READ_API_TOKEN=\'### SET TOKEN HERE ###\'\r\n\r\ngcloud secrets create gitlab-api-token \\\r\n --replication-policy="automatic" --project=$PROJECT_ID\r\necho -n "$GITLAB_API_TOKEN" | \\\r\n gcloud secrets versions add gitlab-api-token --project=$PROJECT_ID --data-file=-\r\nGITLAB_API_TOKEN_SECRET_REF=$(gcloud secrets versions list gitlab-api-token --format json | jq -r \'.[0].name\')\r\n\r\ngcloud secrets create gitlab-read-token \\\r\n --replication-policy="automatic" --project=$PROJECT_ID\r\necho -n "$GITLAB_READ_API_TOKEN" | \\\r\n gcloud secrets versions add gitlab-read-token --project=$PROJECT_ID --data-file=-\r\nGITLAB_READ_API_TOKEN_SECRET_REF=$(gcloud secrets versions list gitlab-read-token --format json | jq -r \'.[0].name\')\r\n\r\ngcloud secrets create gitlab-webhook-token \\\r\n --replication-policy="automatic" --project=$PROJECT_ID\r\necho -n "not-used-here" | \\\r\n gcloud secrets versions add gitlab-webhook-token --project=$PROJECT_ID --data-file=-\r\nGITLAB_WEBHOOK_TOKEN_SECRET_REF=$(gcloud secrets versions list gitlab-webhook-token --format json | jq -r \'.[0].name\')\r\n\r\nPROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format="value(projectNumber)")"\r\nCLOUD_BUILD_SA_MEMBER="serviceAccount:service-$PROJECT_NUMBER@gcp-sa-cloudbuild.iam.gserviceaccount.com"\r\ngcloud secrets add-iam-policy-binding gitlab-api-token --member=$CLOUD_BUILD_SA_MEMBER --role=roles/secretmanager.secretAccessor --project=$PROJECT_ID\r\ngcloud secrets add-iam-policy-binding gitlab-read-token --member=$CLOUD_BUILD_SA_MEMBER --role=roles/secretmanager.secretAccessor --project=$PROJECT_ID\r\ngcloud secrets add-iam-policy-binding gitlab-webhook-token --member=$CLOUD_BUILD_SA_MEMBER --role=roles/secretmanager.secretAccessor --project=$PROJECT_ID'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dc0335d0>)])]

Configuring the Cloud Build Repository and triggers with Terraform

With the authentication credentials configured in Secret Manager we can move on to the Terraform configuration. For simplicity we put everything in a single main.tf file that looks as follows:

code_block

[StructValue([(u'code', u'variable "project_id" {\r\n type = string\r\n}\r\n\r\nvariable "gitlab_api_token_secret" {\r\n type = string\r\n}\r\n\r\nvariable "gitlab_read_api_token_secret" {\r\n type = string\r\n}\r\n\r\nvariable "gitlab_webhook_token_secret" {\r\n type = string\r\n}\r\n\r\nvariable "gitlab_repo_uri" {\r\n type = string\r\n}\r\n\r\nvariable "build_location" {\r\n type = string\r\n default = "europe-west1"\r\n}\r\n\r\nprovider "google" {\r\n project = var.project_id\r\n}\r\n\r\nresource "google_cloudbuildv2_connection" "gitlab-connection" {\r\n location = var.build_location\r\n name = "gitlab-connection"\r\n\r\n gitlab_config {\r\n authorizer_credential {\r\n user_token_secret_version = var.gitlab_api_token_secret\r\n }\r\n read_authorizer_credential {\r\n user_token_secret_version = var.gitlab_read_api_token_secret\r\n }\r\n webhook_secret_secret_version = var.gitlab_webhook_token_secret\r\n }\r\n}\r\n\r\nresource "google_cloudbuildv2_repository" "demo-repo" {\r\n name = "gitlab-demo-repo"\r\n location = var.build_location\r\n parent_connection = google_cloudbuildv2_connection.gitlab-connection.id\r\n remote_uri = var.gitlab_repo_uri\r\n}\r\n\r\nresource "google_cloudbuild_trigger" "demo-trigger" {\r\n location = var.build_location\r\n repository_event_config {\r\n repository = google_cloudbuildv2_repository.demo-repo.id\r\n push {\r\n branch = ".*"\r\n }\r\n }\r\n filename = "cloudbuild.yaml"\r\n}'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dc033ad0>)])]

We provide variables for the Google Cloud project ID, references to the externally created secrets in Secret Manager, and an optional location override for the Cloud Build resources.

The Cloud Build specific Terraform resources are:

• google_cloudbuildv2_connection to specify the host connection with a name, region and the credentials. Host connections can be used by multiple repositories so the credentials can be managed centrally.

• google_cloudbuildv2_repository to specify the Gitlab repo we want to use and associate it with a host connection. This requires that the host connections’ credentials have access to the repository specified as remote_uri.

• google_cloudbuild_trigger to run the Cloud Build pipeline on push events on any branch in the Gitlab repository.

To apply the Terraform configuration we execute the following two commands from within the folder that contains our main.tf file:

code_block

[StructValue([(u'code', u'terraform init\r\nterraform apply \\\r\n --var=project_id="$PROJECT_ID" \\\r\n --var=gitlab_api_token_secret="$GITLAB_API_TOKEN_SECRET_REF" \\\r\n --var=gitlab_read_api_token_secret="$GITLAB_READ_API_TOKEN_SECRET_REF" \\\r\n --var=gitlab_webhook_token_secret="$GITLAB_WEBHOOK_TOKEN_SECRET_REF" \\\r\n --var=gitlab_repo_uri=$GITLAB_REPO_URI'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dc033e10>)])]

Once the resources are created, we can see them in the Google Cloud Console under Cloud Build > Repositories:

To test our simple pipeline and the repo trigger we can push an empty commit to our sample repo:

code_block

[StructValue([(u'code', u'cd cloud-build-demo\r\ngit commit -m "trigger pipeline" --allow-empty\r\ngit push -u gitlab main'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e32dd1950d0>)])]

In the Cloud Build dashboard in the Google Cloud Console you can see the build that was kicked off.

The same result is visible in the Gitlab UI under Builds > Pipelines.

Next steps

You can find more information about Cloud Build repositories and a side by side comparison of the features of the Gen 1 and Gen 2 repositories in the official Cloud Build repositories documentation. If you are planning to use Cloud Build with a Git repository hosted on GitHub, you should follow these instructions for your Terraform configuration.