cloud-build-watcher

Build Status

This repository contains a Cloud Function that watches for Cloud Build messages via a Pub/Sub topic and takes actions in response to them. Currently, those actions include sending email notifications and writing SVG badge images (like the one above this paragraph) and brief HTML reports to a Cloud Storage bucket.

I wrote this because I was setting up automated testing and deployment using Cloud Build and was dismayed by how hard it was to do seemingly-simple things like sending email notifications about failed builds or displaying build status badges.

Deploying

Replace $PROJECT_ID in the following commands with the Google Cloud Project ID of the project in which the builds will be performed, or omit the --project flag if you only have a single project.

Replace $REGION with the Google Cloud Project region to which you want the function to be deployed.

You may need to create a cloud-builds Pub/Sub topic first. Cloud Build will allegedly send messages to it automatically.

gcloud --project=$PROJECT_ID pubsub topics create cloud-builds

The WatchBuilds Cloud Function can be deployed by running a command similar to the following. Deploying a Cloud Function can take several minutes.

gcloud --project=$PROJECT_ID run deploy watchbuilds \
 --function=WatchBuilds --region=$REGION --source=. \
 --base-image=go125 --min-instances=0 --max-instances=1

After deploying the function, add an Eventarc trigger via the Cloud Run page that receives events from the cloud-builds topic, uses watchbuilds as the destination, and uses the "Cloud Run Pub/Sub Invoker" service account.

You may also need to assign the "Cloud Run Invoker" role to the "Cloud Run Pub/Sub Invoker" service account via the Cloud Console IAM page to prevent errors like the following:

The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header. Read more at https://cloud.google.com/run/docs/securing/authenticating

Additional troubleshooting documentation can be found at: https://cloud.google.com/run/docs/troubleshooting#unauthorized-client

If you want to generate badge images, you'll also need to create a publicly-readable Cloud Storage bucket. I recommend following the "Creating a bucket" and "Sharing your files" steps from the Hosting a static website document. You may need to give the Cloud Function permission to write to the bucket if it doesn't already have it.

Configuration

Configuration is performed via environment variables, which can be passed via the command line when deploying the function (via the --set-env-vars or --env-vars-file flags) or configured via the Google Cloud Console (by navigating to your project's Cloud Functions page, clicking the WatchBuilds function, clicking "Edit", and expanding the "Runtime, build, connections and security settings" section). Note that the --set-env-vars flag won't work with values that contain commas since it interprets commas as delimiters between variables.

See Using Environment Variables for more information about setting environment variables for Cloud Functions.

How email is sent

At least EMAIL_HOSTNAME and EMAIL_PORT must be set in order for email to be sent.

Where email is sent (and related details)

EMAIL_FROM and EMAIL_RECIPIENTS must be set in order for email to be sent.

Which build events result in email

Items in the three above lists are separated by commas with optional spaces.

If either EMAIL_BUILD_TRIGGER_IDS or EMAIL_BUILD_TRIGGER_NAMES is supplied, email is only sent for events originating from a trigger in either list. Globs in EMAIL_BUILD_TRIGGER_NAMES are evaluated using filepath.Match.

Badge images and reports

If the BADGE_BUCKET environment variable is set, SVG images will be written to Cloud Storage for SUCCESS, FAILURE, INTERNAL_ERROR, and TIMEOUT build statuses.

Badge filenames take the form <trigger-id>.svg and can be accessed via a URL like https://storage.googleapis.com/<bucket>/<trigger-id>.svg. To find a trigger's ID in the Cloud Console, go to Cloud Build, click "Triggers" in the sidebar, click on the trigger's name, and look for a long hexadecimal-and-dashes ID in the URL.

If the BADGE_REPORTS variable is set to 1 or true, brief HTML build reports will be written to BADGE_BUCKET alongside badge images as <trigger-id>.html. These reports intentionally contain minimal information about builds (to avoid leaking internal details): just status, start time, end time, and duration.

Markdown similar to the following can be used to display a badge image that links to the corresponding build report:

[![Build Status](https://storage.googleapis.com/<bucket>/<trigger-id>.svg)](https://storage.googleapis.com/<bucket>/<trigger-id>.html)

Customization

Email notifications can be customized by modifying the BuildEmail function and textTemplate and htmlTemplate constants in email.go. The test_email program can be used to send example notifications so you can see how they're rendered by your email client.

Badge images and HTML reports can be customized by modifying the badgeLeft and badgeStatuses variables and the badgeTemplate and reportTemplate constants in badge.go. The test_badge program can be used to generate badges and save them locally so you can see what they look like.

Seeing more information

The Cloud Function logs information about what it's doing, including why it chooses not to send email about build messages. You can see this information via the Cloud Console (go to Cloud Functions, click on the WatchBuilds function, and click the "Logs" tab) or by running a command like the following:

gcloud --project=$PROJECT_ID functions logs read WatchBuilds

A rant

Google provides a Configuring SMTP notifications page with instructions for setting up email notifications for Cloud Build. It sends you on a whirlwind tour of Google Cloud products that have nothing to do with build notifications:

• Save your email password to Secret Manager.
• Use IAM to find your Cloud Build (or is it Cloud Run?) service account.
• Go back to Secret Manager to give the service account access to the password.
• Go back to IAM to give the service account read and write access to Cloud Storage. (?!)
• Write a bespoke YAML configuration file for the notifier (which you'll have a hard time finding on your computer when you want to change the config later).
• Copy your YAML file to Cloud Storage, maybe creating a new bucket in the process (there's no guidance on where you should put it).
• Deploy a prebuilt Docker image containing the notifier to Cloud Run from the command line.
• Run a weird add-iam-policy-binding command to give Pub/Sub some permissions.
• Create a service account "to represent your Pub/Sub subscription identity".
• Run another weird add-iam-policy-binding command to give more permissions.
• Create a cloud-builds Pub/Sub topic.
• Create a Pub/Sub push subscriber for the notifier.

You'll also need to learn enough of a Google-designed non-Turing-complete language called CEL to write expressions describing which build events you want to be notified about.

There's a different Automating configuration for notifications page which contains a subset of the above instructions and points you at the cloud-build-notifiers repo, which contains the source code for the notifier used on the other page plus a setup.sh script that tries to automate many of the previous steps. It prints a lot of output, including many (harmless?) errors when I run it.

Assuming it works, you'll end up with bare-bones email notifications about builds. I tried to add some more details, but to use my code, I had to build a new Docker container in the (deprecated?) Container Registry and hack the setup.sh script to use it.

For some reason, I'm reminded of "So, how do I query the database?".

Just to mention it, here's how I configured email notifications in Travis:

notifications:email:on_success:neveron_failure:always