Build and run a Flex Template

Dataflow Flex Templates allow you to package a Dataflow pipeline for deployment. This tutorial shows you how to build a Dataflow Flex Template and then run a Dataflow job using that template.

Objectives

  • Build a Dataflow Flex Template.
  • Use the template to run a Dataflow job.

Costs

In this document, you use the following billable components of Google Cloud Platform:

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud Platform account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get 300ドル in free credits to run, test, and deploy workloads.

  2. Install the Google Cloud CLI.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command: 

    gcloudinit

  5. Create or select a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Verify that billing is enabled for your Google Cloud project.

  7. Enable the Dataflow, Compute Engine, Logging, Cloud Storage, Cloud Storage JSON, Resource Manager, Artifact Registry, and Cloud Build API:

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles. 

    gcloudservicesenabledataflow compute_component logging storage_component storage_api cloudresourcemanager.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com

  8. If you're using a local shell, then create local authentication credentials for your user account: 

    gcloudauthapplication-defaultlogin

    You don't need to do this if you're using Cloud Shell.

    If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.

  9. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser 

    gcloudprojectsadd-iam-policy-bindingPROJECT_ID--member="user:USER_IDENTIFIER"--role=ROLE

    Replace the following:

    • PROJECT_ID: Your project ID.
    • USER_IDENTIFIER: The identifier for your user account. For example, myemail@example.com.
    • ROLE: The IAM role that you grant to your user account.

  10. Install the Google Cloud CLI.

  11. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  12. To initialize the gcloud CLI, run the following command: 

    gcloudinit

  13. Create or select a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  14. Verify that billing is enabled for your Google Cloud project.

  15. Enable the Dataflow, Compute Engine, Logging, Cloud Storage, Cloud Storage JSON, Resource Manager, Artifact Registry, and Cloud Build API:

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles. 

    gcloudservicesenabledataflow compute_component logging storage_component storage_api cloudresourcemanager.googleapis.com artifactregistry.googleapis.com cloudbuild.googleapis.com

  16. If you're using a local shell, then create local authentication credentials for your user account: 

    gcloudauthapplication-defaultlogin

    You don't need to do this if you're using Cloud Shell.

    If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.

  17. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser 

    gcloudprojectsadd-iam-policy-bindingPROJECT_ID--member="user:USER_IDENTIFIER"--role=ROLE

    Replace the following:

    • PROJECT_ID: Your project ID.
    • USER_IDENTIFIER: The identifier for your user account. For example, myemail@example.com.
    • ROLE: The IAM role that you grant to your user account.
    18.

  18. Grant roles to your Compute Engine default service account. Run the following command once for each of the following IAM roles:

    • roles/dataflow.admin
    • roles/dataflow.worker
    • roles/storage.objectAdmin
    • roles/artifactregistry.writer
    gcloudprojectsadd-iam-policy-bindingPROJECT_ID--member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com"--role=SERVICE_ACCOUNT_ROLE

    Replace the following:

    • PROJECT_ID: your project ID
    • PROJECT_NUMBER your project number
    • SERVICE_ACCOUNT_ROLE: each individual role

Prepare the environment

Install the SDK and any requirements for your development environment.

Java

  1. Download and install the Java Development Kit (JDK) version 17. Verify that the JAVA_HOME environment variable is set and points to your JDK installation.

  2. Download and install Apache Maven by following Maven's installation guide for your specific operating system.

Python

Install the Apache Beam SDK for Python.

Go

Use Go's Download and install guide to download and install Go for your specific operating system. To learn which Go runtime environments are supported by Apache Beam, see Apache Beam runtime support.

Download the code sample.

Java

  1. Clone the java-docs-samples repository.

    gitclonehttps://github.com/GoogleCloudPlatform/java-docs-samples.git

  2. Navigate to the code sample for this tutorial.

    cdjava-docs-samples/dataflow/flex-templates/getting_started

  3. Build the Java project into an Uber JAR file.

    mvncleanpackage

    This Uber JAR file has all the dependencies embedded in it. You can run this file as a standalone application with no external dependencies on other libraries.

Python

  1. Clone the python-docs-samples repository.

    gitclonehttps://github.com/GoogleCloudPlatform/python-docs-samples.git

  2. Navigate to the code sample for this tutorial.

    cdpython-docs-samples/dataflow/flex-templates/getting_started

Go

  1. Clone the golang-samples repository.

    gitclonehttps://github.com/GoogleCloudPlatform/golang-samples.git

  2. Navigate to the code sample for this tutorial.

    cdgolang-samples/dataflow/flex-templates/wordcount

  3. Compile the Go binary.

    CGO_ENABLED=0GOOS=linuxGOARCH=amd64gobuild-owordcount.

Create a Cloud Storage bucket

Use the gcloud storage buckets create command to create a Cloud Storage bucket:

gcloudstoragebucketscreategs://BUCKET_NAME

Replace BUCKET_NAME with a name for your Cloud Storage bucket. Cloud Storage bucket names must be globally unique and meet the bucket naming requirements.

Create an Artifact Registry repository

Create an Artifact Registry repository where you will push the Docker container image for the template.

  1. Use the gcloud artifacts repositories create command to create a new Artifact Registry repository.

    gcloudartifactsrepositoriescreateREPOSITORY\
--repository-format=docker\
--location=LOCATION

    Replace the following:

    • REPOSITORY: a name for your repository. Repository names must be unique for each repository location in a project.
    • LOCATION: the regional or multi-regional location for the repository.

  2. Use the gcloud auth configure-docker command to configure Docker to authenticate requests for Artifact Registry. This command updates your Docker configuration, so that you can connect with Artifact Registry to push images.

    gcloudauthconfigure-dockerLOCATION-docker.pkg.dev

Flex Templates can also use images stored in private registries. For more information, see Use an image from a private registry.

Build the Flex Template

In this step, you use the gcloud dataflow flex-template build command to build the Flex Template.

A Flex Template consists of the following components:

  • A Docker container image that packages your pipeline code. For Java and Python Flex Templates, the Docker image is built and pushed to your Artifact Registry repository when you run the gcloud dataflow flex-template build command.
  • A template specification file. This file is a JSON document that contains the location of the container image plus metadata about the template, such as pipeline parameters.

The sample repository in GitHub contains the metadata.json file.

To extend your template with additional metadata, you can create your own metadata.json file.

Java 

gclouddataflowflex-templatebuildgs://BUCKET_NAME/getting_started-java.json\
--image-gcr-path"LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/getting-started-java:latest"\
--sdk-language"JAVA"\
--flex-template-base-imageJAVA17\
--metadata-file"metadata.json"\
--jar"target/flex-template-getting-started-1.0.jar"\
--envFLEX_TEMPLATE_JAVA_MAIN_CLASS="com.example.dataflow.FlexTemplateGettingStarted"

Replace the following:

  • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
  • LOCATION: the location
  • PROJECT_ID: the Google Cloud project ID
  • REPOSITORY: the name of the Artifact Registry repository that you created earlier

Python 

gclouddataflowflex-templatebuildgs://BUCKET_NAME/getting_started-py.json\
--image-gcr-path"LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/getting-started-python:latest"\
--sdk-language"PYTHON"\
--flex-template-base-image"PYTHON3"\
--metadata-file"metadata.json"\
--py-path"."\
--env"FLEX_TEMPLATE_PYTHON_PY_FILE=getting_started.py"\
--env"FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE=requirements.txt"

Replace the following:

  • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
  • LOCATION: the location
  • PROJECT_ID: the Google Cloud project ID
  • REPOSITORY: the name of the Artifact Registry repository that you created earlier

Go

  1. Use the gcloud builds submit command to build the Docker image using a Dockerfile with Cloud Build. This command builds the file and pushes it to your Artifact Registry repository.

    gcloudbuildssubmit--tagLOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/dataflow/wordcount-go:latest.

    Replace the following:

    • LOCATION: the location
    • PROJECT_ID: the Google Cloud project ID
    • REPOSITORY: the name of the Artifact Registry repository that you created earlier

  2. Use the gcloud dataflow flex-template build command to create a Flex Template named wordcount-go.json in your Cloud Storage bucket.

    gclouddataflowflex-templatebuildgs://BUCKET_NAME/samples/dataflow/templates/wordcount-go.json\
--image"LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/dataflow/wordcount-go:latest"\
--sdk-language"GO"\
--metadata-file"metadata.json"

    Replace BUCKET_NAME with the name of the Cloud Storage bucket that you created earlier.

Run the Flex Template

In this step, you use the template to run a Dataflow job.

Java

  1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

    gclouddataflowflex-templaterun"getting-started-`date +%Y%m%d-%H%M%S`"\
--template-file-gcs-location"gs://BUCKET_NAME/getting_started-java.json"\
--parametersoutput="gs://BUCKET_NAME/output-"\
--additional-user-labels"LABELS"\
--region"REGION"

    Replace the following:

    • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
    • REGION: the region
    • LABELS: Optional. Labels attached to your job, using the format <key1>=<val1>,<key2>=<val2>,...

  2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

    Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/output--00000-of-00001.txt in your Cloud Storage bucket.

Python

  1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

    gclouddataflowflex-templaterun"getting-started-`date +%Y%m%d-%H%M%S`"\
--template-file-gcs-location"gs://BUCKET_NAME/getting_started-py.json"\
--parametersoutput="gs://BUCKET_NAME/output-"\
--additional-user-labels"LABELS"\
--region"REGION"

    Replace the following:

    • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
    • REGION: the region
    • LABELS: Optional. Labels attached to your job, using the format <key1>=<val1>,<key2>=<val2>,...

  2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

    Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/output--00000-of-00001.txt in your Cloud Storage bucket.

Go

  1. Use the gcloud dataflow flex-template run command to run a Dataflow job that uses the Flex Template.

    gclouddataflowflex-templaterun"wordcount-go-`date +%Y%m%d-%H%M%S`"\
--template-file-gcs-location"gs://BUCKET_NAME/samples/dataflow/templates/wordcount-go.json"\
--parametersoutput="gs://BUCKET_NAME/samples/dataflow/templates/counts.txt"\
--additional-user-labels"LABELS"\
--region"REGION"

    Replace the following:

    • BUCKET_NAME: the name of the Cloud Storage bucket that you created earlier
    • REGION: the region
    • LABELS: Optional. Labels attached to your job, using the format <key1>=<val1>,<key2>=<val2>,...

  2. To view the status of the Dataflow job in the Google Cloud console, go to the Dataflow Jobs page.

    Go to Jobs

If the job runs successfully, it writes the output to a file named gs://BUCKET_NAME/samples/dataflow/templates/count.txt in your Cloud Storage bucket.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

    Delete a Google Cloud project:

    gcloud projects delete PROJECT_ID

Delete individual resources

  1. Delete the Cloud Storage bucket and all the objects in the bucket. 
    gcloudstoragermgs://BUCKET_NAME--recursive
  2. Delete the Artifact Registry repository. 
    gcloudartifactsrepositoriesdeleteREPOSITORY\
--location=LOCATION
  3. Revoke the roles that you granted to the Compute Engine default service account. Run the following command once for each of the following IAM roles:
    • roles/dataflow.admin
    • roles/dataflow.worker
    • roles/storage.objectAdmin
    • roles/artifactregistry.writer
    gcloudprojectsremove-iam-policy-bindingPROJECT_ID\
--member=serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com\
--role=SERVICE_ACCOUNT_ROLE

  4. Optional: Revoke the authentication credentials that you created, and delete the local credential file. 

    gcloudauthapplication-defaultrevoke

  5. Optional: Revoke credentials from the gcloud CLI. 

    gcloudauthrevoke

What's next

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025年11月13日 UTC.