Workflow scheduling solutions

This section describes Google Cloud options you can use to schedule workflows.

Dataproc Workflow Templates

Managed Service for Apache Spark Workflow templates provide a flexible and easy-to-use mechanism for managing and executing workflows. A Workflow Template is a reusable workflow configuration. It defines a graph of jobs with information on where to run those jobs.

Cloud Scheduler

Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, and Cloud infrastructure operations. It provides simple time-based scheduling, for example, daily or hourly, without requiring you to write code.

Advantages:

• Enables time-based instantiation of workflow templates based on familiar cron expressions

• No code to write

Tutorial: Workflow using Cloud Scheduler

Cloud Functions

Cloud Run functions is a lightweight compute solution you can use to create single-purpose, stand-alone functions that respond to Cloud events without the need to manage a server or runtime environment. You can use Cloud Run functions to launch Workflows in response to Pub/Sub events or file changes in Cloud Storage. You can use Cloud Run functions with Cloud Scheduler for workflows that require the calculation of time-based parameters.

Advantages:

• Enables workflow instantiation in response to data events, such as new files in Cloud Storage or Pub/Sub events.

• Minimal coding required using Managed Service for Apache Spark Go, Node.js, or Python client libraries

• Dynamically generate workflows and workflow parameters

Tutorial: Workflow using Cloud Run functions

Cloud Composer

Managed Airflow is a managed Apache Airflow service you can use to create, schedule, monitor, and manage workflows.

Advantages:

• Supports time- and event-based scheduling

• Simplified calls to Managed Service for Apache Spark using Operators

• Dynamically generate workflows and workflow parameters

• Build data flows that span multiple Google Cloud products

Tutorial: Workflow using Managed Service for Apache Airflow