gcloud beta dataflow jobs update-options

gcloud beta dataflow jobs update-options - update pipeline options on-the-fly for running Dataflow jobs

gcloud beta dataflow jobs update-options JOB_ID [--max-num-workers =MAX_NUM_WORKERS] [--min-num-workers =MIN_NUM_WORKERS] [--region =REGION_ID] [--unset-worker-utilization-hint | --worker-utilization-hint =WORKER_UTILIZATION_HINT] [GCLOUD_WIDE_FLAG …]

Adjust the autoscaling settings for Streaming Engine Dataflow jobs by providing at-least one of --min-num-workers or --max-num-workers or --worker-utilization-hint (or all 3), or --unset-worker-utilization-hint (which cannot be run at the same time as --worker-utilization-hint but works with the others). Allow a few minutes for the changes to take effect.

Note that autoscaling settings can only be modified on-the-fly for Streaming Engine jobs. Attempts to modify batch job or Streaming Appliance jobs will fail.

gcloudbetadataflowjobsupdate-options--min-num-workers=5--max-num-workers=10

Require a job to use at least 2 workers:

gcloudbetadataflowjobsupdate-options--min-num-workers=2

Require a job to use at most 20 workers:

gcloudbetadataflowjobsupdate-options--max-num-workers=20

Adjust the hint of target worker utilization to 70% for horizontal autoscaling:

gcloudbetadataflowjobsupdate-options--worker-utilization-hint=0.7

"Unset" worker utilization hint so that horizontal scaling will rely on its default CPU utilization target:

gcloudbetadataflowjobsupdate-options--unset-worker-utilization-hint

JOB_ID

Job ID to operate on.

--max-num-workers=MAX_NUM_WORKERS

Upper-bound for autoscaling, between 1-1000. Only supported for streaming-engine jobs.

--min-num-workers=MIN_NUM_WORKERS

Lower-bound for autoscaling, between 1-1000. Only supported for streaming-engine jobs.

--region=REGION_ID

Region ID of the job's regional endpoint. Defaults to 'us-central1'.

At most one of these can be specified:

--unset-worker-utilization-hint

Unset --worker-utilization-hint. This causes the job autoscaling to fall back to internal tunings if they exist, or otherwise use the default hint value.

--worker-utilization-hint=WORKER_UTILIZATION_HINT

Target CPU utilization for autoscaling, ranging from 0.1 to 0.9. Only supported for streaming-engine jobs with autoscaling enabled.

Run $ gcloud help for details.