-
Notifications
You must be signed in to change notification settings - Fork 1.8k
We're hitting BigQuery's concurrent LOAD jobs limit on a single GCP project.
We implemented a pattern where we create bigquery.Client(project=runner_project) using separate GCP projects as billing projects, while destination tables remain in the original project. We rotate across multiple runner projects randomly.
INFORMATION_SCHEMA.JOBS_BY_PROJECT on the runner projects confirms jobs are recorded there.
Question: Does the billing project specified in bigquery.Client(project=...) determine which project's concurrent job quota is consumed for LOAD jobs? Is this a valid approach to scale past the concurrent LOAD jobs limit by distributing across multiple GCP projects?
Cloud Monitoring bigquery/job/num_in_flight only shows data on the destination project, making quota attribution hard to confirm visually