Skip to content

Navigation Menu

Sign in
Sign up

Does using a separate billing project for BigQuery LOAD jobs split concurrent job quota across projects? #16114

Unanswered
y0zg asked this question in Q&A
Discussion options

We're hitting BigQuery's concurrent LOAD jobs limit on a single GCP project.

We implemented a pattern where we create bigquery.Client(project=runner_project) using separate GCP projects as billing projects, while destination tables remain in the original project. We rotate across multiple runner projects randomly.

INFORMATION_SCHEMA.JOBS_BY_PROJECT on the runner projects confirms jobs are recorded there.

Question: Does the billing project specified in bigquery.Client(project=...) determine which project's concurrent job quota is consumed for LOAD jobs? Is this a valid approach to scale past the concurrent LOAD jobs limit by distributing across multiple GCP projects?

Cloud Monitoring bigquery/job/num_in_flight only shows data on the destination project, making quota attribution hard to confirm visually

You must be logged in to vote

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
1 participant

AltStyle によって変換されたページ (->オリジナル) /