You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We're hitting BigQuery's concurrent LOAD jobs limit on a single GCP project.
We implemented a pattern where we create bigquery.Client(project=runner_project) using separate GCP projects as billing projects, while destination tables remain in the original project. We rotate across multiple runner projects randomly.
INFORMATION_SCHEMA.JOBS_BY_PROJECT on the runner projects confirms jobs are recorded there.
Question: Does the billing project specified in bigquery.Client(project=...) determine which project's concurrent job quota is consumed for LOAD jobs? Is this a valid approach to scale past the concurrent LOAD jobs limit by distributing across multiple GCP projects?
Cloud Monitoring bigquery/job/num_in_flight only shows data on the destination project, making quota attribution hard to confirm visually
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
We're hitting BigQuery's concurrent LOAD jobs limit on a single GCP project.
We implemented a pattern where we create
bigquery.Client(project=runner_project)using separate GCP projects as billing projects, while destination tables remain in the original project. We rotate across multiple runner projects randomly.INFORMATION_SCHEMA.JOBS_BY_PROJECT on the runner projects confirms jobs are recorded there.
Question: Does the billing project specified in
bigquery.Client(project=...)determine which project's concurrent job quota is consumed for LOAD jobs? Is this a valid approach to scale past the concurrent LOAD jobs limit by distributing across multiple GCP projects?Cloud Monitoring
bigquery/job/num_in_flightonly shows data on the destination project, making quota attribution hard to confirm visuallyBeta Was this translation helpful? Give feedback.
All reactions