2024-10-23 - BigQuery Transient Errors¶

Release¶

Status: Available

Type: Bug-Squash

Date: 2024-10-23

Problem¶

When we split a large event table (eg GA4) into multiple concepts and details, our config starts all of those BigQuery pipelines simultaneously. Occasionally BQ will throw transient errors for this simultaneous startups.

Solution¶

By checking the job status for transient error messages we can restart any failed jobs so they run to completion successfully.

Leverage the Magic¶

We leverage the existing check_job_status that runs whenever a job completes, and simply re-submit the failed job to run again.

ADI¶

Now thats magical !

Last Refreshed¶

Doc Refreshed: 2024-11-21