2024-10-23 - BigQuery Transient Errors¶
Release¶
Improved Data Processing for Large Datasets
This release focuses on improving the reliability of data processing for large datasets, especially those from platforms like Google Analytics 4 (GA4).
The Issue:
When you split a large event table into multiple concepts and details, our system starts all the associated BigQuery pipelines simultaneously. Occasionally, BigQuery might experience temporary (transient) errors during this simultaneous startup. These errors could cause some pipelines to fail, leading to incomplete data processing.
The Solution:
We’ve implemented a new feature that automatically detects and resolves these transient errors. Now, whenever a BigQuery job completes, our system checks for specific error messages indicating a transient issue. If detected, the failed job is automatically restarted, ensuring it runs to completion and your data is processed correctly.
What This Means for You:
You can now enjoy more reliable and consistent data processing, even when working with large datasets. This means less downtime and fewer interruptions, allowing you to focus on analyzing your data and gaining valuable insights.
Last Refreshed¶
Doc Refreshed: 2025-01-15