2024-01-16 - Loads triggered more than once¶
Problem¶
If a load is triggered multiple times then we insert duplicates as the second load will insert the same data as the first because the first hasnt committed its data yet ……
Solution¶
Extend the bigquery_job_completed function to use the bq_job_manifest config table, add a job_completed attribute (boolean) that we can check and mark as ‘completed’ the first time we see a job_id, which prevents that job_if being processed twice
Leverage the Magic¶
A small tweak to the bigquery_job_completed cloud function (to look up a job_completed flag)
Last Refreshed¶
Doc Refreshed: 2024-03-02