Table Of Contents
Table Of Contents

2024-01-16 - Loads triggered more than once

Release

Status: Available

Type: Bug-Squash

Date: 2024-01-16

Problem

If a load is triggered multiple times then we insert duplicates as the second load will insert the same data as the first because the first hasnt committed its data yet ……

Solution

Extend the bigquery_job_completed function to use the bq_job_manifest config table, add a job_completed attribute (boolean) that we can check and mark as ‘completed’ the first time we see a job_id, which prevents that job_if being processed twice

Leverage the Magic

A small tweak to the bigquery_job_completed cloud function (to look up a job_completed flag)

Last Refreshed

Doc Refreshed: 2024-05-20