2024-10-23 - Handling deleted records from source¶
Problem¶
When we load from source systems that ‘delete’ records we receive change records that are flagged as deleted.
Data processing needs to update the records we have already loaded with a ‘deleted’ flag so that the historical data matches source.
Solution¶
By making a small change to the data load templates (sql) we check for ‘deleted’ records and update the latest record we have loaded for the specified business key and mark it as ‘deleted’ (boolean).
This preserves all the historical change records up until that point then marks that record as deleted as it wont change again. Users can choose to use deleted records downstream or exclude them depending on the reporting use case.
Leverage the Magic¶
This is where templated patterns really shine ! Changing the way deleted records are loaded, we can just update a jinja template so that the next time its used to create a load rule, the new pattern is applied.
ADI¶
Now thats magical !
Last Refreshed¶
Doc Refreshed: 2024-11-21