2024-10-23 - Event ensembles - late arriving details¶
Problem¶
Customers may have applications that update their data using disconnected processes, that result in record created and updated timestamps that are different between tables - for example, a customer may be created at 9am, but the address table wont have a record created until 9:30am by a different process.
We load the customer and address tables using the record created and updated timestamps to determine the business ‘effectivity’ timestamp of each record so we can track changes over time.
When we go to join customer to address we have dont have an address record that was effective when the customer was created (because the address was created later)
Solution¶
To get around sources that update their data independantly we have updated the join templates to join on the ‘closest’ effective record for the key, for example - joining customer to address at the time the customer was created, we look for the closest address record (which may have been created a second or a minute later)
Leverage the Magic¶
This is where templated patterns really shine ! Changing the way rules and ensembles join tables is complex, but with our pattern architecture we can just update a jinja template so that the next time its used to create a change rule, the new pattern is applied.
ADI¶
Now thats magical !
Last Refreshed¶
Doc Refreshed: 2024-11-21