2024-10-23 - Handling deleted records from source¶

Release¶

AgileData.io Release: Handling Deleted Records from Source

This release brings a powerful new feature to AgileData.io, allowing us to accurately reflect deleted records from your source systems.

The Problem:

Previously, when your source systems deleted records, AgileData.io would receive change records flagged as deleted. This meant that historical data in AgileData.io might not accurately reflect the state of your source systems.

The Solution:

We’ve implemented a smart solution that updates existing records in AgileData.io with a “deleted” flag when we receive a deletion notification from your source system. This ensures that your historical data remains consistent with your source systems.

How it Works:

We’ve made a small change to our data load templates (SQL) to check for deleted records. When a deleted record is detected, we update the latest record we have loaded for that specific business key and mark it as “deleted” (a simple boolean flag).

Benefits for You:

Accurate Historical Data: Your historical data in AgileData.io will now accurately reflect the state of your source systems, even when records are deleted.
Flexibility in Reporting: You can choose to include or exclude deleted records in your reports, depending on your specific needs.
Simplified Data Management: This new feature streamlines data management by automatically handling deleted records, saving you time and effort.

Leveraging Templated Patterns:

This update leverages our powerful templated patterns. By simply updating a Jinja template, the new pattern will be applied to all future data load rules, ensuring consistent handling of deleted records across your data pipelines.

This release empowers you with greater control and accuracy over your data, ensuring that your insights are always based on the most up-to-date information.

Last Refreshed¶

Doc Refreshed: 2025-01-15