2026-01-10 - Duplicate File Detection Improvement¶
Release¶
AgileData.io - Bug Fix: Smarter Duplicate File Detection
We improved our ability to detect duplicate files in your file drops, preventing accidental reprocessing of the same data.
What was happening:
Files with the same content but different names could be processed multiple times
Checksum-based detection wasn’t catching all duplicate scenarios
What we did:
Enhanced duplicate detection to compare actual file bytes and size
Added checks for files with same size and target but different names
Improved logic to identify truly identical files regardless of naming
What this means for you:
Prevents accidentally processing the same data multiple times even if filenames differ
More reliable data loading with intelligent duplicate detection
Reduced risk of data duplication in your pipelines
Last Refreshed¶
Doc Refreshed: 2026-01-10