2024-10-23 - Filedrop queueing for large volumes of files - take 2¶
Release¶
AgileData.io Release: Streamlined Data Loading for Large File Batches
What’s New:
We’ve made a significant improvement to how AgileData.io handles large batches of files. Previously, if you dropped multiple files at once, they would all be processed simultaneously, which could lead to performance issues, especially when dealing with hundreds of files.
The Problem:
Imagine you have a large batch of files, like 50 audit logs, that you need to load into your “audit logs” table. Our previous system would try to load all 50 files at the same time, potentially causing delays and conflicts.
The Solution:
We’ve introduced a new file queue system! Now, when you drop a batch of files, they are added to a queue and processed one at a time. This ensures smooth and efficient loading, even for large volumes of data.
How it Works:
File Queue: We’ve implemented a queue in our database to manage the order of file processing.
Controlled Processing: Files are loaded one by one, ensuring that each file is fully processed before moving on to the next.
Scalability: We leverage Google Spanner’s powerful transactional processing and record locking to handle multiple pipelines simultaneously accessing the queue.
Benefits for You:
Faster Loading: Experience smoother and faster data loading, even with large batches of files.
Improved Performance: Avoid performance bottlenecks and ensure consistent data processing.
Enhanced Reliability: The queue system ensures that all files are processed correctly, preventing data loss or errors.
Enjoy the benefits of this new feature and experience a more efficient and reliable data loading process with AgileData.io!
Last Refreshed¶
Doc Refreshed: 2025-01-15