2023-04-11 - Multi File Drop File Queuing¶
Problem¶
Sometimes Data Magicians drop lots of files at the same time causing upstream issues with appending and loading data.
Solution¶
Deploy a file queue pattern that queues 1 to many dropped files then loads them sequentially.
Leverage the Magic¶
Data Magicians can drop as many files as they like and they get managed automagically.
ADI¶
Bazinga!, this is a truly magical pattern that takes care of file loading and appending data safely.
Customer¶
On point!, I love being able to just upload all my data files in one go and and they get loaded safely.
Magician Partner¶
Deploy a new file queueing pattern which wraps the existing create_landing_table function, to handle multiple dropped files going into the same target. Files are queued into a separate storage bucket then de-queued one at a time back into the filedrop bucket.
Last Refreshed¶
Doc Refreshed: 2024-05-23