Table Of Contents
Table Of Contents

2023-04-11 - Multi File Drop File Queuing


Status: Available

Type: DataOps

Date: 2023-04-11


Sometimes Data Magicians drop lots of files at the same time causing upstream issues with appending and loading data.


Deploy a file queue pattern that queues 1 to many dropped files then loads them sequentially.

Leverage the Magic

Data Magicians can drop as many files as they like and they get managed automagically.


Bazinga!, this is a truly magical pattern that takes care of file loading and appending data safely.


On point!, I love being able to just upload all my data files in one go and and they get loaded safely.

Magician Partner

Deploy a new file queueing pattern which wraps the existing create_landing_table function, to handle multiple dropped files going into the same target. Files are queued into a separate storage bucket then de-queued one at a time back into the filedrop bucket.

Last Refreshed

Doc Refreshed: 2024-03-02