.. _2023-04-11-multi-file-drop-file-queuing_release:

=================================================================================
2023-04-11 - Multi File Drop File Queuing
=================================================================================

Release
==================
*Status: Available*

*Type: DataOps*

*Date: 2023-04-11*

Problem
==================
Sometimes Data Magicians drop lots of files at the same time causing upstream issues with appending and loading data.

Solution
==================
Deploy a file queue pattern that queues 1 to many dropped files then loads them sequentially.

Leverage the Magic
====================================
Data Magicians can drop as many files as they like and they get managed automagically.

ADI
==================
Bazinga!, this is a truly magical pattern that takes care of file loading and appending data safely.

Customer
==================
On point!, I love being able to just upload all my data files in one go and and they get loaded safely.

Magician Partner
===========================
Deploy a new file queueing pattern which wraps the existing create_landing_table function, to handle multiple dropped files going into the same target. Files are queued into a separate storage bucket then de-queued one at a time back into the filedrop bucket.

Last Refreshed
===========================
*Doc Refreshed: 2024-03-02*