Table Of Contents
Table Of Contents

2023-04-11 - Multi File Drop File Queuing

Release

Status: Available

Type: DataOps

Date: 2023-04-11

Problem

Sometimes Data Magicians drop lots of files at the same time causing upstream issues with appending and loading data.

Solution

Deploy a file queue pattern that queues 1 to many dropped files then loads them sequentially.

Leverage the Magic

Data Magicians can drop as many files as they like and they get managed automagically.

ADI

Bazinga!, this is a truly magical pattern that takes care of file loading and appending data safely.

Customer

On point!, I love being able to just upload all my data files in one go and and they get loaded safely.

Magician Partner

Deploy a new file queueing pattern which wraps the existing create_landing_table function, to handle multiple dropped files going into the same target. Files are queued into a separate storage bucket then de-queued one at a time back into the filedrop bucket.

Last Refreshed

Doc Refreshed: 2024-05-20