2026-09-09 - Large File Transfers Now Reliable for Files Over the 30-Second Limit¶
Release¶
AgileData.io - No More Silent Failures: Large File Copies Now Complete Reliably
When AgileData transfers large files between storage locations, it previously used a copy API that has a hard 30-second timeout. Files that took longer than this would fail silently — the copy would abort, but the system would continue as if the file had moved successfully, leading to missing data and jobs that appeared to succeed but hadn’t.
Three fixes were applied together to resolve this: the copy mechanism was switched to Google Cloud Storage’s blob rewrite API (which has no time limit on large transfers), file transfer functions now confirm the file has actually moved before proceeding, and large file dequeuing was moved outside the Spanner transaction to prevent aborted inserts when a copy exceeds the transaction timeout.
What was happening:
Large file copies silently failing after 30 seconds
Downstream jobs proceeding as if the file existed when it did not
Errors not surfaced in logs, making diagnosis difficult
What this means for you:
Large file transfers complete reliably regardless of file size
Jobs that depend on file transfers will not proceed until the file is confirmed present
Copy failures are now logged with enough detail to diagnose problems
Last Refreshed¶
Doc Refreshed: 2026-09-09