2023-08-10 - Error Message Parsing - Take 2¶
Problem¶
Jobs with less than 10 errors were showing up as failed, instead of completed with warnings. This meant users were being incorrectly notified the job had failed and the load pipelines were stopping.
Solution¶
Error parsing logic was updated to let the pipeline continue and return a data contract ‘warning’ notification to the user, and show a job status of ‘warning’ in the event history (although its technically a success)
Leverage the Magic¶
Update the existing check_job_statsus() pattern which checks as jobs finish, notifies the user and starts the next job in the pipeline
ADI¶
Smashed ! I love it when annoying bugs get squashed.
Magician Partner¶
We have a set a limit of 10 errors before a BQ job fails. But, when we had 1 or 2 errors thrown by a job, the job was being marked as failed and the pipeline was stopped and the user notified that the load had failed.
Error parsing logic was updated to let the pipeline continue and return a data contract ‘warning’ notification to the user, and show a job status of ‘warning’ in the event history (although its technically a success)
Last Refreshed¶
Doc Refreshed: 2024-03-02