Improve aggregate reingestion reporting #4121
Labels
💻 aspect: code
Concerns the software code in the repository
🧰 goal: internal improvement
Improvement that benefits maintainers, not users
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
Current Situation
#4074 modified reingestion workflows to skip Slack error reporting during the
pull_data
tasks, and instead report errors in aggregate in a task at the end of the Dagrun. Because a reingestion workflow can have >100 ingestion days, it's possible this aggregate message could contain links to 100s of logfiles in the worst case. Consequently the message is configured to only link the first 5 error logs.There is also no context as to what the errors were in the Slack message itself, just the link to the log file for the failed task.
Suggested Improvement
It would be great to include information about the errors that were detected in the Slack message. In particular, when there is more than one failed reingestion day it would be very useful to know whether all the failed days encountered the same error, in which case it may not be necessary to investigate them individually.
We should weigh the benefits of presenting this information against the desire to keep the Slack notifications as brief as possible.
Additional context
Comment thread on the aggregate reporting PR which discussed this.
The text was updated successfully, but these errors were encountered: