fix/k8s-executor-logs → development: Add Retry Logic to Executor Job and Logs
#1141
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for ohsu-comp-bio-funnel canceled.
|
lbeckman314
changed the title
fix/k8s → development: Add Retry Logic to Executor Job and Logs
- The only way an error will arise is if we try to check the Executor pod's logs before or while it's being created. This commit resolves by simply waiting for the pod to be created.
lbeckman314
changed the title
fix/k8s → development: Add Retry Logic to Executor Job and Logsfix/k8s-executor-logs → development: Add Retry Logic to Executor Job and Logs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview ⚙️
When using the Kubernetes compute backend, the Funnel Worker fails when trying to read the logs of the Executor before the Executor Pod has fully started. This then results in a Error loop due to not having a retry mechanism in place.
Changes 🌀
This PR resolves this issue by adding retry mechanisms in two places:
Creating the Executor Job
Reading the Executor Logs
Credits 🤝
This PR is made possible by testing and development recommendations by the CTDS Team of Pauline Ribeyre, Jawad Qureshi, Sai Shanmukha Narumanchi, Aidan Hilt, and Ajo Augustine as part of the Gen3 Data Platform 🌱