[Bug] Sidecar mode shouldn't restart head pod when head pod is deleted #4141
Conversation
Signed-off-by: 400Ping <fourhundredping@gmail.com>
400Ping
requested review from
MortalHappiness,
andrewsykim,
kevin85421 and
rueian
as code owners
Signed-off-by: 400Ping <fourhundredping@gmail.com>
|
@400Ping, the change should be made in the raycluster_controller. We need to make the raycluster_controller not recreate the head pod if the cluster belongs to a RayJob, so that we can avoid races where the raycluster_controller recreates the head before the rayjob_controller checks it. |
ok, thanks |
Signed-off-by: 400Ping <fourhundredping@gmail.com>
Signed-off-by: 400Ping <fourhundredping@gmail.com>
| originatedFrom := utils.GetCRDType(instance.Labels[utils.RayOriginatedFromCRDLabelKey]) | ||
| if originatedFrom == utils.RayJobCRD { | ||
| logger.Info( | ||
| "reconcilePods: Found 0 head Pods for a RayJob-managed RayCluster; skipping head creation to let RayJob controller handle the failure", |
There was a problem hiding this comment.
Won't this cause no head pod to be created at all? We still need to create the first head pod. I think you can check the RayClusterProvisioned condition to decide whether to create one or not.
There was a problem hiding this comment.
I think this is related to the flaky test in CI now.
https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/11427#019a2849-e063-47f4-8aef-9143855d8976
3 participants
Why are these changes needed?
When using sidecar mode, the head pod should not be recreated after it is deleted. The RayJob should be marked as Failed.
Related issue number
Closes #4130
Checks