[Flaky Test] job level LifecyclePolicy, Event: PodFailed, Action: RestartPod; Event PodEvicted, Action: TerminateJob, Timeout: 5m

[JesseStutler]

Description

The E2E Sequence Job has a flaky test: job level LifecyclePolicy, Event: PodFailed, Action: RestartPod; Event PodEvicted, Action: TerminateJob, Timeout: 5m, see the failed action's logs: https://github.com/volcano-sh/volcano/actions/runs/21574062351/job/62284826782?pr=5012

Steps to reproduce the issue

Describe the results you received and expected

Pass the failed e2e test

What version of Volcano are you using?

v1.14

Any other relevant information

No response

[aabhinavvvvvvv]

The test expects the job to transition: Pending → Running → Restarting → Running

But the job gets stuck in Restarting because of an infinite restart loop:

• Pod runs sleep 10s && xxx
• Pod fails after 10 seconds
• RestartPodAction recreates the pod
• New pod fails again after 10 seconds
• Repeat forever

The job never returns to Running because pods keep failing.
Should I submit a PR for this fix?

[JesseStutler]

@aabhinavvvvvvv Sure, if you would like to, but I prefer to first test in local environment multiple times to make sure that there will be no more flake test

[aabhinavvvvvvv]

I tested the fix locally and it works.
Before I fixed:
Line 154 test: Failed - "expected Running, got Restarting"
Same error as your CI log

After I fixed:
Line 154 test: Passed
Root cause: The test was waiting for the job to return to Running phase, but with RestartPodAction, the pod command (sleep 10s && xxx) keeps failing and restarting indefinitely, so the job stays in Restarting forever.

My local environment (WSL) had 2 additional test failures (TaskFailed at line 525, TensorFlow test) that passed in your CI log. I'm assuming these are environment specific issue unrelated to this fix.

[JesseStutler]

I tested the fix locally and it works. Before I fixed: Line 154 test: Failed - "expected Running, got Restarting" Same error as your CI log

After I fixed: Line 154 test: Passed Root cause: The test was waiting for the job to return to Running phase, but with RestartPodAction, the pod command (sleep 10s && xxx) keeps failing and restarting indefinitely, so the job stays in Restarting forever.

My local environment (WSL) had 2 additional test failures (TaskFailed at line 525, TensorFlow test) that passed in your CI log. I'm assuming these are environment specific issue unrelated to this fix.

Great, you can submit your PR about the fixing

[aabhinavvvvvvv]

I tested the fix locally and it works. Before I fixed: Line 154 test: Failed - "expected Running, got Restarting" Same error as your CI log
After I fixed: Line 154 test: Passed Root cause: The test was waiting for the job to return to Running phase, but with RestartPodAction, the pod command (sleep 10s && xxx) keeps failing and restarting indefinitely, so the job stays in Restarting forever.
My local environment (WSL) had 2 additional test failures (TaskFailed at line 525, TensorFlow test) that passed in your CI log. I'm assuming these are environment specific issue unrelated to this fix.

Great, you can submit your PR about the fixing

I've opened the PR #5021
Can you please review it