Preemption pipeline failure can apply eviction side effects

[Aman-Cool]

Description

Summary

Preemption pipeline failures can cause victim task evictions to be applied even when preemption does not successfully complete, breaking the intended atomicity of scheduler Statement operations.

---

Impact

• Victim tasks may be evicted without the preemptor being scheduled
• Gang scheduling guarantees (e.g. MinAvailable) can be violated
• Failed preemption attempts can cause unintended workload disruption

---

Background

In the current preemption flow:

• Evictions are recorded on the scheduler Statement
• The preemptor is then pipelined
• If the pipeline fails, the preemptor state is rolled back, but recorded eviction operations may still be committed

This allows eviction side effects to escape failed preemption attempts.

---

Affected Code

• pkg/scheduler/actions/preempt/preempt.go

  • Preemption pipeline execution and failure handling

• pkg/scheduler/framework/statement.go

  • Operation recording and commit semantics

• pkg/scheduler/framework/statement_test.go

  • Boundary behavior around rollback / discard scenarios

---

Design Question

What is the correct architectural approach to ensure eviction side effects are only committed when preemption succeeds?

Option A: Full Statement Save / Discard (Preferred Pattern)

• Save the statement before eviction
• Discard or recover it on pipeline failure
• Only merge the statement when preemption completes successfully

Option B: Operation-Level Rollback

• Track and explicitly roll back eviction operations on failure

This fixes the issue locally but may introduce undo semantics on Statement that were not originally intended.

---

Goal

Ensure failed preemption attempts never apply eviction side effects, while staying consistent with existing Statement design patterns.

---

Context

Identified while implementing a fix for eviction rollback on preemption pipeline failure. The initial implementation raised concerns about exposing rollback behavior on Statement, prompting this issue to align on the intended design before proceeding.

---

Steps to reproduce the issue

1. Configure a workload where preemption is required to schedule a pending task
   (e.g. a gang-scheduled job that cannot be placed without evicting victim tasks).

2. Trigger the preemption workflow so that victim task evictions are recorded on the scheduler Statement.

3. Force the preemption pipeline to fail after eviction operations are recorded but before the preemptor is successfully scheduled
   (e.g. pipeline error, scheduling failure, or preemptor pod creation failure).

4. Allow the scheduler Statement to be committed.

5. Observe that victim tasks are evicted even though the preemptor was never scheduled.

Describe the results you received and expected

Expected Behavior

• If the preemption pipeline fails, no eviction side effects should be committed
• Victim tasks should remain running
• Scheduler state should remain unchanged after a failed preemption attempt

---

Actual Behavior

• Eviction operations recorded prior to pipeline failure are committed
• Victim tasks are evicted despite preemption not succeeding

What version of Volcano are you using?

main branch @ current HEAD (pre-merge)

Any other relevant information

This issue is independent of Kubernetes version, OS, or kernel configuration.

The behavior is triggered by the scheduler preemption control flow and statement commit semantics, and can be reproduced in unit tests without a running cluster.

No additional logs or manifests are attached, as the issue is reproducible via scheduler logic and unit tests.

[Aman-Cool]

@hajnalmt ,Opening this issue as discussed on the PR to track the preemption atomicity problem and align on the intended approach.
The issue describes the current behavior, impact, and the design question around statement discard vs. rollback.
Happy to iterate here or discuss further in the community meeting.
Thanks again for the guidance.

[hajnalmt]

/area scheduling

Thanks!

[hajnalmt]

We discussed this issue on friday!

It came out that we can expose more things on the statement handling, like Unevict but we should follow a Merge API logic which is there already in allocate. So instead of the OperationCount() solution you introduced in #5010 we can reuse that logic or expose a Merge API directly.

Is this a duplicated issue of #5026 now?

[Aman-Cool]

@hajnalmt ,Thanks for the clarification and for sharing the context.

I think of #5022 and #5026 as talking about the same underlying problem, just from slightly different angles.
#5022 is mainly about the concrete behavior we see today and the impact it has, where eviction side effects can still get applied even when preemption fails.
#5026 steps back and writes down the full preemption flow and the design discussion around how this should ideally work using merge based statement semantics.

My intention wasn’t to split the discussion or treat these as separate problems, but to keep both the symptom and the design reasoning visible. I am totally fine with them being treated as one logical issue and linked together, with most of the design and implementation discussion happening under #5026.

If later on it feels cleaner to consolidate or close one of them once the direction is fully settled, I am completely okay with that too.