[Feature]: Leverage Additional SDK v0.2.25 Capabilities

[sjnims]

Feature Type

SDK integration (Anthropic or Agent SDK)

Problem or Need

The recent SDK alignment brought cc-plugin-eval up to date with the Claude Agent SDK v0.2.25 type definitions by fixing type mismatches and removing workarounds. However, several newly available SDK capabilities remain unused. These represent enhancement opportunities for improving execution control, safety, and observability.

Proposed Solution

Incrementally adopt the following SDK v0.2.25 features as needed:

• interrupt() method - Use as graceful timeout alternative to AbortController for scenario execution. Provides cleaner termination semantics.
• forkSession option - Enable session forking as alternative to /clear + continue for batch isolation. May improve checkpoint handling.
• sandbox settings - Configure execution safety with filesystem/network restrictions. Useful for testing untrusted plugins.
• Stop hook - Capture explicit completion signals from scenarios. Enables detection of clean vs forced termination.
• SessionStart/SessionEnd hooks - Track session lifecycle events for diagnostics. Helps with session batching metrics.
• New message types - Handle SDKToolProgressMessage, SDKToolUseSummaryMessage, SDKFilesPersistedEvent, SDKTaskNotificationMessage, SDKAuthStatusMessage in transcript capture and evaluation.

Pipeline Stage Affected

Stage 3 - Execution

Component Type (if applicable)

Not component-specific

Alternatives Considered

Current workarounds are functional:

• AbortController works for timeouts (just less semantic)
• /clear + continue handles batch isolation (slight overhead)
• Existing message types cover core evaluation needs

These enhancements are nice-to-have improvements, not critical gaps.

How important is this feature to you?

Low - Just a suggestion

Additional Context

These items were identified during the SDK v0.2.25 alignment work. The core alignment (type corrections, bug fixes) is complete. These represent future opportunities to leverage additional SDK capabilities as the framework matures.

Related commit: See recent Agent SDK alignment commits in the repository.

---

🤖 Created with Claude Code

[claude]

Analysis: Leverage Additional SDK v0.2.25 Capabilities

1. Problem Assessment

Is the problem clearly defined? Yes. This is a well-structured enhancement request that catalogs six SDK v0.2.25 capabilities not yet utilized by the framework. The issue correctly identifies these as nice-to-have improvements over functional workarounds.

Root cause: The recent SDK v0.2.25 alignment work focused on type correctness and bug fixes. These newly available capabilities were noted but intentionally deferred.

Pipeline impact: Primarily Stage 3 (Execution), with downstream effects on transcript capture and Stage 4 evaluation for new message types.

Missing information: The issue is well-scoped. One area that could use clarification is the priority ordering between the six items — some are clearly more impactful than others.

---

2. Proposed Solution(s)

Evaluating each capability individually:

a) interrupt() method — Graceful timeout alternative

• Current workaround: AbortController with setTimeout in src/stages/3-execution/agent-executor.ts:224-225 and session-batching.ts:727
• Benefit: Cleaner termination semantics — interrupt() signals the agent to stop gracefully rather than force-killing via abort. Could allow the agent to produce a partial result message.
• Tradeoff: AbortController is well-understood, widely tested, and already integrated into retry logic. The interrupt() method may not produce a result message in all cases, potentially breaking metrics extraction (extractResultMetrics at agent-executor.ts:390).
• Risk: Medium — need to verify interrupt() still triggers the for await loop to complete and produce an SDKResultMessage.
• Files affected: agent-executor.ts (prepareExecutionContext, executeScenario, executeScenarioWithCheckpoint), session-batching.ts (executeScenarioWithRetry), sdk-client.ts (QueryObject interface)

b) forkSession option — Alternative to /clear + continue

• Current workaround: persistSession: true + continue: true + explicit /clear command between scenarios in session-batching.ts:563-608
• Benefit: Could eliminate the /clear query round-trip entirely, reducing per-scenario overhead in batches. Fork semantics may provide better isolation than /clear.
• Tradeoff: The current /clear approach is battle-tested and explicitly documented. forkSession semantics need verification: does it fork the subprocess or the conversation state? How does it interact with enableFileCheckpointing and rewindFiles()?
• Risk: High — session batching is a core optimization (80% overhead reduction). Any regression here impacts all batch execution. The interaction between forkSession, persistSession, continue, and file checkpointing needs careful testing.
• Files affected: session-batching.ts (executeBatch, sendClearCommand, buildScenarioQueryInput), sdk-client.ts (QueryOptions)

c) sandbox settings — Filesystem/network restrictions

• Current workaround: None — relies on permissionMode: "bypassPermissions" which is the opposite of sandboxing.
• Benefit: Critical safety improvement for evaluating untrusted plugins. Could restrict filesystem writes and network access during execution.
• Tradeoff: May conflict with permission_bypass mode. Plugins that legitimately need filesystem/network access (MCP servers, hooks) would need exceptions. Would require new configuration schema in src/config/.
• Risk: Low for implementation, medium for correctness — overly restrictive sandboxing could cause false negatives in evaluation.
• Files affected: sdk-client.ts (QueryOptions), agent-executor.ts (buildQueryInput), session-batching.ts (buildScenarioQueryInput), src/config/schema.ts (new sandbox config fields)

d) Stop hook — Clean vs forced termination detection

• Current workaround: Timeout detection via AbortError check at agent-executor.ts:504. No explicit clean completion signal.
• Benefit: Enables reliable distinction between: (1) clean completion, (2) timeout/abort, (3) error termination. This is valuable metadata for evaluation quality.
• Tradeoff: Minimal — additive hook that follows existing hook patterns.
• Risk: Low — follows the same HookCallbackMatcher pattern already used for five other hooks.
• Files affected: hooks-factory.ts (SDKHooksConfig, createCaptureHooksConfig, assembleHooksConfig), sdk-client.ts (QueryOptions hooks type), agent-executor.ts (execution context), src/types/transcript.ts (new capture type or field on ExecutionResult)

e) SessionStart/SessionEnd hooks — Session lifecycle tracking

• Current workaround: Manual timing with Date.now() at agent-executor.ts:226 and session-batching.ts:703.
• Benefit: SDK-provided session lifecycle events would give accurate timing for session startup (subprocess spawn + plugin load + MCP initialization) vs actual execution. Useful for batching metrics and debugging.
• Tradeoff: Adds two more hook types to the already-complex hook factory. Value is primarily diagnostic.
• Risk: Low — purely observational hooks.
• Files affected: hooks-factory.ts (new hook types), sdk-client.ts (QueryOptions hooks, new input types), src/types/transcript.ts (new capture types)

f) New message types — Extended transcript capture

• Current workaround: Unknown message types are silently ignored during iteration in agent-executor.ts:485-497.
• Benefit: SDKToolProgressMessage and SDKToolUseSummaryMessage could enrich tool capture data. SDKFilesPersistedEvent could improve file checkpointing reliability. SDKTaskNotificationMessage could aid subagent tracking. SDKAuthStatusMessage could detect auth issues.
• Tradeoff: Each new message type needs: (1) type guard in sdk-client.ts, (2) handling in the message iteration loop, (3) storage in transcript or execution result, (4) consideration in Stage 4 evaluation.
• Risk: Low individually, but cumulative complexity across 5+ new types.
• Files affected: sdk-client.ts (new type guards), agent-executor.ts (message processing), session-batching.ts (message processing), transcript-builder.ts (new event types), src/types/transcript.ts (new types)

---

3. Alternative Approaches

Approach A: Incremental adoption (recommended by issue)

• Implement each capability independently as separate PRs
• Pros: Low risk, easy to review, can pause at any point
• Cons: Slower overall, may miss cross-cutting optimizations

Approach B: Grouped by theme

• Group 1 — Execution control: interrupt() + Stop hook + sandbox
• Group 2 — Session management: forkSession + SessionStart/SessionEnd
• Group 3 — Observability: New message types
• Pros: Related changes reviewed together, better integration testing
• Cons: Larger PRs, higher review burden

Approach C: Feature-flag gated adoption

• Add all capabilities behind configuration flags (e.g., execution.use_interrupt_timeout: true)
• Pros: Safe rollout, easy A/B comparison with existing behavior
• Cons: Configuration complexity, maintenance of two code paths

Recommendation: Approach A with priority ordering (see Section 5). The capabilities are largely independent and the codebase already handles the current approaches well.

---

4. Implementation Considerations

Files/areas affected (by frequency of change):

1. src/stages/3-execution/sdk-client.ts — All six features touch type definitions here
2. src/stages/3-execution/hooks-factory.ts — Items d, e add new hook types
3. src/stages/3-execution/agent-executor.ts — Items a, c, d, f modify execution logic
4. src/stages/3-execution/session-batching.ts — Items a, b, c, d, e, f
5. src/types/transcript.ts — Items d, e, f add new capture/event types
6. src/config/schema.ts — Item c adds sandbox configuration

Breaking changes: None expected if implemented additively. All current behavior should remain as defaults.

Migration concerns: None — these are new capabilities, not replacements. The forkSession feature (item b) could eventually deprecate the /clear + continue pattern, but should be opt-in initially.

Testing requirements:

• Unit tests: Each new hook type, type guard, and configuration option needs unit tests. The existing test patterns in the __tests__ directories provide good templates.
• Integration tests: forkSession (item b) and interrupt() (item a) need integration tests with the real SDK to verify behavior matches expectations.
• State/resume impact: Items a, b, and d could affect how execution results are recorded, which feeds into state persistence. The ExecutionResult type changes need corresponding updates to migrateState() in src/state/operations.ts if new fields are added.

---

5. Recommendation

Priority ordering (by value/risk ratio):

1. Stop hook (item d) — Highest value, lowest risk. Follows existing hook patterns exactly. Enables clean/forced termination distinction which directly improves evaluation quality. Should be a straightforward addition to SDKHooksConfig and hooks-factory.ts.

2. interrupt() method (item a) — High value for timeout handling semantics. Requires verification that interrupted queries still produce SDKResultMessage for metrics extraction. Implement as an alternative to AbortController with a configuration toggle initially.

3. New message types (item f) — Medium value, low risk per type. Implement SDKToolProgressMessage and SDKToolUseSummaryMessage first as they directly enrich existing ToolCapture data. Others can follow incrementally.

4. SessionStart/SessionEnd hooks (item e) — Medium value for diagnostics. Useful for session batching metrics but not critical for evaluation correctness.

5. sandbox settings (item c) — High potential value for safety, but requires design work around configuration schema and interaction with permission_bypass. Worth a separate design issue.

6. forkSession option (item b) — Highest potential impact but also highest risk. The current /clear + continue pattern works and is well-tested. forkSession should be thoroughly evaluated in an experimental branch before any production adoption.

Suggested follow-up issues:

• Separate issue for sandbox configuration design (item c) — needs input on default restrictions and exception patterns
• Separate issue for forkSession experimentation (item b) — needs benchmarking vs current /clear approach
• The remaining items (a, d, e, f) can be implemented directly from this issue as individual PRs

---

This analysis was generated by reviewing the Stage 3 execution code, SDK integration layer, hook factory system, session batching implementation, and transcript type definitions.