[Feature]: Expose session timing metrics in Stage 4 evaluation reports

[sjnims]

Parent Issue

Sub-issue of #344 (Leverage Additional SDK v0.2.25 Capabilities)
Follow-up to #349 (SessionStart/SessionEnd hooks)

Feature Type

Output / reporting formats

Problem or Need

#349 adds session_timing data to ExecutionResult via SessionStart/SessionEnd hooks, capturing session lifecycle events (startup source, end reason, timestamps). However, Stage 4 evaluation doesn't currently use this data — it's captured but not surfaced in reports or available for evaluation logic.

Exposing session timing would enable:

• Reporting session startup latency (time from SessionStart to first tool use)
• Identifying slow plugin/MCP initialization in batch sessions
• Distinguishing infrastructure overhead from actual LLM execution time
• Tracking /clear-triggered session restarts within batches

Proposed Solution

1. Add session timing to evaluation report output — Include session_timing summary in the per-scenario evaluation results (e.g., startup duration, total session duration, number of restarts)

2. Surface in CLI output — Show timing breakdown in the results table or summary

3. Optionally use in evaluation logic — Stage 4 could flag scenarios with unusually long startup times or unexpected session restarts as diagnostic notes

Pipeline Stage Affected

Stage 4 - Evaluation

Component Type

Not component-specific

Alternatives Considered

• Leave session_timing as raw data only, accessible via JSON output but not surfaced in reports. Simpler but less useful for diagnostics.

How important is this feature to you?

Low - Just a suggestion

Additional Context

The SessionTimingCapture type from #349 provides:

• starts: SessionStartCapture[] — source ('startup'|'resume'|'clear'|'compact'), timestamp, agent_type?, model?
• end?: SessionEndCapture — reason (ExitReason), timestamp

Risk Assessment

Low risk — Purely additive reporting. No changes to execution or evaluation behavior.

Files Affected

• src/stages/4-evaluation/ — Report generation
• CLI output formatters

---

🤖 Created with Claude Code

[claude]

Analysis: Expose session timing metrics in Stage 4 evaluation reports

1. Problem Assessment

Clearly defined? Yes — the issue describes a well-scoped, additive reporting feature. The data flow is clear: session timing data captured in Stage 3 → surfaced in Stage 4 reports and CLI output.

Key dependency: This issue depends on #349 (SessionStart/SessionEnd hooks), which is still open. The SessionTimingCapture, SessionStartCapture, and SessionEndCapture types referenced in this issue do not yet exist in the codebase. Neither does a session_timing field on ExecutionResult (src/types/transcript.ts:239-264). This issue cannot be implemented until #349 lands.

What exists today:

• ExecutionResult.api_duration_ms (src/types/transcript.ts:244) — per-scenario SDK API duration
• EvalMetrics.total_api_duration_ms (src/types/evaluation.ts:226) — aggregate sum of all scenario API durations
• ExecutionOutput.total_duration_ms (src/stages/3-execution/index.ts:72) — wall-clock time for all of Stage 3
• EvaluationOutput.total_duration_ms (src/stages/4-evaluation/index.ts:172) — wall-clock time for Stage 4
• TerminationType (src/types/transcript.ts:234) — scenario termination reason (clean/timeout/interrupted/error)

What's missing (to be provided by #349):

• SessionTimingCapture type with starts[] and end? fields
• session_timing field on ExecutionResult
• Per-session startup source tracking (startup/resume/clear/compact)
• Session end reason tracking

Pipeline stages affected: Primarily Stage 4 (evaluation/reporting). Stage 3 is affected only via #349.

2. Proposed Solution(s)

The issue proposes three incremental pieces. Analysis of each:

2a. Add session timing to evaluation report output

Approach: Derive computed timing metrics from the raw session_timing data and include them in EvaluationResult or as a new optional stats block in EvalMetrics.

Computed metrics to derive:

• Startup latency: Time from first SessionStart to first tool use (requires correlating with detected_tools[0].timestamp or transcript)
• Total session duration: Time from first SessionStart.timestamp to SessionEnd.timestamp
• Number of restarts: Count of starts[] entries with source clear or compact
• Infrastructure overhead: total_session_duration - api_duration_ms

Tradeoffs:

• Adding fields to EvaluationResult (src/types/evaluation.ts:133-147) increases its size; optional fields are better
• Startup latency calculation depends on reliable timestamp correlation between session events and tool captures — edge cases where no tools are captured need handling
• Restart count is straightforward; infrastructure overhead is approximate

Files affected:

• src/types/evaluation.ts — Add optional SessionTimingStats interface and field to EvalMetrics
• src/stages/4-evaluation/metrics.ts:398-436 — Extend calculateCostMetrics() or add a new calculateSessionTimingStats() function
• src/stages/4-evaluation/index.ts — Thread session timing through to output

2b. Surface in CLI output

Approach: Add timing breakdown to outputFinalSummary() and/or outputCLISummary().

Files affected:

• src/cli/formatters.ts:147-172 — outputFinalSummary() to add timing lines
• src/cli/formatters.ts:52-84 — outputCLISummary() could show per-scenario timing

Tradeoffs:

• CLI output should remain concise; a separate --verbose or --timing flag could gate detailed timing info
• JSON/YAML output formats (src/cli/commands/report.ts) would get it for free if it's in EvalMetrics

2c. Optionally use in evaluation logic (diagnostic flags)

Approach: Stage 4 could flag scenarios with unusually long startup times or unexpected restarts as diagnostic notes in issues[].

Tradeoffs:

• Adds complexity to evaluation logic for marginal value
• Threshold for "unusually long" is subjective and plugin-dependent
• Risk of false positives in diagnostic notes
• Recommend deferring this to a follow-up issue

3. Alternative Approaches

A. Minimal: Pass-through only (Recommended for initial implementation)

Include raw session_timing in the JSON output alongside EvaluationResult, without deriving computed metrics. Users can analyze timing themselves.

Pros: Simplest to implement, no risk of incorrect derived metrics, immediately useful for power users examining JSON output.
Cons: No CLI visibility, no computed convenience metrics.

B. Moderate: Computed stats block in EvalMetrics

Add a SessionTimingStats optional block to EvalMetrics (parallel to existing CacheStats, MultiSampleStats, etc.), with aggregate timing breakdowns.

Pros: Follows established pattern (src/types/evaluation.ts:239-250), clean integration, available in all output formats.
Cons: More implementation work, needs careful handling of edge cases (missing timestamps, no session events).

C. Full: Per-scenario + aggregate + CLI + diagnostics

Everything proposed in the issue, including diagnostic flagging.

Pros: Maximum visibility and utility.
Cons: Significant scope for a low-priority feature; diagnostic thresholds are hard to get right.

4. Implementation Considerations

Files to modify:

File	Change
src/types/evaluation.ts	Add SessionTimingStats interface, optional field on EvalMetrics
src/stages/4-evaluation/metrics.ts	Add calculateSessionTimingStats(), integrate into calculateEvalMetrics()
src/stages/4-evaluation/index.ts	Thread timing data through evaluation pipeline
src/cli/formatters.ts	Display timing in outputFinalSummary()
src/cli/commands/report.ts	Timing included automatically via metrics

Breaking changes: None — all additions are optional fields following the existing pattern of CacheStats, RepetitionStats, etc.

State/resume impact: None — session timing is read-only data from Stage 3 results. The evaluation stage re-derives metrics from execution results on resume.

Migration: No state migration needed (src/state/operations.ts) since this adds optional output fields, not new component types.

Testing requirements:

• Unit tests for calculateSessionTimingStats() with edge cases (no session events, multiple restarts, missing end event)
• Unit tests for formatter output with and without timing data
• Integration test verifying timing flows through the full Stage 4 pipeline
• Tests for scenarios with session_timing: undefined (backward compatibility with pre-[Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 data)

5. Recommendation

Recommended approach: B (Computed stats block in EvalMetrics)

This follows the established pattern for optional metrics (see CacheStats at src/types/evaluation.ts:196-205 and addOptionalStats() at src/stages/4-evaluation/metrics.ts:438-460) and provides useful computed values without over-engineering.

Implementation order:

1. Wait for [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 to land — this is a hard dependency
2. Define SessionTimingStats interface in src/types/evaluation.ts
3. Implement calculateSessionTimingStats() in src/stages/4-evaluation/metrics.ts
4. Wire into addOptionalStats() and calculateEvalMetrics()
5. Add timing display to outputFinalSummary() in src/cli/formatters.ts
6. Add unit tests for the new stats calculation
7. Integration test with mock session timing data

Defer to follow-up issues:

• Diagnostic flagging of slow sessions (2c) — needs design around thresholds and should be its own issue
• Per-scenario timing in EvaluationResult — evaluate need after aggregate stats are available

Estimated scope: Small — approximately 4-5 files changed, following well-established patterns in the codebase.