[Feature]: Add session timing breakdown to dry-run cost estimation

[sjnims]

Parent Issue

Sub-issue of #344 (Leverage Additional SDK v0.2.25 Capabilities)
Follow-up to #349 (SessionStart/SessionEnd hooks)

Feature Type

Cost estimation

Problem or Need

The --dry-run mode estimates costs and scenario counts but doesn't account for session lifecycle overhead. With #349 capturing session timing data, dry-run output could include timing projections that help users estimate total wall-clock time, not just API costs.

Currently, dry-run shows:

• Estimated scenario count
• Estimated cost per scenario
• Total estimated cost

Missing:

• Estimated session startup overhead per batch
• Projected wall-clock time factoring in session lifecycle
• Number of session starts/restarts expected per batch strategy

Proposed Solution

1. Add timing estimates to dry-run output — Based on historical session timing data (if available from previous runs) or conservative defaults, include:

   • Estimated session startup time per batch
   • Estimated total wall-clock time
   • Number of expected session starts (1 per batch in batched_by_component, 1 per scenario in isolated)

2. Use historical data — If previous run results exist in the state file, extract average session_timing durations to improve estimates

Pipeline Stage Affected

General / Multiple stages

Component Type

Not component-specific

Alternatives Considered

• Only show timing data in post-execution reports (via [Feature]: Expose session timing metrics in Stage 4 evaluation reports #357). This is simpler but doesn't help with pre-execution planning.
• Show a simple "sessions needed" count without timing estimates. Lower effort but less useful.

How important is this feature to you?

Low - Just a suggestion

Additional Context

This pairs with #357 (session timing in Stage 4 reports). Together they provide pre-execution estimates and post-execution actuals for session lifecycle overhead.

Risk Assessment

Low risk — Additive output in dry-run mode. No changes to execution behavior.

Files Affected

• src/stages/3-execution/dry-run.ts (or equivalent) — Cost estimation output
• CLI formatters for dry-run display

---

🤖 Created with Claude Code

[claude]

Analysis: Add session timing breakdown to dry-run cost estimation

1. Problem Assessment

Clearly defined? Yes — the issue is well-scoped. The current dry-run output (formatPipelineCostEstimate in src/stages/2-generation/cost-estimator.ts:428-468) shows only API cost projections (token counts and USD estimates per stage). It provides no information about wall-clock time or session lifecycle overhead, which can be significant — the codebase itself documents ~5 seconds of startup overhead per session (src/stages/3-execution/session-batching.ts:937-938).

Missing information:

• No historical session timing data exists yet. Issue [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 (SessionStart/SessionEnd hooks) is still open, meaning there is no session_timing data being captured in state or execution results. The proposed "use historical data" approach has no data source to draw from today.
• The issue references src/stages/3-execution/dry-run.ts as an affected file, but this file does not exist. Dry-run logic lives in src/cli/commands/run.ts:136-140 (early exit) and src/stages/2-generation/index.ts:172-195 (skip generation, return cost estimate).

Root cause: The dry-run path returns after Stage 2 generation, outputting only PipelineCostEstimate (src/types/cost.ts:34-39) which contains token/cost data but no time or session information. The cost estimator (estimatePipelineCost at src/stages/2-generation/cost-estimator.ts:353-419) has access to AnalysisOutput and EvalConfig — enough to compute session counts — but currently doesn't.

Pipeline stages affected: Primarily Stage 2 (cost estimation output) and the CLI formatting layer. No changes to Stage 3 execution behavior.

2. Proposed Solution(s)

Issue's proposed solution — evaluation:

The two-part proposal (defaults + historical data) is reasonable, but the historical data part depends on #349 and #357 which are both open. The default-based approach is independently valuable.

Tradeoffs:

• Conservative defaults (5s/session startup) are already documented in session-batching.ts:937-938. These are reasonable estimates but could be inaccurate for different environments.
• Historical data from previous runs is a good idea but creates a dependency chain: [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 → [Feature]: Expose session timing metrics in Stage 4 evaluation reports #357 → this feature. Without those, there's no session_timing field in state or results.
• Session count calculation can be derived from the same data available during dry-run: component counts from AnalysisOutput + session strategy from ExecutionConfig. The logic already exists in estimateScenarioCount (cost-estimator.ts:329-344) and groupScenariosByComponent (session-batching.ts:136-156).

Specific files needing changes:

File	Change
src/types/cost.ts	Extend PipelineCostEstimate with session timing fields
src/stages/2-generation/cost-estimator.ts	Add session count estimation and timing projection to estimatePipelineCost and formatPipelineCostEstimate
src/stages/3-execution/session-batching.ts	Possibly export the overhead constant (currently hardcoded 5)

Edge cases:

• isolated strategy: sessions = scenario count (could be very high)
• batched_by_component strategy: sessions = unique component count (from calculateComponentCounts)
• MCP server scenarios and additional plugins affect grouping keys (session-batching.ts:141-145)
• The scenario count in dry-run is an estimate (e.g., counts.hooks * 5), so the session count derived from it is also an estimate

3. Alternative Approaches

A. Minimal: Session count only (no timing)

• Just add "Expected sessions: N" to dry-run output based on strategy
• Pro: Simple, no assumptions about timing, no dependency on [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349
• Con: Less useful for planning

B. Default-based timing (proposed, phase 1)

• Use the 5s/session constant already in the codebase
• Show estimated startup overhead and simple wall-clock projection
• Pro: Immediately useful, no dependencies
• Con: Estimates may not match reality; could mislead users

C. Historical data enrichment (proposed, phase 2)

• Pull timing from previous runs' state files when available
• Pro: More accurate over time
• Con: Depends on [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349/[Feature]: Expose session timing metrics in Stage 4 evaluation reports #357; adds complexity; first run always falls back to defaults anyway

D. Per-scenario execution time estimate

• Estimate per-scenario execution time based on max_turns and model speed
• Pro: More complete wall-clock projection
• Con: Highly variable, speculative; would need model-specific benchmarks

Comparison: Approach A is lowest effort and risk. Approach B adds moderate value with minimal risk. Approach C is premature until #349/#357 land. Approach D is over-engineered for a "low importance" feature.

4. Implementation Considerations

Files affected:

1. src/types/cost.ts — Add optional session_estimate field to PipelineCostEstimate
2. src/stages/2-generation/cost-estimator.ts — Compute session counts, format timing section
3. Possibly src/stages/3-execution/session-batching.ts — Extract SESSION_STARTUP_OVERHEAD_SEC as exported constant

Breaking changes: None. The PipelineCostEstimate type extension should use optional fields. The formatter output is additive.

Migration concerns: None. No state schema changes needed for phase 1 (default-based).

Testing requirements:

• Unit tests for session count estimation logic (given component counts + strategy → expected session count)
• Unit tests for formatter output including timing section
• Snapshot or integration test for dry-run CLI output format

State/resume impact: None for phase 1. Phase 2 (historical data) would read from existing state files but not modify them.

5. Recommendation

Recommended approach: Phase 1 only (Approach B — defaults-based session timing)

Rationale:

• The feature is marked "low importance" by the author
• The key value — knowing how many sessions will be created — is achievable with zero dependencies
• Historical data (phase 2) should wait until [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 and [Feature]: Expose session timing metrics in Stage 4 evaluation reports #357 are merged and session timing data actually exists
• The 5s/session overhead constant is already validated in the codebase

Implementation steps:

1. Export SESSION_STARTUP_OVERHEAD_SEC = 5 constant from session-batching.ts
2. Add SessionEstimate type to src/types/cost.ts with fields: session_count, strategy, estimated_overhead_sec
3. Add estimateSessionOverhead(analysis, config) function in cost-estimator.ts that computes session count from component counts and strategy
4. Extend PipelineCostEstimate with optional session_estimate field
5. Update formatPipelineCostEstimate to append a "Session Overhead" section when session_estimate is present
6. Add unit tests for the new estimation and formatting logic

Follow-up issues:

• Phase 2 (historical data enrichment) should be a separate issue created after [Feature]: Add SessionStart/SessionEnd hooks for lifecycle tracking #349 and [Feature]: Expose session timing metrics in Stage 4 evaluation reports #357 are merged
• Consider adding per-scenario execution time estimates as a future enhancement if users request it