Skip to content

fix: enforce expected_status in test execution pipeline#91

Merged
mikolajbadyl merged 2 commits intomainfrom
fix/expected-status-inference
Feb 28, 2026
Merged

fix: enforce expected_status in test execution pipeline#91
mikolajbadyl merged 2 commits intomainfrom
fix/expected-status-inference

Conversation

@mikolajbadyl
Copy link
Member

The agent was always comparing results against status 200, even when testing error cases (400, 404).

  • agent.go — local test plan generator now includes expected_status in the JSON format and knows which status codes to use per method/scenario
  • chat.go — added expected_status to ExecuteTestGroup tool schema (required field) with clear instructions
  • update_tests.goexpected_status is now included in test cases returned to the agent after GenerateTestPlan

@kilo-code-bot
Copy link

kilo-code-bot bot commented Feb 28, 2026

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (7 files)
  • internal/agents/agent.go - System prompt updates to require expected_status
  • internal/agents/chat.go - Tool schema updates making expected_status required
  • internal/agents/types.go - Example value update (200→404)
  • internal/cli/handlers.go - Validation and execution logic
  • internal/cli/update.go - Test case construction with expected_status
  • internal/cli/update_tests.go - Test generation with expected_status
  • internal/llm/claude/client.go - Schema handling refactor

Review Notes

This PR introduces a required expected_status field for test cases, which is a significant improvement for test accuracy. The changes ensure that:

  1. Agent prompts now instruct the LLM to include expected_status for each test
  2. Validation in ExecuteTestGroup rejects tests missing expected_status
  3. Test execution uses the provided status code instead of defaulting to 200
  4. Claude client was refactored to preserve full JSON schema structure

The flow is consistent: validation prevents tests without expected_status from proceeding, and the agent is informed to re-call with the correct field.

@mikolajbadyl mikolajbadyl merged commit d2cb78c into main Feb 28, 2026
1 check passed
@mikolajbadyl mikolajbadyl deleted the fix/expected-status-inference branch February 28, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant