Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 17, 2026

Summary

  • What: Strengthen test_orchestrate_persist to validate the structure and contents of the generated field
  • Why: Original test only checked key presence, not the actual data structure created by process_module()

Changes

  • Add assertions validating generated is a dictionary with expected schema:
    • humanized_text: non-empty string
    • ttft_s: non-negative number
  • Previously only verified "generated" in data[0], which could pass with incorrect data

Testing

  • All 9 tests in test_orchestrate_content.py pass
  • Code review found no issues

Checklist

  • Test strengthened with structural validation
  • All tests pass
  • Code review complete

Reviewers


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

…ture

Co-authored-by: 73junito <86015877+73junito@users.noreply.github.com>
Copilot AI changed the title [WIP] Address feedback from PR 158 review comments Strengthen test validation for generated field structure in orchestrate_content tests Jan 17, 2026
Copilot AI requested a review from 73junito January 17, 2026 18:51
Copy link
Owner

@73junito 73junito left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed

@73junito 73junito marked this pull request as ready for review January 17, 2026 19:04
Copilot AI review requested due to automatic review settings January 17, 2026 19:04
@73junito 73junito merged commit 2cce672 into merge/pr-158-local Jan 17, 2026
4 of 7 checks passed
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens test validation in test_orchestrate_persist by adding assertions that validate the structure and contents of the generated field created by process_module(). Previously, the test only verified that the generated key existed without checking its actual structure or data.

Changes:

  • Add validation for generated field structure as a dictionary
  • Add validation for humanized_text field (type, presence, and non-empty content)
  • Add validation for ttft_s field (presence, numeric type, and non-negative value)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +50 to +53
# Verify TTFT is present and valid when returned from call_ollama
assert "ttft_s" in generated, "generated should contain ttft_s"
assert isinstance(generated["ttft_s"], (int, float)), "ttft_s should be a number"
assert generated["ttft_s"] >= 0, "ttft_s should be non-negative"
Copy link

Copilot AI Jan 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assertions for ttft_s assume it will always be present in the generated field, but this doesn't match the actual implementation behavior.

Looking at process_module() in scripts/orchestrate_content.py (lines 129-130), ttft_s is only set conditionally: if "ttft" in res: module["generated"]["ttft_s"] = float(res["ttft"]).

The real call_ollama() only returns ttft when stream=True (line 77), but NOT in non-streaming mode (lines 84-88 only return text). This test uses stream=False (line 39), so with the real implementation, ttft would not be returned and ttft_s would not be set.

While the test's fake_call returns ttft even in non-streaming mode, this doesn't reflect realistic behavior. Consider either:

  1. Changing the test to use stream=True to match a realistic scenario, or
  2. Making the ttft_s validation conditional by checking if it exists before asserting its type and value
Suggested change
# Verify TTFT is present and valid when returned from call_ollama
assert "ttft_s" in generated, "generated should contain ttft_s"
assert isinstance(generated["ttft_s"], (int, float)), "ttft_s should be a number"
assert generated["ttft_s"] >= 0, "ttft_s should be non-negative"
# Verify TTFT is valid when returned from call_ollama (optional field)
if "ttft_s" in generated:
assert isinstance(generated["ttft_s"], (int, float)), "ttft_s should be a number"
assert generated["ttft_s"] >= 0, "ttft_s should be non-negative"

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants