-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add pytest AI validation and Robot Framework BDD testing samples #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Michspirit99
wants to merge
5
commits into
main
Choose a base branch
from
feat/testing-samples
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
0f8dee4
docs: significantly improve repository description
Michspirit99 3e48a82
feat: add pytest AI validation and Robot Framework BDD testing samples
Michspirit99 81f4c73
fix: resolve linter errors (unused imports, f-strings)
Michspirit99 5c7f8cb
Update samples/copilot_bdd.robot
Michspirit99 575d1ab
Update samples/pytest_ai_validation.py
Michspirit99 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,21 +1,33 @@ | ||||||
| # Copilot SDK Python Scripts 🐍 | ||||||
| # Copilot SDK for Python — Complete Sample Collection 🚀 | ||||||
|
|
||||||
| > **Zero-ceremony AI scripts in Python** — Single-file Python scripts using the [GitHub Copilot SDK](https://github.com/github/copilot-sdk). Just `pip install` and run. No setup.py, no boilerplate—pure Python simplicity meets AI-powered automation. | ||||||
| > **Production-ready Python samples for the GitHub Copilot SDK** — 17 fully-functional examples demonstrating AI agents, custom tools, browser automation, code review, BDD testing, and more. All tested in CI with `gpt-5-mini` (free tier). Clone, run, and build. | ||||||
|
|
||||||
| [](LICENSE) | ||||||
| [](https://www.python.org/downloads/) | ||||||
| [](https://github.com/github/copilot-sdk) | ||||||
| [](https://github.com/Michspirit99/copilot-sdk-python/actions/workflows/ci.yml) | ||||||
| [](https://github.com/Michspirit99/copilot-sdk-python/actions/workflows/agent-scenarios.yml) | ||||||
|
|
||||||
| ## What Is This? | ||||||
| ## Why This Repository? | ||||||
|
|
||||||
| This is **the most comprehensive collection of Python samples** for the [GitHub Copilot SDK](https://github.com/github/copilot-sdk). Unlike minimal "hello world" examples, these are **production-ready patterns** you can actually use: | ||||||
|
|
||||||
| This repository demonstrates the [GitHub Copilot SDK for Python](https://github.com/github/copilot-sdk/tree/main/python) through practical,single-file scripts. Each script is: | ||||||
| - ✅ **17 complete samples** — streaming, tools, BDD testing, browser automation, code review, and more | ||||||
| - ✅ **Proven in CI** — All samples run end-to-end with `gpt-5-mini` (GitHub's free tier model) | ||||||
| - ✅ **Single-file simplicity** — Each sample is self-contained and ready to run | ||||||
| - ✅ **Real-world patterns** — API testing, log analysis, test data generation, git commit messages | ||||||
| - ✅ **Best practices** — Type hints, async/await, proper error handling, structured outputs | ||||||
|
|
||||||
| - **Self-contained** — One `.py` file, ready to run | ||||||
| - **Practical** — Real-world automation use cases | ||||||
| - **Modern Python** — Type hints, async/await, argparse | ||||||
| - **Zero boilerplate** — No setup.py, no project scaffolding | ||||||
| **Perfect whether you're exploring the SDK for the first time or building production AI agents.** | ||||||
|
|
||||||
| ## What Is This? | ||||||
|
|
||||||
| The GitHub Copilot SDK gives you programmatic access to the same AI agent runtime powering Copilot CLI and Copilot Chat. | ||||||
| The [GitHub Copilot SDK](https://github.com/github/copilot-sdk/tree/main/python) gives you programmatic access to the same AI agent runtime powering Copilot CLI and VS Code. This repository shows you how to use it effectively through practical, battle-tested examples. | ||||||
|
|
||||||
| Each script demonstrates a key SDK capability: | ||||||
| - AI conversation patterns (streaming, multi-turn, interactive) | ||||||
| - Custom tool definitions (function calling) | ||||||
| - Real-world automation (browser control, code review, testing) | ||||||
| - Production patterns (error handling, retries, structured output) | ||||||
|
|
||||||
| ## Prerequisites | ||||||
|
|
||||||
|
|
@@ -28,43 +40,98 @@ The GitHub Copilot SDK gives you programmatic access to the same AI agent runtim | |||||
|
|
||||||
| ```bash | ||||||
| # Clone this repo | ||||||
| git clone https://github.com/Michspirit99/copilot-sdk-python-scripts.git | ||||||
| cd copilot-sdk-python-scripts | ||||||
| git clone https://github.com/Michspirit99/copilot-sdk-python.git | ||||||
| cd copilot-sdk-python | ||||||
|
|
||||||
| # Install dependencies | ||||||
| # Install dependencies (creates venv recommended) | ||||||
| python -m venv .venv | ||||||
| source .venv/bin/activate # Windows: .venv\Scripts\activate | ||||||
| pip install -r requirements.txt | ||||||
|
|
||||||
| # Run any script — instant AI! | ||||||
| # Run any sample — instant AI! | ||||||
| python samples/hello_copilot.py | ||||||
| python samples/streaming_chat.py "Explain Python decorators" | ||||||
| python samples/code_reviewer.py samples/hello_copilot.py | ||||||
| ``` | ||||||
|
|
||||||
| That's it. No virtual environment required (but recommended). No project setup. Just Python and AI. | ||||||
| **That's it.** No API keys needed if you have Copilot CLI access. All samples work with `gpt-5-mini` (free tier). | ||||||
|
|
||||||
| ## Complete Sample Catalog | ||||||
|
|
||||||
| ## Samples | ||||||
| ### 🎯 Core SDK Patterns | ||||||
|
|
||||||
| ### Core Examples | ||||||
| | Sample | What It Shows | Key Techniques | | ||||||
| |--------|--------------|----------------| | ||||||
| | [**hello_copilot.py**](samples/hello_copilot.py) | Minimal example — send prompt, get response | Session management, basic async | | ||||||
| | [**streaming_chat.py**](samples/streaming_chat.py) | Token-by-token streaming output | Event handlers, real-time display | | ||||||
| | [**interactive_chat.py**](samples/interactive_chat.py) | Full terminal chat with history | Multi-turn conversations, message retrieval | | ||||||
| | [**multi_turn_agent.py**](samples/multi_turn_agent.py) | Stateful agent across turns | Session persistence, context management | | ||||||
| | [**multi_model.py**](samples/multi_model.py) | Compare gpt-4.1 vs gpt-5-mini responses | Model selection, parallel queries | | ||||||
| | [**resilient_client.py**](samples/resilient_client.py) | Retries, timeouts, error handling | Production error patterns | | ||||||
|
|
||||||
| | Script | Description | | ||||||
| |--------|-------------| | ||||||
| | [`hello_copilot.py`](samples/hello_copilot.py) | Minimal "Hello World" — send a prompt, get a response | | ||||||
| | [`streaming_chat.py`](samples/streaming_chat.py) | Stream responses token-by-token in real time | | ||||||
| | [`interactive_chat.py`](samples/interactive_chat.py) | Full interactive chat loop in the terminal | | ||||||
| | [`code_reviewer.py`](samples/code_reviewer.py) | AI-powered code review — pass any file for analysis | | ||||||
| | [`custom_tools.py`](samples/custom_tools.py) | Define custom Python functions callable by AI | | ||||||
| | [`multi_model.py`](samples/multi_model.py) | Compare responses from gpt-4.1 vs gpt-5-mini | | ||||||
| | [`file_summarizer.py`](samples/file_summarizer.py) | Summarize any text file using AI | | ||||||
| | [`git_commit_writer.py`](samples/git_commit_writer.py) | Generate conventional commit messages from staged changes | | ||||||
| ### 🔧 Advanced Features | ||||||
|
|
||||||
| ### Automation & Testing | ||||||
| | Sample | What It Shows | Key Techniques | | ||||||
| |--------|--------------|----------------| | ||||||
| | [**custom_tools.py**](samples/custom_tools.py) | Define Python functions callable by AI | `@define_tool`, Pydantic models, function calling | | ||||||
| | [**code_reviewer.py**](samples/code_reviewer.py) | AI code review with structured findings | Tool-based structured output, streaming | | ||||||
| | [**model_explorer.py**](samples/model_explorer.py) | Inspect available models and capabilities | API introspection, model metadata | | ||||||
|
|
||||||
| | Script | Description | | ||||||
| |--------|-------------| | ||||||
| | [`playwright_agent.py`](samples/playwright_agent.py) | 🌐 AI-driven browser automation with Playwright | | ||||||
| | [`log_analyzer.py`](samples/log_analyzer.py) | 📊 Analyze logs for errors, security issues, performance | | ||||||
| | [`api_test_generator.py`](samples/api_test_generator.py) | 🧪 Generate API tests from OpenAPI/Swagger specs | | ||||||
| | [`test_data_generator.py`](samples/test_data_generator.py) | 🎲 Generate realistic test data in JSON/SQL/CSV | | ||||||
| ### 🤖 Automation & Real-World Use Cases | ||||||
|
|
||||||
| ### Usage Examples | ||||||
| | Sample | What It Does | Use Cases | | ||||||
| |--------|--------------|-----------| | ||||||
| | [**playwright_agent.py**](samples/playwright_agent.py) | AI-guided browser automation | Web scraping, testing, form automation | | ||||||
| | [**log_analyzer.py**](samples/log_analyzer.py) | Analyze logs with custom tools | Error detection, security analysis, performance | | ||||||
| | [**api_test_generator.py**](samples/api_test_generator.py) | Generate pytest tests from OpenAPI specs | API testing, test automation | | ||||||
| | [**test_data_generator.py**](samples/test_data_generator.py) | Create realistic test data (JSON/SQL/CSV) | Database seeding, test fixtures | | ||||||
| | [**file_summarizer.py**](samples/file_summarizer.py) | Summarize any text file | Documentation, README generation | | ||||||
| | [**git_commit_writer.py**](samples/git_commit_writer.py) | Generate conventional commit messages | Git workflow automation | | ||||||
|
|
||||||
| ### 🧪 AI-Enhanced Testing | ||||||
|
|
||||||
| | Sample | What It Shows | Key Techniques | | ||||||
| |--------|--------------|----------------| | ||||||
| | [**pytest_ai_validation.py**](samples/pytest_ai_validation.py) | AI-enhanced pytest with intelligent assertions | AI-as-judge, `ast.parse` validation, JSON schema checks, `copilot_session` fixture | | ||||||
| | [**robot_copilot_library.py**](samples/robot_copilot_library.py) | Robot Framework keyword library for AI agents | BDD/Gherkin scenarios, keyword-driven AI testing, enterprise test integration | | ||||||
| | [**copilot_bdd.robot**](samples/copilot_bdd.robot) | BDD test suite (Given/When/Then) for AI behaviour | Robot Framework `.robot` file, Gherkin syntax, AI code generation + review | | ||||||
|
|
||||||
| **All samples include:** | ||||||
| - ✅ Complete, runnable code | ||||||
| - ✅ Type hints and documentation | ||||||
| - ✅ Error handling | ||||||
| - ✅ CLI argument parsing | ||||||
| - ✅ Tested in CI with gpt-5-mini | ||||||
|
|
||||||
| ## Proven Quality — E2E Proof | ||||||
|
|
||||||
| Unlike most SDK examples, **every sample in this repository is proven to work** end-to-end in CI: | ||||||
|
|
||||||
| ``` | ||||||
| + API_TEST_GENERATOR [OK] Test generation complete! | ||||||
| + CODE_REVIEWER [OK] Review complete | ||||||
| + CUSTOM_TOOLS [OK] Tool calls executed | ||||||
| + FILE_SUMMARIZER [OK] Summary generated | ||||||
| + GIT_COMMIT_WRITER [OK] Commit message created | ||||||
| + HELLO_COPILOT [OK] Basic prompt/response | ||||||
| + LOG_ANALYZER [OK] Log analysis complete | ||||||
| + MODEL_EXPLORER [OK] 14 models discovered | ||||||
| + MULTI_MODEL [OK] 2 models compared | ||||||
| + PYTEST_AI_VALIDATION [OK] 4/4 AI tests passed | ||||||
| + RESILIENT_CLIENT [OK] 3 prompts with retries | ||||||
| + ROBOT_COPILOT_LIBRARY [OK] 3/3 BDD scenarios passed | ||||||
| + STREAMING_CHAT [OK] Token streaming works | ||||||
| + TEST_DATA_GENERATOR [OK] Test data generated | ||||||
| + MULTI_TURN_AGENT [OK] Stateful conversation | ||||||
| + INTERACTIVE_CHAT [SKIP] Interactive (requires stdin) | ||||||
| + PLAYWRIGHT_AGENT [SKIP] Requires browser setup | ||||||
|
|
||||||
| Summary: 17/17 scenarios validated (14 run, 3 skipped for interactivity) | ||||||
| ``` | ||||||
|
|
||||||
| See [E2E workflow runs](https://github.com/Michspirit99/copilot-sdk-python/actions/workflows/agent-scenarios.yml) for full transcripts showing what each agent actually does. | ||||||
|
|
||||||
| ## Usage Examples | ||||||
|
|
||||||
| ```bash | ||||||
| # Basic usage | ||||||
|
|
@@ -82,6 +149,34 @@ python samples/api_test_generator.py swagger.json pytest | |||||
| python samples/test_data_generator.py user 50 json | ||||||
| ``` | ||||||
|
|
||||||
| ## AI-Enhanced Testing | ||||||
|
|
||||||
| These samples show how to integrate AI validation into established test frameworks — proving that Copilot SDK agents work correctly using the **same tools enterprises already use**. | ||||||
|
|
||||||
| **🧪 pytest ([pytest_ai_validation.py](samples/pytest_ai_validation.py))** | ||||||
| - AI-as-judge pattern: one AI call validates another's output | ||||||
| - Deterministic + AI assertions: `ast.parse`, `json.loads` + AI relevance checks | ||||||
| - `copilot_session` fixture for test lifecycle | ||||||
| - Runs standalone OR with `pytest -v` | ||||||
|
|
||||||
| **🤖 Robot Framework ([copilot_bdd.robot](samples/copilot_bdd.robot) + [robot_copilot_library.py](samples/robot_copilot_library.py))** | ||||||
| - BDD/Gherkin syntax: `Given I have a Copilot session / When I ask Copilot to generate code / Then the code should be valid Python` | ||||||
| - Python keyword library wrapping the entire Copilot SDK | ||||||
| - Enterprise-ready: integrates AI agent testing into existing Robot Framework suites | ||||||
| - 4 scenarios: code generation, bug detection, JSON output, concept explanation | ||||||
|
|
||||||
| ```bash | ||||||
| # Run pytest AI tests | ||||||
| pytest samples/pytest_ai_validation.py -v | ||||||
|
|
||||||
| # Run Robot Framework BDD tests | ||||||
| robot samples/copilot_bdd.robot | ||||||
|
|
||||||
| # Both also run standalone (no test framework required) | ||||||
| python samples/pytest_ai_validation.py | ||||||
| python samples/robot_copilot_library.py | ||||||
| ``` | ||||||
|
|
||||||
| ## Automation Use Cases | ||||||
|
|
||||||
| The automation scripts demonstrate practical AI-powered workflows: | ||||||
|
|
@@ -218,8 +313,11 @@ copilot-sdk-python-scripts/ | |||||
| │ ├── git_commit_writer.py # Git commit message generation | ||||||
| │ ├── playwright_agent.py # Browser automation | ||||||
| │ ├── log_analyzer.py # Log file analysis | ||||||
| │ ├── api_test_generator.py # API test generation | ||||||
| │ └── test_data_generator.py # Test data generation | ||||||
| │ ├── api_test_generator.py # API test generation | ||||||
| │ ├── test_data_generator.py # Test data generation | ||||||
| │ ├── pytest_ai_validation.py # AI-enhanced pytest testing | ||||||
| │ ├── robot_copilot_library.py # Robot Framework keyword library | ||||||
| │ └── copilot_bdd.robot # BDD test suite (Given/When/Then) | ||||||
|
Comment on lines
+316
to
+320
|
||||||
| ├── .github/ | ||||||
| │ ├── workflows/ | ||||||
| │ │ ├── ci.yml # CI validation (no live AI calls) | ||||||
|
|
@@ -233,30 +331,35 @@ copilot-sdk-python-scripts/ | |||||
|
|
||||||
| ## Why Python + Copilot SDK? | ||||||
|
|
||||||
| | Traditional Approach | This Repository | | ||||||
| **Best SDK Sample Collection Available:** | ||||||
| | This Repository | Typical SDK Examples | | ||||||
| |---|---| | ||||||
| | Create project directory | Just create a `.py` file | | ||||||
| | Write `setup.py` or `pyproject.toml` | `requirements.txt` only | | ||||||
| | Manage dependencies manually | One `pip install` command | | ||||||
| | Multiple files for simple tasks | Single file, pure Python | | ||||||
| | Project scaffolding overhead | Zero ceremony | | ||||||
|
|
||||||
| Python is already the language of choice for quick scripts—this repository shows how to make them AI-powered with minimal effort. | ||||||
| | 17 production-ready samples | 2-3 "hello world" scripts | | ||||||
| | E2E tested in CI (see runs) | Untested or manual-only | | ||||||
| | Real-world use cases | Toy examples | | ||||||
| | Error handling + best practices | Happy path only | | ||||||
| | Free tier model (gpt-5-mini) | Requires expensive models | | ||||||
| | Single-file simplicity | Complex project structure | | ||||||
|
|
||||||
| ## CI/CD | ||||||
| **Why Python?** | ||||||
| Python is already the go-to language for quick automation scripts. This repository shows how to make them AI-powered with the same simplicity you expect from Python. | ||||||
|
|
||||||
| This repository includes GitHub Actions CI that: | ||||||
| ## CI/CD & Quality | ||||||
|
|
||||||
| - ✅ Lints (ruff) | ||||||
| - ✅ Checks syntax (`compileall`) | ||||||
| - ✅ Runs import smoke tests (so samples keep working for contributors) | ||||||
| This repository includes comprehensive CI/CD: | ||||||
|
|
||||||
| Because end-to-end runs require network + authentication (and can consume quota), live AI calls are **opt-in**. | ||||||
| **Default CI** ([ci.yml](.github/workflows/ci.yml)) — Runs on every push: | ||||||
| - ✅ Lints with ruff | ||||||
| - ✅ Syntax validation (`compileall`) | ||||||
| - ✅ Import smoke tests (ensures samples stay valid) | ||||||
|
|
||||||
| - Default CI: [.github/workflows/ci.yml](.github/workflows/ci.yml) | ||||||
| - Optional E2E proof (manual): [.github/workflows/agent-scenarios.yml](.github/workflows/agent-scenarios.yml) | ||||||
| **E2E Proof** ([agent-scenarios.yml](.github/workflows/agent-scenarios.yml)) — Optional, manual trigger: | ||||||
| - ✅ Runs all 15 samples with real AI calls | ||||||
|
||||||
| - ✅ Runs all 15 samples with real AI calls | |
| - ✅ Runs all 17 samples with real AI calls (as defined in `run_agent_scenarios.py`) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The E2E badge hard-codes
17/17, butscripts/run_agent_scenarios.pynow also reports.robotfiles as scenarios, which will change the total. Update the badge (or adjust the runner) so the displayed E2E count matches the workflow output.