Skip to content

Commit c263dfc

Browse files
patnikoCopilot
patniko
and
Copilot
authored
Add E2E scenario tests/examples for all SDK languages (#512)
* Commit base e2e across SDKs * Add C# samples to 8 representative scenarios Add csharp/ subdirectories with Program.cs and csproj for: - transport/stdio, transport/tcp - bundling/fully-bundled - tools/no-tools, tools/custom-agents - sessions/streaming - callbacks/permissions - prompts/system-message All samples reference the local .NET SDK via ProjectReference. Each verify.sh updated with dotnet build/run steps. Added C# build artifacts to .gitignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix go.mod replace paths for test scenarios All 30 go.mod files had incorrect relative paths to the SDK Go module. Scenarios live at test/scenarios/<cat>/<scenario>/go/ (5 levels deep), so the replace directive needs ../../../../../go instead of ../../../../go. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace copilot-core with Copilot CLI across all test scenarios copilot-core is not a product name. Updated 63 files to use the correct terminology: - Prose/comments: "Copilot CLI" - Binary name in code/commands: copilot - COPILOT_CLI_PATH env var: unchanged (already correct) - Dockerfile ENTRYPOINT/COPY: copilot - docker-compose service name: copilot-cli Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add scenario build verification workflow for PR checks Runs build verification for all test/scenarios across 4 languages (TypeScript, Python, Go, C#) as parallel jobs. Triggered on PRs that touch scenarios or SDK source, and on push to main. Build-only — no E2E execution (no API keys or Copilot CLI needed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add full language parity: all 34 scenarios × 4 languages Every scenario now has TypeScript, Python, Go, and C# implementations. New additions: - C# for 26 scenarios (auth, bundling, callbacks, modes, prompts, sessions, tools, transport) - Python + Go + C# for 3 BYOK scenarios (anthropic, azure, ollama) - Python + Go + C# stubs for multi-user scenarios (SKIP pattern) All 136 scenario builds verified: 34 TS, 34 PY, 34 GO, 34 CS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CI caching and justfile targets for scenario builds - Add npm, Go module, and NuGet caching to scenario-builds.yml - Add just scenario-build, scenario-verify, scenario-build-lang targets Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Quality fixes: C# in all verify.sh, fix stubs, consistent patterns - Add C# build/run steps to all 26 verify.sh scripts that were missing them - Replace infinite-sessions TS stub with real implementation - Rework modes: remove filesystem-preset, rename cli-preset→default and minimal-preset→minimal with real implementations in all 4 languages - Fix C# patterns across all 33 scenarios: consistent await using, StartAsync/StopAsync, proper disposal 33 scenarios × 4 languages = 132 builds, all passing. 2 multi-user scenarios remain as stubs (require memory FS features). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen Python CI: py_compile + import check instead of AST-only Install the Python SDK and use py_compile for proper compilation checking, plus verify the copilot module is importable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update scenario model references to claude-sonnet-4.6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove soft-pass fallbacks in verify.sh scenario scripts Replace else/elif fallback branches that always passed with proper failure reporting. Previously, when the expected grep pattern wasn't found, the test would emit a warning but still count as passed. Now these cases correctly report failure and increment the FAIL counter. Changed 18 files with 21 soft-pass fixes across: - 10 standard else-branch fallbacks (modes, prompts, tools, sessions) - 3 elif-branch fallbacks (callbacks, streaming, virtual-filesystem) - 3 multi-branch fixes with both 'partial' and 'got response' fallbacks (concurrent-sessions, multi-user-long-lived, multi-user-short-lived) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen tools scenario verifications - tool-filtering: use word-boundary grep (-w) for blacklisted tools to avoid false positives on substrings like 'bashing' - no-tools: change question to directly request bash tool usage; update verify grep to check for inability patterns (can't, cannot, unable) - virtual-filesystem: require both 'Virtual filesystem contents' AND 'plan.md' in output; fix dead elif branch - custom-agents: tighten grep to only match 'researcher' or 'Research' instead of also matching generic tool names - skills: add lowercase 'skill' to grep pattern for broader matching - mcp-servers: replace soft-pass (non-empty output) with meaningful content grep; add separate failure message for pattern mismatch Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support latest .NET * Move to haiku * Fix scenario tests: paths, verifications, streaming, and parallel execution - Fix relative paths in TS package.json (43 files) and Python requirements.txt (33 files) - Add RollForward to C# csproj files for .NET 8/10 compat - Remove soft-pass fallbacks in verify.sh — tests now hard-fail on missing patterns - Fix Go permissions bug (req.Kind → req.ToolName) and add ToolName to SDK type - Fix Python/Go availableTools: empty list was omitted instead of sent as [] - Fix streaming event names (assistant.message.chunk → assistant.message_delta) - Fix TS streaming subscription (session.on('event') → typed subscription) - Fix Python streaming enum comparison (event.type.value) - Add permission handlers to Go skills scenario - Switch scenarios to claude-haiku-4.5 for faster execution - Parallelize verify.sh with live progress bar and per-SDK status icons - Fix parallel pip install race with pre-install and import check - Remove go.sum files from tracking - Remove hardcoded OAuth client ID from C# gh-app scenario Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore go.sum files needed for CI builds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert ToolName addition to Go PermissionRequest — use Extra map instead The ToolName field doesn't exist on PermissionRequest in other SDKs. The scenario test now reads toolName from the Extra map to stay consistent without modifying SDK types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use o4-mini for reasoning-effort scenario tests claude-haiku-4.5 does not support the reasoningEffort configuration, causing all 4 SDK scenario tests to fail. Switch to o4-mini which supports reasoning effort. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 886e5aa commit c263dfc

File tree

369 files changed

+15056
-6
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

369 files changed

+15056
-6
lines changed

‎.github/workflows/scenario-builds.yml‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
name: "Scenario Build Verification"
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- "test/scenarios/**"
7+
- "nodejs/src/**"
8+
- "python/copilot/**"
9+
- "go/**/*.go"
10+
- "dotnet/src/**"
11+
- ".github/workflows/scenario-builds.yml"
12+
push:
13+
branches:
14+
- main
15+
paths:
16+
- "test/scenarios/**"
17+
- ".github/workflows/scenario-builds.yml"
18+
workflow_dispatch:
19+
merge_group:
20+
21+
permissions:
22+
contents: read
23+
24+
jobs:
25+
# ── TypeScript ──────────────────────────────────────────────────────
26+
build-typescript:
27+
name: "TypeScript scenarios"
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@v6
31+
32+
- uses: actions/setup-node@v6
33+
with:
34+
node-version: 22
35+
36+
- uses: actions/cache@v4
37+
with:
38+
path: ~/.npm
39+
key: ${{ runner.os }}-npm-scenarios-${{ hashFiles('test/scenarios/**/package.json') }}
40+
restore-keys: |
41+
${{ runner.os }}-npm-scenarios-
42+
43+
# Build the SDK so local file: references resolve
44+
- name: Build SDK
45+
working-directory: nodejs
46+
run: npm ci --ignore-scripts
47+
48+
- name: Build all TypeScript scenarios
49+
run: |
50+
PASS=0; FAIL=0; FAILURES=""
51+
for dir in $(find test/scenarios -path '*/typescript/package.json' -exec dirname {} \; | sort); do
52+
scenario="${dir#test/scenarios/}"
53+
echo "::group::$scenario"
54+
if (cd "$dir" && npm install --ignore-scripts 2>&1); then
55+
echo "✅ $scenario"
56+
PASS=$((PASS + 1))
57+
else
58+
echo "❌ $scenario"
59+
FAIL=$((FAIL + 1))
60+
FAILURES="$FAILURES\n $scenario"
61+
fi
62+
echo "::endgroup::"
63+
done
64+
echo ""
65+
echo "TypeScript builds: $PASS passed, $FAIL failed"
66+
if [ "$FAIL" -gt 0 ]; then
67+
echo -e "Failures:$FAILURES"
68+
exit 1
69+
fi
70+
71+
# ── Python ──────────────────────────────────────────────────────────
72+
build-python:
73+
name: "Python scenarios"
74+
runs-on: ubuntu-latest
75+
steps:
76+
- uses: actions/checkout@v6
77+
78+
- uses: actions/setup-python@v6
79+
with:
80+
python-version: "3.12"
81+
82+
- name: Install Python SDK
83+
run: pip install -e python/
84+
85+
- name: Compile and import-check all Python scenarios
86+
run: |
87+
PASS=0; FAIL=0; FAILURES=""
88+
for main in $(find test/scenarios -path '*/python/main.py' | sort); do
89+
dir=$(dirname "$main")
90+
scenario="${dir#test/scenarios/}"
91+
echo "::group::$scenario"
92+
if python3 -m py_compile "$main" 2>&1 && python3 -c "import copilot" 2>&1; then
93+
echo "✅ $scenario"
94+
PASS=$((PASS + 1))
95+
else
96+
echo "❌ $scenario"
97+
FAIL=$((FAIL + 1))
98+
FAILURES="$FAILURES\n $scenario"
99+
fi
100+
echo "::endgroup::"
101+
done
102+
echo ""
103+
echo "Python builds: $PASS passed, $FAIL failed"
104+
if [ "$FAIL" -gt 0 ]; then
105+
echo -e "Failures:$FAILURES"
106+
exit 1
107+
fi
108+
109+
# ── Go ──────────────────────────────────────────────────────────────
110+
build-go:
111+
name: "Go scenarios"
112+
runs-on: ubuntu-latest
113+
steps:
114+
- uses: actions/checkout@v6
115+
116+
- uses: actions/setup-go@v6
117+
with:
118+
go-version: "1.24"
119+
cache: true
120+
cache-dependency-path: test/scenarios/**/go.sum
121+
122+
- name: Build all Go scenarios
123+
run: |
124+
PASS=0; FAIL=0; FAILURES=""
125+
for mod in $(find test/scenarios -path '*/go/go.mod' | sort); do
126+
dir=$(dirname "$mod")
127+
scenario="${dir#test/scenarios/}"
128+
echo "::group::$scenario"
129+
if (cd "$dir" && go build ./... 2>&1); then
130+
echo "✅ $scenario"
131+
PASS=$((PASS + 1))
132+
else
133+
echo "❌ $scenario"
134+
FAIL=$((FAIL + 1))
135+
FAILURES="$FAILURES\n $scenario"
136+
fi
137+
echo "::endgroup::"
138+
done
139+
echo ""
140+
echo "Go builds: $PASS passed, $FAIL failed"
141+
if [ "$FAIL" -gt 0 ]; then
142+
echo -e "Failures:$FAILURES"
143+
exit 1
144+
fi
145+
146+
# ── C# ─────────────────────────────────────────────────────────────
147+
build-csharp:
148+
name: "C# scenarios"
149+
runs-on: ubuntu-latest
150+
steps:
151+
- uses: actions/checkout@v6
152+
153+
- uses: actions/setup-dotnet@v5
154+
with:
155+
dotnet-version: "8.0.x"
156+
157+
- uses: actions/cache@v4
158+
with:
159+
path: ~/.nuget/packages
160+
key: ${{ runner.os }}-nuget-scenarios-${{ hashFiles('test/scenarios/**/*.csproj') }}
161+
restore-keys: |
162+
${{ runner.os }}-nuget-scenarios-
163+
164+
- name: Build all C# scenarios
165+
run: |
166+
PASS=0; FAIL=0; FAILURES=""
167+
for proj in $(find test/scenarios -name '*.csproj' | sort); do
168+
dir=$(dirname "$proj")
169+
scenario="${dir#test/scenarios/}"
170+
echo "::group::$scenario"
171+
if (cd "$dir" && dotnet build --nologo 2>&1); then
172+
echo "✅ $scenario"
173+
PASS=$((PASS + 1))
174+
else
175+
echo "❌ $scenario"
176+
FAIL=$((FAIL + 1))
177+
FAILURES="$FAILURES\n $scenario"
178+
fi
179+
echo "::endgroup::"
180+
done
181+
echo ""
182+
echo "C# builds: $PASS passed, $FAIL failed"
183+
if [ "$FAIL" -gt 0 ]; then
184+
echo -e "Failures:$FAILURES"
185+
exit 1
186+
fi

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11

22
# Documentation validation output
33
docs/.validation/
4+
.DS_Store

‎dotnet/src/Client.cs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,9 +105,15 @@ public CopilotClient(CopilotClientOptions? options = null)
105105
_options = options ?? new();
106106

107107
// Validate mutually exclusive options
108-
if (!string.IsNullOrEmpty(_options.CliUrl) && (_options.UseStdio || _options.CliPath != null))
108+
if (!string.IsNullOrEmpty(_options.CliUrl) && _options.CliPath != null)
109109
{
110-
throw new ArgumentException("CliUrl is mutually exclusive with UseStdio and CliPath");
110+
throw new ArgumentException("CliUrl is mutually exclusive with CliPath");
111+
}
112+
113+
// When CliUrl is provided, disable UseStdio (we connect to an external server, not spawn one)
114+
if (!string.IsNullOrEmpty(_options.CliUrl))
115+
{
116+
_options.UseStdio = false;
111117
}
112118

113119
// Validate auth options with external server

‎go/types.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -640,7 +640,7 @@ type createSessionRequest struct {
640640
ReasoningEffort string `json:"reasoningEffort,omitempty"`
641641
Tools []Tool `json:"tools,omitempty"`
642642
SystemMessage *SystemMessageConfig `json:"systemMessage,omitempty"`
643-
AvailableTools []string `json:"availableTools,omitempty"`
643+
AvailableTools []string `json:"availableTools"`
644644
ExcludedTools []string `json:"excludedTools,omitempty"`
645645
Provider *ProviderConfig `json:"provider,omitempty"`
646646
RequestPermission *bool `json:"requestPermission,omitempty"`
@@ -671,7 +671,7 @@ type resumeSessionRequest struct {
671671
ReasoningEffort string `json:"reasoningEffort,omitempty"`
672672
Tools []Tool `json:"tools,omitempty"`
673673
SystemMessage *SystemMessageConfig `json:"systemMessage,omitempty"`
674-
AvailableTools []string `json:"availableTools,omitempty"`
674+
AvailableTools []string `json:"availableTools"`
675675
ExcludedTools []string `json:"excludedTools,omitempty"`
676676
Provider *ProviderConfig `json:"provider,omitempty"`
677677
RequestPermission *bool `json:"requestPermission,omitempty"`

‎justfile‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,3 +117,112 @@ validate-docs-go:
117117
validate-docs-cs:
118118
@echo "=== Validating C# documentation ==="
119119
@cd scripts/docs-validation && npm run validate:cs
120+
121+
# Build all scenario samples (all languages)
122+
scenario-build:
123+
#!/usr/bin/env bash
124+
set -euo pipefail
125+
echo "=== Building all scenario samples ==="
126+
TOTAL=0; PASS=0; FAIL=0
127+
128+
build_lang() {
129+
local lang="$1" find_expr="$2" build_cmd="$3"
130+
echo ""
131+
echo "── $lang ──"
132+
while IFS= read -r target; do
133+
[ -z "$target" ] && continue
134+
dir=$(dirname "$target")
135+
scenario="${dir#test/scenarios/}"
136+
TOTAL=$((TOTAL + 1))
137+
if (cd "$dir" && eval "$build_cmd" >/dev/null 2>&1); then
138+
printf " ✅ %s\n" "$scenario"
139+
PASS=$((PASS + 1))
140+
else
141+
printf " ❌ %s\n" "$scenario"
142+
FAIL=$((FAIL + 1))
143+
fi
144+
done < <(find test/scenarios $find_expr | sort)
145+
}
146+
147+
# TypeScript: npm install
148+
(cd nodejs && npm ci --ignore-scripts --silent 2>/dev/null) || true
149+
build_lang "TypeScript" "-path '*/typescript/package.json'" "npm install --ignore-scripts"
150+
151+
# Python: syntax check
152+
build_lang "Python" "-path '*/python/main.py'" "python3 -c \"import ast; ast.parse(open('main.py').read())\""
153+
154+
# Go: go build
155+
build_lang "Go" "-path '*/go/go.mod'" "go build ./..."
156+
157+
# C#: dotnet build
158+
build_lang "C#" "-name '*.csproj' -path '*/csharp/*'" "dotnet build --nologo -v quiet"
159+
160+
echo ""
161+
echo "══════════════════════════════════════"
162+
echo " Scenario build summary: $PASS passed, $FAIL failed (of $TOTAL)"
163+
echo "══════════════════════════════════════"
164+
[ "$FAIL" -eq 0 ]
165+
166+
# Run the full scenario verify orchestrator (build + E2E, needs real CLI)
167+
scenario-verify:
168+
@echo "=== Running scenario verification ==="
169+
@bash test/scenarios/verify.sh
170+
171+
# Build scenarios for a single language (typescript, python, go, csharp)
172+
scenario-build-lang LANG:
173+
#!/usr/bin/env bash
174+
set -euo pipefail
175+
echo "=== Building {{LANG}} scenarios ==="
176+
PASS=0; FAIL=0
177+
178+
case "{{LANG}}" in
179+
typescript)
180+
(cd nodejs && npm ci --ignore-scripts --silent 2>/dev/null) || true
181+
for target in $(find test/scenarios -path '*/typescript/package.json' | sort); do
182+
dir=$(dirname "$target"); scenario="${dir#test/scenarios/}"
183+
if (cd "$dir" && npm install --ignore-scripts >/dev/null 2>&1); then
184+
printf " ✅ %s\n" "$scenario"; PASS=$((PASS + 1))
185+
else
186+
printf " ❌ %s\n" "$scenario"; FAIL=$((FAIL + 1))
187+
fi
188+
done
189+
;;
190+
python)
191+
for target in $(find test/scenarios -path '*/python/main.py' | sort); do
192+
dir=$(dirname "$target"); scenario="${dir#test/scenarios/}"
193+
if python3 -c "import ast; ast.parse(open('$target').read())" 2>/dev/null; then
194+
printf " ✅ %s\n" "$scenario"; PASS=$((PASS + 1))
195+
else
196+
printf " ❌ %s\n" "$scenario"; FAIL=$((FAIL + 1))
197+
fi
198+
done
199+
;;
200+
go)
201+
for target in $(find test/scenarios -path '*/go/go.mod' | sort); do
202+
dir=$(dirname "$target"); scenario="${dir#test/scenarios/}"
203+
if (cd "$dir" && go build ./... >/dev/null 2>&1); then
204+
printf " ✅ %s\n" "$scenario"; PASS=$((PASS + 1))
205+
else
206+
printf " ❌ %s\n" "$scenario"; FAIL=$((FAIL + 1))
207+
fi
208+
done
209+
;;
210+
csharp)
211+
for target in $(find test/scenarios -name '*.csproj' -path '*/csharp/*' | sort); do
212+
dir=$(dirname "$target"); scenario="${dir#test/scenarios/}"
213+
if (cd "$dir" && dotnet build --nologo -v quiet >/dev/null 2>&1); then
214+
printf " ✅ %s\n" "$scenario"; PASS=$((PASS + 1))
215+
else
216+
printf " ❌ %s\n" "$scenario"; FAIL=$((FAIL + 1))
217+
fi
218+
done
219+
;;
220+
*)
221+
echo "Unknown language: {{LANG}}. Use: typescript, python, go, csharp"
222+
exit 1
223+
;;
224+
esac
225+
226+
echo ""
227+
echo "{{LANG}} scenarios: $PASS passed, $FAIL failed"
228+
[ "$FAIL" -eq 0 ]

‎python/copilot/client.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -481,7 +481,7 @@ async def create_session(self, config: Optional[SessionConfig] = None) -> Copilo
481481

482482
# Add tool filtering options
483483
available_tools = cfg.get("available_tools")
484-
if available_tools:
484+
if available_tools is not None:
485485
payload["availableTools"] = available_tools
486486
excluded_tools = cfg.get("excluded_tools")
487487
if excluded_tools:
@@ -652,7 +652,7 @@ async def resume_session(
652652

653653
# Add available/excluded tools if provided
654654
available_tools = cfg.get("available_tools")
655-
if available_tools:
655+
if available_tools is not None:
656656
payload["availableTools"] = available_tools
657657

658658
excluded_tools = cfg.get("excluded_tools")

0 commit comments

Comments
 (0)