SDK e2e tests never run against a real CLI binary in CI

[darthmolen]

Summary

The SDK ships 10 e2e test files in nodejs/test/e2e/ that spawn the CLI via CopilotClient.start(). These tests would have caught the --headless removal in CLI v0.0.410 before it broke every SDK consumer. They did not catch it because CI never installs or runs the CLI binary.

Evidence

1. CI installs dependencies with --ignore-scripts

The nodejs-sdk-tests.yml workflow runs:

npm ci --ignore-scripts

The --ignore-scripts flag suppresses the postinstall hook that @github/copilot uses to install its platform-specific binary. Without it, the CLI binary does not exist on disk. Any e2e test that calls client.start() either silently skips or fails — either way, it tests nothing.

2. No COPILOT_CLI_PATH is set

The e2e test harness respects COPILOT_CLI_PATH for locating the CLI. The CI workflow never sets this variable and never installs the CLI through any other mechanism.

3. Scenario tests compile but never execute

The scenario-builds.yml workflow compiles scenarios across four languages (TypeScript, Python, Go, C#) but never runs them. The test/scenarios/verify.sh orchestrator exists and accepts COPILOT_CLI_PATH, but no workflow invokes it.

4. Unit tests mock away CLI spawning entirely

nodejs/test/client.test.ts uses vi.spyOn() to mock internal methods. It tests option validation (URL parsing, port conflicts) but never exercises startCLIServer(), --headless, or --stdio.

Impact

CLI v0.0.410 removed --headless --stdio on February 13. Nine days later, SDK v0.1.26-preview.0 still passes --headless unconditionally (lines 693-704 of dist/client.js). A single CI job that installed the CLI and ran client.start() would have caught this before the CLI shipped.

The --acp flag was added in CLI v0.0.397 — a 13-version window where both flags coexisted. The SDK had ample runway to detect the deprecation and prepare.

Suggested fix

Add a CI job to nodejs-sdk-tests.yml that:

1. Installs @github/copilot with postinstall scripts (drop --ignore-scripts or add a separate step)
2. Sets COPILOT_CLI_PATH to the installed binary
3. Runs the existing e2e tests against it

The tests already exist. The infrastructure to run them does not.

Related

• CLI v0.0.410+ auto-update breaks all SDK versions — --headless --stdio no longer accepted #530 — CLI v0.0.410+ breaks all SDK versions (--headless removed)
• Node SDK: getBundledCliPath() breaks in CJS bundles (VS Code extensions) #528 — getBundledCliPath uses ESM-only import.meta.resolve, breaks CJS consumers

[github-actions]

Thank you for this detailed bug report! This is an excellent analysis of a critical gap in our CI infrastructure.

Labels applied:

• bug — E2E tests are not running against real CLI binaries as intended
• testing — Test infrastructure issue
• priority/high — This prevented detection of breaking changes (e.g., --headless removal in CLI v0.0.410)

Your evidence is thorough and the suggested fix is actionable. The fact that existing e2e tests exist but aren't being executed in CI is a serious oversight that should be addressed promptly.

AI generated by Issue Triage Agent

[SteveSandersonMS]

I think there's some misunderstanding about --acp and --headless here. We're not using ACP as the protocol for this SDK and --headless is not removed.

CI here uses the bundled version of Copilot CLI, which is the only version that we make any claims about compatibility for. As of today that's version 0.0.414 and previously was 0.0.411.

We do have an E2E gap in that we don't exercise the auto-downloader during E2E tests. But apart from that I think the coverage is accurate.

Hope you don't mind me closing this issue! I'll do so because there's quite a few confusing points in the report and I think it will be clearer for everyone if - for any follow up - we have a new clean issue that doesn't inherit any confusion.