Copilot CLI does not recognize HTTP-based custom MCP server tools despite successful gateway connection

[lupinthe14th]

Description

Copilot CLI agent does not recognize or invoke custom MCP server tools configured via mcp-servers: frontmatter, despite the MCP gateway successfully starting and connecting to all servers.

Environment

• gh-aw: v0.50.6
• awf: v0.23.0
• MCP Gateway: v0.1.5
• Copilot CLI model: claude-sonnet-4.6
• Runner: ubuntu-latest

Workflow Configuration

mcp-servers:
  custom-server:
    url: "https://example.com/mcp-endpoint"
    headers:
      API-KEY: "${{ secrets.API_KEY }}"
      APP-KEY: "${{ secrets.APP_KEY }}"
    allowed:
      - "tool_a"
      - "tool_b"
      - "tool_c"

tools:
  github:
    toolsets: [issues]

Observed Behavior

1. MCP Gateway starts successfully

All servers connect and health check passes:

Health response body: {"status":"healthy","specVersion":"1.8.0","gatewayVersion":"v0.1.5","servers":{"custom-server":{"status":"running","uptime":0},"github":{"status":"running","uptime":0},"safeoutputs":{"status":"running","uptime":0}}}
✓ custom-server: connected
✓ github: connected
✓ safeoutputs: connected
✓ All checks passed (3 succeeded, 0 skipped)
Copilot configuration written to /home/runner/.copilot/mcp-config.json

2. Copilot CLI does not recognize custom MCP tools

The agent reads skill files, then immediately determines that custom MCP tools are unavailable without attempting to call any of them, and falls back to error handling.

3. gh aw mcp inspect shows 0 custom server tools

$ gh aw mcp inspect <workflow-name>
# Shows only 41 GitHub MCP tools, 0 custom server tools

$ gh aw mcp list-tools custom-server <workflow-name>
# Exit code 1, empty output

Expected Behavior

Copilot CLI should discover and invoke the custom MCP tools configured in mcp-servers: frontmatter via the MCP gateway.

Reproduction

1. Configure an HTTP-based custom MCP server in workflow frontmatter
2. Compile and run the workflow
3. Observe that the MCP gateway connects successfully but the Copilot CLI agent does not use the tools

Notes

• The stdio-based GitHub MCP server (41 tools) is correctly discovered
• Only HTTP-based custom MCP servers appear to be affected

Possibly Related Issues

• Agent tools from MCP are not visible when agent itself is a tool in Copilot CLI >= 0.0.361 copilot-cli#693 (Agent tools from MCP not visible)
• Copilot appears to not use github mcp server copilot-cli#268 (Copilot appears to not use MCP server)

[lupinthe14th]

Additional Finding: Root Cause in Gateway Logs

After further investigation, the MCP Gateway logs reveal the actual error:

[WARN] [backend] Failed to register tools from datadog (took 268.911618ms): backend error listing tools: code=-32603, message=HTTP 400: Bad Request

Timeline

1. Gateway configures HTTP MCP backend for the custom server — succeeds
2. Gateway sends tools/list request to the server — fails with HTTP 400: Bad Request
3. Gateway health check still reports "status":"running" despite tool registration failure
4. Copilot CLI receives no tools for the custom server

Tested Variations (all produce the same HTTP 400)

• MCP Gateway v0.1.5 and v0.1.6
• gh-aw v0.50.6 and v0.50.7
• With and without network: allowed: [defaults]
• Different HTTP header name formats

Observations

• The stdio-based GitHub MCP server's tools/list succeeds and returns 41 tools
• Only HTTP-based MCP servers fail at the tools/list step
• The 400 Bad Request suggests the Gateway's tools/list request format may be incompatible with the Streamable HTTP MCP protocol used by the server

[lupinthe14th]

Root Cause Identified: MCP Gateway Missing initialize Handshake

After further investigation, we identified the root cause of the HTTP 400 error.

Finding

The Datadog MCP server uses Streamable HTTP transport (MCP protocol version 2025-03-26), which requires a session establishment flow:

1. Client sends initialize request → Server returns Mcp-Session-Id header
2. Subsequent requests (tools/list, tools/call, etc.) must include the Mcp-Session-Id header

Verification

We verified this by calling the Datadog MCP endpoint directly (bypassing the MCP Gateway):

Step 1 - initialize succeeds:

POST https://mcp.datadoghq.com/api/unstable/mcp-server/mcp
Body: {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0.0"}}}

→ HTTP 200, returns Mcp-Session-Id header

Step 2 - tools/list with session ID succeeds:

POST https://mcp.datadoghq.com/api/unstable/mcp-server/mcp
Header: Mcp-Session-Id: <session-id-from-step-1>
Body: {"jsonrpc":"2.0","id":2,"method":"tools/list"}

→ HTTP 200, returns full tool list (20+ tools)

Without session ID - tools/list fails (reproduces the gateway error):

POST https://mcp.datadoghq.com/api/unstable/mcp-server/mcp
Body: {"jsonrpc":"2.0","id":1,"method":"tools/list"}

→ HTTP 400: "Invalid session ID"

Conclusion

The MCP Gateway (v0.1.6) sends tools/list directly to the Datadog HTTP MCP endpoint without first establishing a session via initialize. The Datadog server rejects this with HTTP 400 (Invalid session ID), which the gateway logs as:

[WARN] [backend] Failed to register tools from datadog: backend error listing tools: code=-32603, message=HTTP 400: Bad Request

The gateway likely needs to implement the full Streamable HTTP session lifecycle (initialize → get session ID → use session ID in subsequent requests) for HTTP-based MCP servers that require it.