Add local AI metrics dashboard for self-hosted usage analytics

[Copilot]

Implements opt-in local metrics collection and visualization for GitHub Copilot usage. Data stays on-disk via globalState, no external transmission. Metrics computed on-demand only.

Changes

Configuration

• github.copilot.metrics.enabled (boolean, default false) - enables local collection
• github.copilot.metrics.retentionDays (number, 7-365, default 90) - auto-prune threshold

Domain Model (src/extension/aiMetrics/common/metrics.ts)

Defines 5 metric categories with ~15 measurement types:

• Token Usage: total tokens, per-model/feature breakdown, cache hit ratio
• Model Distribution: usage counts by model and provider
• Code Acceptance: NES/completion acceptance rates, rejection reason breakdown
• Feature Usage: chat message count, NES opportunity count, completion count
• Performance: avg/p95 TTFT, fetch time, debounce time

Storage (src/extension/aiMetrics/node/aiMetricsStorageService.ts)

• Events stored as aiMetrics.events.<YYYY-MM-DD>[] in globalState
• Day-grouped for efficient range queries
• On-demand aggregation via computeMetrics(events, startDate, endDate)
• Automatic pruning on extension activation

Telemetry Interceptor (src/extension/aiMetrics/common/aiMetricsCollector.ts)

ITelemetryService decorator that:

• Intercepts events matching patterns: provideInlineEdit.*, conversation.message, ghostText.*, request.response
• Extracts minimal fields (timestamp, event name, model, tokens, measurements)
• Forwards to storage asynchronously
• Passes through to base telemetry service unchanged

Dashboard (src/extension/aiMetrics/vscode-node/aiMetricsDashboardPanel.ts)

Webview panel with:

• Time range selector (Today/Week/Month/All)
• Overview cards: total tokens, acceptance rate, top model, event count
• Breakdown tables: tokens by model/feature, feature usage, performance metrics
• Manual refresh only (no background processing)
• VS Code theme-aware styling

Command

github.copilot.viewMetrics - opens dashboard

Example Usage

// settings.json
{
  "github.copilot.metrics.enabled": true,
  "github.copilot.metrics.retentionDays": 90
}

Command Palette → "Copilot: View AI Metrics Dashboard" → Select time range → Refresh

Testing

Unit tests for domain model (date formatting, time range calculations) - 7/7 passing

Original prompt

Plan: Local AI Metrics Dashboard for Developer Self-Hosting

Build a local developer dashboard to visualize personal AI usage metrics from GitHub Copilot, including tokens consumed, models used, code acceptance rates, completions usage, and NES (Next Edit Suggestions) performance. Telemetry data is intercepted and stored locally with 90-day default retention (configurable), enabled via a new github.copilot.metrics.* setting separate from GitHub telemetry controls. Dashboard refreshes on manual reload only, with on-demand aggregation to avoid background CPU usage.

Steps:

1. Add configuration schema to package.json contributions under github.copilot.metrics.enabled (boolean, default false), github.copilot.metrics.retentionDays (number, default 90, min 7, max 365), and expose these via IConfigurationService in src/extension/configuration/.

2. Create metrics domain model in src/extension/aiMetrics/common/metrics.ts defining ~15-20 metric types organized as: Token Usage (totalTokens, tokensByModel, tokensByFeature, cachedTokensRatio), Model Distribution (modelUsageCount, providerDistribution), Code Acceptance (nesAcceptanceRate, completionAcceptanceRate, rejectionReasonBreakdown), Feature Usage (chatMessageCount, nesOpportunityCount, completionCount), and Performance (avgTTFT, avgFetchTime, avgDebounceTime).

3. Implement storage service at src/extension/aiMetrics/node/aiMetricsStorageService.ts using context.globalState with schema aiMetrics.events.<YYYY-MM-DD>[] to store events grouped by day, including methods addEvent() (appends to today's array), getEventsInRange(startDate, endDate) (retrieves raw events), pruneOldData() (called only on extension activation and dashboard refresh), and computeMetrics(events) (on-demand aggregation).

4. Build telemetry interceptor at src/extension/aiMetrics/common/aiMetricsCollector.ts as an ITelemetryService decorator/wrapper that captures relevant events (provideInlineEdit.*, conversation.message, ghostText.*, request.response) when github.copilot.metrics.enabled is true, extracts minimal metric-relevant fields (timestamp, event name, key measurements), forwards to AiMetricsStorageService.addEvent(), and passes through to original telemetry service unchanged.

5. Create webview dashboard at src/extension/aiMetrics/webview/aiMetricsDashboardPanel.ts following src/extension/completions-core/vscode-node/extension/src/panelShared/baseSuggestionsPanel.ts pattern, with dashboard HTML/CSS/JS at src/extension/aiMetrics/webview/dashboard.html using VS Code Webview UI Toolkit, implementing message handler for refresh requests that queries storage, computes metrics on-demand, and posts results back to webview.

6. Build dashboard UI in src/extension/aiMetrics/webview/dashboard.html with sections: header with time-range selector (Today/Week/Month/All) and manual refresh button, metric overview cards (total tokens with trend, acceptance rate %, top model used, session count), charts section (token consumption timeline chart, model distribution donut chart, acceptance rate trends, feature usage bar chart), and performance panel (avg TTFT, avg fetch time, p95 metrics).

7. Register command and services in src/extension/extension/vscode/extension.ts by instantiating AiMetricsStorageService and AiMetricsCollector, wrapping the existing ITelemetryService instance with collector when metrics enabled, registering github.copilot.viewMetrics command in package.json that creates/shows dashboard panel, and calling pruneOldData() once during activation to clean up expired events.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.