Add ToolMetadataExporter #992

conniey · 2025-10-28T18:03:42Z

What does this PR do?

Adds tool to export current metadata from azmcp.

GitHub issue number?

[Link to the GitHub issue this PR addresses]
https://github.com/microsoft/mcp-pr/issues/123

Pre-merge Checklist

conniey · 2025-11-05T21:42:15Z

eng/tools/ToolMetadataExporter/Services/AzmcpProgram.cs

+
+    public virtual Task<ListToolsResult?> LoadToolsDynamicallyAsync()
+    {
+        return Utility.LoadToolsDynamicallyAsync(_toolDirectory, false);


These are currently utility methods but the logic is from ToolDescriptionEvaluator where I think the logic can be shared.

conniey · 2025-11-05T21:43:35Z

eng/tools/ToolMetadataExporter/Utility.cs

+
+namespace ToolMetadataExporter;
+
+internal class Utility


Taken from ToolDescriptionEvaluator and modified. Need to update logic together.

Copilot

Pull Request Overview

This PR adds a new ToolMetadataExporter tool that exports and tracks metadata changes from the Azure MCP server (azmcp). The tool compares current tool definitions against historical data stored in Azure Data Explorer (Kusto) and records creation, update, and deletion events.

Key changes:

New ToolMetadataExporter console application that dynamically loads tools from azmcp and compares them against a Kusto datastore
Integration with Azure Data Explorer for querying existing tools and ingesting change events
Support for tracking tool lifecycle events (Created, Updated, Deleted) with version information

Reviewed Changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
eng/tools/ToolMetadataExporter/Program.cs	Entry point configuring DI, Azure authentication, and Kusto clients
eng/tools/ToolMetadataExporter/ToolAnalyzer.cs	Core analysis logic comparing current tools against datastore and detecting changes
eng/tools/ToolMetadataExporter/Utility.cs	Helper methods for executing azmcp commands and parsing tool metadata
eng/tools/ToolMetadataExporter/Services/AzmcpProgram.cs	Service wrapper for interacting with the azmcp executable
eng/tools/ToolMetadataExporter/Services/AzureMcpKustoDatastore.cs	Kusto datastore implementation for querying and ingesting tool events
eng/tools/ToolMetadataExporter/Services/IAzureMcpDatastore.cs	Interface defining datastore operations
eng/tools/ToolMetadataExporter/Models/Kusto/McpToolEvent.cs	Model representing tool lifecycle events with Kusto column mappings
eng/tools/ToolMetadataExporter/Models/Kusto/McpToolEventType.cs	Enum defining event types (Created, Updated, Deleted)
eng/tools/ToolMetadataExporter/Models/ModelsSerializationContext.cs	JSON source generation context for AOT compatibility
eng/tools/ToolMetadataExporter/Models/AzureMcpTool.cs	Record representing a tool with ID, name, and area
eng/tools/ToolMetadataExporter/AppConfiguration.cs	Configuration model for Kusto endpoints and settings
eng/tools/ToolMetadataExporter/appsettings.json	Production configuration template
eng/tools/ToolMetadataExporter/appsettings.Development.json	Development environment configuration
eng/tools/ToolMetadataExporter/Resources/queries/GetAvailableTools.kql	KQL query to retrieve latest tool states
eng/tools/ToolMetadataExporter/Resources/queries/CreateTable.kql	KQL script to create the McpToolEvents table
eng/tools/ToolMetadataExporter/ToolMetadataExporter.csproj	Project file with dependencies on Azure.Identity and Kusto SDKs
eng/tools/ToolMetadataExporter.UnitTests/ToolAnalyzerTests.cs	Empty test class placeholder for unit tests
eng/tools/ToolMetadataExporter.UnitTests/ToolMetadataExporter.UnitTests.csproj	Test project configuration
eng/tools/ToolDescriptionEvaluator/Models/McpModels.cs	Added Id property to Tool model
Directory.Packages.props	Updated Kusto package versions to 14.0.1 and added Ingest package
AzureMcp.sln	Added ToolMetadataExporter projects to solution

Copilot · 2025-11-11T19:39:23Z

eng/tools/ToolMetadataExporter/Services/AzureMcpKustoDatastore.cs

+            if (string.IsNullOrEmpty(latestEvent.ToolId))
+            {
+                throw new InvalidOperationException(
+                    $"Cannot have an event with no id. Name: {latestEvent.ToolArea}, Area: {latestEvent.ToolArea}");


The error message incorrectly displays ToolArea for the Name parameter. It should display latestEvent.ToolName instead of latestEvent.ToolArea for the name field.

Fix:

$"Cannot have an event with no id. Name: {latestEvent.ToolName}, Area: {latestEvent.ToolArea}");

Suggested change

$"Cannot have an event with no id. Name: {latestEvent.ToolArea}, Area: {latestEvent.ToolArea}");

$"Cannot have an event with no id. Name: {latestEvent.ToolName}, Area: {latestEvent.ToolArea}");

Copilot · 2025-11-11T19:39:24Z

eng/tools/ToolMetadataExporter/Utility.cs

+    private static string EscapeCharacters(string text)
+    {
+        if (string.IsNullOrEmpty(text))
+            return text;
+
+        // Normalize only the fancy “curly” quotes to straight ASCII. Identity replacements were removed.
+        return text.Replace(UnicodeChars.LeftSingleQuote, "'")
+               .Replace(UnicodeChars.RightSingleQuote, "'")


The variable dir can be null after the while loop exits if no repo root is found. The null check should occur before dereferencing dir.Parent in the loop.

Consider using a null-conditional operator:

dir = dir?.Parent;

Or check for null before the iteration:

while (dir != null) { if (File.Exists(Path.Combine(dir.FullName, "AzureMcp.sln")) || Directory.Exists(Path.Combine(dir.FullName, ".git"))) { return dir.FullName; } dir = dir.Parent; }

Copilot · 2025-11-11T19:39:24Z

eng/tools/ToolMetadataExporter/Program.cs

+                if (existing.WorkDirectory == null)
+                {
+                    string exeDir = AppContext.BaseDirectory;
+                    var repoRoot = Utility.FindRepoRoot(exeDir);
+                    existing.WorkDirectory = Path.Combine(repoRoot, ".work");
+                }


[nitpick] The PostConfigure block sets a default WorkDirectory if it's null. However, the QueriesFolder property also has a default value ("Resources/queries"), but the AzureMcpKustoDatastore constructor validates this path existence. If QueriesFolder is not explicitly set, it will use the default relative path, which may fail validation. Consider also handling QueriesFolder in PostConfigure to ensure it resolves to an absolute path.

Suggested change

if (existing.WorkDirectory == null)

{

string exeDir = AppContext.BaseDirectory;

var repoRoot = Utility.FindRepoRoot(exeDir);

existing.WorkDirectory = Path.Combine(repoRoot, ".work");

}

string exeDir = AppContext.BaseDirectory;

var repoRoot = Utility.FindRepoRoot(exeDir);

if (existing.WorkDirectory == null)

{

existing.WorkDirectory = Path.Combine(repoRoot, ".work");

}

if (string.IsNullOrEmpty(existing.QueriesFolder) || !Path.IsPathRooted(existing.QueriesFolder))

{

existing.QueriesFolder = Path.Combine(repoRoot, existing.QueriesFolder ?? "Resources/queries");

}

Copilot · 2025-11-11T19:39:24Z

eng/tools/ToolMetadataExporter/Utility.cs

+            }
+
+            if (cliArtifact != null)
+            {


The argumentsToUse variable is incorrectly assigned when isDll is true. It should append arguments to the DLL path, but currently it only includes the DLL path without the actual arguments.

Fix:

var argumentsToUse = isDll ? $"{cliArtifact.FullName} {arguments}" : arguments;

Copilot · 2025-11-11T19:39:25Z

eng/tools/ToolMetadataExporter/ToolAnalyzer.cs

+        _logger = logger;
+
+        _workingDirectory = configuration.Value.WorkDirectory ?? throw new ArgumentNullException(nameof(AppConfiguration.WorkDirectory));
+        ;


Extraneous semicolon on an empty line. This should be removed.

Suggested change

;

Copilot · 2025-11-11T19:39:26Z

eng/tools/ToolMetadataExporter/Utility.cs

+
+    private static async Task SaveToolsToJsonAsync(ListToolsResult toolsResult, string filePath)
+    {
+        try
+        {
+            // Normalize only tool and option descriptions instead of escaping the entire JSON document
+            if (toolsResult.Tools != null)


This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.

Copilot · 2025-11-11T19:39:26Z

eng/tools/ToolMetadataExporter/Utility.cs

+
+    internal static async Task<string> ExecuteAzmcpAsync(string arguments, bool isCiMode = false, bool checkErrorCode = true)
+    {
+        // Locate azmcp artifact across common build outputs (servers/core, Debug/Release)
+        var exeDir = AppContext.BaseDirectory;
+        var repoRoot = FindRepoRoot(exeDir);
+        var searchRoots = new List<string>
+            {
+                Path.Combine(repoRoot, "servers", "Azure.Mcp.Server", "src", "bin", "Debug"),
+                Path.Combine(repoRoot, "servers", "Azure.Mcp.Server", "src", "bin", "Release")
+            };


This foreach loop immediately maps its iteration variable to another variable - consider mapping the sequence explicitly using '.Select(...)'.

Copilot · 2025-11-11T19:39:26Z

eng/tools/ToolMetadataExporter/Utility.cs

+            }
+        }
+
+        if (cliArtifact == null)
+        {
+            if (isCiMode)
+            {
+                return string.Empty; // Graceful fallback in CI
+            }
+
+            throw new FileNotFoundException("Could not locate azmcp CLI artifact in Debug/Release outputs under servers.");
+        }


Disposable 'Process' is created but not disposed.

Copilot · 2025-11-11T19:39:26Z

eng/tools/ToolMetadataExporter/ToolAnalyzer.cs

+            catch (Exception ex)
+            {
+                _logger.LogError(ex, "Error writing to {FileName}", outputFile);
+            }


Generic catch clause.

Suggested change

catch (Exception ex)

{

_logger.LogError(ex, "Error writing to {FileName}", outputFile);

}

catch (IOException ex)

{

_logger.LogError(ex, "IO error writing to {FileName}", outputFile);

}

catch (UnauthorizedAccessException ex)

{

_logger.LogError(ex, "Unauthorized access writing to {FileName}", outputFile);

}

Copilot · 2025-11-11T19:39:27Z

eng/tools/ToolMetadataExporter/Utility.cs

+                }
+            }
+
+            var writerOptions = new JsonWriterOptions
+            {


Generic catch clause.

github-project-automation bot added this to Azure MCP Server Oct 28, 2025

github-project-automation bot moved this to Untriaged in Azure MCP Server Oct 28, 2025

joshfree assigned conniey Oct 29, 2025

joshfree added server-Azure.Mcp Azure.Mcp.Server tools-Telemetry labels Oct 29, 2025

joshfree added this to the 2025-11 milestone Oct 29, 2025

joshfree moved this from Untriaged to In Progress in Azure MCP Server Oct 29, 2025

conniey commented Nov 5, 2025

View reviewed changes

conniey changed the title ~~(WIP) Add ToolMetadataExporter~~ Add ToolMetadataExporter Nov 11, 2025

conniey marked this pull request as ready for review November 11, 2025 19:33

conniey requested a review from a team as a code owner November 11, 2025 19:33

Copilot AI review requested due to automatic review settings November 11, 2025 19:33

conniey requested review from a team as code owners November 11, 2025 19:33

conniey requested review from KarishmaGhiya, chidozieononiwu, g2vinay, hallipr, joshfree, msalaman and wbreza November 11, 2025 19:33

Copilot started reviewing on behalf of conniey November 11, 2025 19:34 View session

Copilot finished reviewing on behalf of conniey November 11, 2025 19:36

conniey force-pushed the exporter branch from 6087283 to aff7f4e Compare November 11, 2025 19:37

Copilot AI reviewed Nov 11, 2025

View reviewed changes

alzimmermsft approved these changes Nov 11, 2025

View reviewed changes

conniey added 3 commits November 24, 2025 05:49

Add PackageVersion for Kusto.Data and Kusto.Ingest

3dfd4f9

Add Id to shared McpModels.

1342541

Initial commit

3cba80c

conniey added 29 commits November 24, 2025 05:49

Adding tool to AzureMcp.sln

5c9fd41

Remove AoT

4af70a6

Add initial mock up

4e3adf7

Split to use Query and Ingestion endpoint.

0bbad1a

Add Query to create table.

fe07ce8

Add LaunchSettings.

49e88c3

Set Default queries folder.

725dc59

Adding Appsettings.json. Copy to output

c9ba1c1

Update Json serialization to output Enum names.

7207b1a

Start host

b27c786

Move JsonOutput parsing. Fix command line arguments to pass to exe

ebd8733

Pass in command rather than just name in changes.

ec1e149

Update Logging

901cb27

Change interface to use List. Update Ingestion settings for singlejson

238c33b

Fix column mappings

46e1d4f

Change to use Direct Ingestion client

5bb8bcc

Fix formatting issues

08dd0f5

Fix formatting

12fd8b5

Add Friend assembly

2d0f10b

Add unit test project

dc326ce

Make method virtual

792c53e

Fix formatting issue.

e13fd7e

Add header

bdd8fb5

Add dev settings

a454034

Add header

be54a7f

Add ServerName support.

901c5a9

Remove test template.

bf8aee9

Add ServerName

d244e0e

Propagate cancellation

bbded74

conniey force-pushed the exporter branch from aff7f4e to bbded74 Compare November 24, 2025 13:53

	$"Cannot have an event with no id. Name: {latestEvent.ToolArea}, Area: {latestEvent.ToolArea}");
	$"Cannot have an event with no id. Name: {latestEvent.ToolName}, Area: {latestEvent.ToolArea}");

-            catch (Exception ex)
-            {
-                _logger.LogError(ex, "Error writing to {FileName}", outputFile);
-            }
+            catch (IOException ex)
+            {
+                _logger.LogError(ex, "IO error writing to {FileName}", outputFile);
+            }
+            catch (UnauthorizedAccessException ex)
+            {
+                _logger.LogError(ex, "Unauthorized access writing to {FileName}", outputFile);
+            }

Add ToolMetadataExporter #992

Are you sure you want to change the base?

Add ToolMetadataExporter #992

Uh oh!

Conversation

conniey commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

GitHub issue number?

Pre-merge Checklist

Uh oh!

conniey Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

conniey Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

conniey commented Oct 28, 2025 •

edited

Loading