Adding OpenTelemetry OTLP exporter support

[dasiths]

This PR adds the ability to export telemetry to an OTLP endpoint and has an example of collection telemetry locally.

It also includes an example of how to collect telemetry locally using the grafana/otel-lgtm container.

[stuartleeks]

This is looking good - thanks 😄

I will give it a proper test tomorrow but have left a couple of comments/questions

[dasiths]

Any update on this @stuartleeks

[stuartleeks]

Thanks for you patience @dasiths!

There are a couple of linter errors that look like they've been introduced in this PR but other than that this looks good to me: https://github.com/microsoft/aoai-api-simulator/actions/runs/11256643932?pr=54#summary-31299020880

For the "too many arguments" errors, I'd recommend suppressing the error (see the other methods in that same file). The other errors look like things that should be resolved.

[stuartleeks]

@dasiths I've pushed changes to resolve the conflict on latest main and address the quick-fix linter warnings. There are a couple remaining that I'm less clear on the path forwards. I've left a comment on the changed lines to make it easy to find - would love your help in understanding what the lines are doing :-)

[dasiths]

@dasiths I've pushed changes to resolve the conflict on latest main and address the quick-fix linter warnings. There are a couple remaining that I'm less clear on the path forwards. I've left a comment on the changed lines to make it easy to find - would love your help in understanding what the lines are doing :-)

I replied to your comment up top. We may need to supress this linter error. :)

[dasiths]

@stuartleeks is there anything required to get this landed? Sorry if I missed anything.

[stuartleeks]

@dasiths - I wanted to look at a slightly different way to handle the different conditions to see if it is possible to remove the private access. I started looking at it but and wanted to test the original code but wasn't able to get the exported metrics showing up locally. Sadly I didn't have much time to spend on it at that point. I do want to come back to it as I'd still like to get this working 😄

[dasiths]

@dasiths - I wanted to look at a slightly different way to handle the different conditions to see if it is possible to remove the private access. I started looking at it but and wanted to test the original code but wasn't able to get the exported metrics showing up locally. Sadly I didn't have much time to spend on it at that point. I do want to come back to it as I'd still like to get this working 😄

Can confirm the metrics work but they take a bit of time to show up. It requires doing something like a small load test to really see the metrics light up.

[stuartleeks]

@dasiths I'm considering two potential alternatives currently.

The first option is to remove the call to configure_azure_monitor and explicitly wire up the Azure exporters if the App Insights config is set. I believe this would provide access to the relevant objects and remove the need to use the private accessors.

The second option is to remove the Azure otel distro and just use the standard exporters. When running locally, this would use the grafana/otel-lgtm container to capture/view the telemetry. When deployed to Azure it would either use the open telemetry collector or the corresponding feature in container apps

The second option would make the code simpler but would mean that telemetry when running locally wouldn't be send to App Insights (unless we also run the collector locally). Do you see this as an issue or is your primary goal to have local telemetry?