Validate go-docappender metrics are produced when monitoring is enabled

[kruskall]

We should make sure that go-docappender metrics are produced when monitoring is enabled.

flush latency and similar metrics are important to debug issues

[carsonip]

go-docappender itself exposes metrics as OTel metrics, as opposed to elastic agent monitoring where EA polls metrics from APM server. The go-docappender metrics should be available if instrumentation.enabled is true in apm-server.yml because apm-server configures otel bridge.

[carsonip]

I looked into this again. apm-server does get stats from the appender and publish libbeat monitoring metrics, other than the fact that go-docappender publishes OTel metrics itself (what I mentioned in the previous comment).

The observation is that there are no output.elasticsearch.bulk_requests metrics published by this code, but I see output.events.* metrics. This may reveal a bug somewhere. I wonder if using the monitoring.Default registry is causing a problem. This issue has to be investigated.

[carsonip]

Might be related to #8383

[carsonip]

Observation: when monitoring.enabled: true and monitoring.elasticsearch is configured, the metrics are sent to ES directly under .monitoring-beats-7-*. The output.elasticsearch.* fields exist in source but are unmapped.

[carsonip]

In that case there may be 2 problems:

1. the .monitoring-beats-7-* does not have the mappings for some of our fields
2. metricbeat beat module may be ignoring the fields because some of our fields do not fall into "exported fields" defined by metricbeat. Pending confirmation.

[carsonip]

metricbeat beat module may be ignoring the fields because some of our fields do not fall into "exported fields" defined by metricbeat. Pending confirmation.

(2) is not right. The mapping is actually defined in EA integration. Created #13475 to take care of missing metrics due to EA integration apm-server metrics mapping.

[carsonip]

It should be closed by #13475