Overflows should emit logs and metrics

[carsonip]

Related to elastic/apm-server#11362 and elastic/apm-server#11117

• Overflows should emit logs and metrics such that overflows are observable.
• Create a dashboard to observe overflows.

[carsonip]

In apm-server for txmetrics we had:

	totalOverflow := metrics.servicesOverflow + metrics.perSvcTxnGroupsOverflow + metrics.txnGroupsOverflow
	monitoring.ReportInt(V, "active_groups", metrics.activeGroups)
	monitoring.ReportNamespace(V, "overflowed", func() {
		monitoring.ReportInt(V, "services", metrics.servicesOverflow)
		monitoring.ReportInt(V, "per_service_txn_groups", metrics.perSvcTxnGroupsOverflow)
		monitoring.ReportInt(V, "txn_groups", metrics.txnGroupsOverflow)
		monitoring.ReportInt(V, "total", totalOverflow)
	})

@axw do you think #86 provides sufficient insight into overflows? It does seem that it lacks the granularity that we had in the past, but I'm not sure if that's useful. Given that the next step will be to build dashboards on top of the new metrics in #86 , just wanted to confirm we are happy with what we have now after #86.

[axw]

I think we should go with what we have, and add to it as needed. It may not be complete, but I think what's there is correct.

Regarding what we used to have:

• I think it should be possible to measure active_groups from the indexed metric documents
• overflowed.total doesn't seem useful if we know the number of metric-type overflows
• overflowed.per_service_txn_groups doesn't seem useful either, since it isn't per-service (and I think we probably shouldn't make it per-service, as it could lead to way too many metrics)

[carsonip]

Closing as all work on the topic has been completed.