[googlecloudmonitoring] config collection interval & more flexible metric filtering

[chenlujjj]

Component(s)

receiver/googlecloudmonitoring

Is your feature request related to a problem? Please describe.

1. The default collection interval is 5 mins, which is too large for monitoring
2. The current metric filtering is not flexible enough, if we want to collect a bunch of metrics, the list would be too long

Describe the solution you'd like

1. decrease the default collection interval
2. enrich the metric filtering capabilities, such as include/exclude, regex, etc

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/googlecloudmonitoring: @dashpole @TylerHelmuth @abhishek-at-cloudwerx

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

A 5 minute interval was requested by the cloud monitoring team, so I don't think we should lower the default. Lower intervals can cause query performance problems.

Better filtering makes sense. We should try and be consistent with other filtering-related configuration in collector components if we can, rather than invent something new here.

[chenlujjj]

Hi @dashpole

A 5 minute interval was requested by the cloud monitoring team,

could you tell me where I can find the request? 5 minute interval means that the alert can delay at most 5 minutes, and I'm worried about that.

[dashpole]

I'm a Googler, so I reached out to them internally. @kjordy are there any details you can share here as to why a 5 minute interval is recommended for scraping ListTimeSeries?

[chenlujjj]

Is it possible to use a smaller interval restriction, like 1 minute?

I also checked the documentation of Splunk SignalFx connection to GCP, the minimum poll rate can be 1 minute.

[chenlujjj]

hello , @kjordy, could you give extra information of the collection interval ?

[dashpole]

This is the response I got from the monitoring API team:

This is a balance between freshness and QPS. This is also the default of many 3P vendors like Datadog. There are both costs and quotas associated with queries. Scraping at a low interval may be ok for a small number of metrics but is not recommended for doing bulk export.

[chenlujjj]

Hi @dashpole , thanks for the clarification, I understand the default interval setting now.

I think maybe it's better to separate the default interval and the minimum interval settings ?

Datadog has 5 min as default and 1 min as minimum:

Now the default interval also serves as minimum limit:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/googlecloudmonitoringreceiver/config.go#L31-L33

[dashpole]

yeah, SGTM. Lets set a minimum of 1 minute. We should also document the costs of setting a lower interval.