Periodic metric reader causing 1,000 object references to be read when calculating the size of the object graph

[JBodkin-Amphora]

I've been seeing the following log being output in the periodic metric reader thread, however I'm unsure if this is actually a problem?

{
  "@timestamp":"2022-08-05T15:07:30.459Z",
  "message":"The configured limit of 1,000 object references was reached while attempting to calculate the size of the object graph. Severe performance degradation could occur if the sizing operation continues. This can be avoided by setting the CacheManger or Cache <sizeOfPolicy> elements maxDepthExceededBehavior to \"abort\" or adding stop points with @IgnoreSizeOf annotations. If performance degradation is NOT an issue at the configured limit, raise the limit value using the CacheManager or Cache <sizeOfPolicy> elements maxDepth attribute. For more information, see the Ehcache configuration documentation.",
  "logger_name":"net.sf.ehcache.pool.sizeof.ObjectGraphWalker",
  "thread_name":"PeriodicMetricReader-1",
  "level":"WARN"
}

[jack-berg]

I'm not familiar with Ehcache, and nothing in the metrics SDK uses it. Not sure why it would be traversing object graphs in PeriodicMetricReader threads.

[nluk]

@jack-berg I don't know why a PeriodicMetricReader thread performed the read - in my case it's prometheus-http-%d threads doing that, but that's with OTEL_METRICS_EXPORTER=prometheus set on the java agent. Regardless, I encountered kind of the same behaviour - I didn't reach the warning, because the ObjectGraphWalker failed reflection on unopened java.util module. The culprit was this micrometer metric for ehcache, which is reported by OpenTelemetry on behalf of micrometer.

As this is a Spring-specific thing, the required properties to disable it are for example:

management.metrics.enable.cache.size=false
management.metrics.enable.cache.local=false

as mentioned here.

[jack-berg]

So if I understand correctly:

• Micrometer metrics are being bridged into OpenTelemetry
• Micrometer metrics are trying to report some metrics about ehcache
• In the process of "attempting to calculate calculate the size of the [ehcache] object graph", some limit was exceeded.

If that's correct, then you'll want to configure some combination of ehcache, micrometer, or OpenTelemetry to avoid the issue. I would guess that the management.metrics.enable.cache.* properties you've concluded are used to configure spring's configuration of micrometer. Based on this blurb from the original error message "This can be avoided by setting the CacheManger or Cache elements maxDepthExceededBehavior to "abort" or adding stop points with @IgnoreSizeOf annotations.", I would guess there's ehcache configuration options to help avoid the issue.

On the OpenTelemetry side of things, you can use views to disable the problematic metric. I would bet its an async (i.e. observable) instrument. Disabling async instruments via views means that the callback which is invoked to fetch the current value when reading metrics is never invoked.

[nluk]

Yes, all your statements are correct. This is the code that either prints the warning or throws. It's just recursive object graph traversal via reflection. No issues on OpenTelemetry side, it just tries to fetch the metric value. I responded to make sure that anyone who stumbles upon this is gonna know one of possible reasons for this error being logged.