receiver/prometheusreceiver: add option to fallback to collector starttime #36365
Conversation
ridwanmsharif
force-pushed
the
ridwanmsharif/starttime-fallback
branch
2 times, most recently
from
4ac0c41 to
05b85e8
Compare
…ttime This change adds an option to the metric adjuster to use an approximation of the collector starttime as a fallback for the start time of scraped cumulative metrics. This is useful when no start time is found and when the collector starts up alongside its targets (like in serverless environments or sidecar approaches). Signed-off-by: Ridwan Sharif <ridwanmsharif@google.com>
ridwanmsharif
force-pushed
the
ridwanmsharif/starttime-fallback
branch
from
05b85e8 to
d67a526
Compare
|
The code itself looks correct! To be completely honest, I'm pretty new to this component and haven't used it myself. I noticed we have waaaay too many fallbacks for Created Timestamps; that looks weird 🤔. Do I understand correctly that the flow is like this:
Did I get the flow correctly? It makes me wonder, when would an OpenTelemetry metric not have |
| func NewStartTimeMetricAdjuster(logger *zap.Logger, startTimeMetricRegex *regexp.Regexp, useCollectorStartTimeFallback bool) MetricsAdjuster { | ||
| var fallbackStartTime *time.Time | ||
| if useCollectorStartTimeFallback { | ||
| now := time.Now() |
There was a problem hiding this comment.
Rather than assume that this is always called at/near start time of collector, should we use the github.com/shirou/gopsutil/v4/host package to request uptime like hostmetricsreceiver does for boottime?
There was a problem hiding this comment.
Is that correct in a containerized environment, or would it give the start time of the host?
There was a problem hiding this comment.
I don't know the answer to that question, would need to be tested (I don't currently have capacity to test myself).
| func NewStartTimeMetricAdjuster(logger *zap.Logger, startTimeMetricRegex *regexp.Regexp, useCollectorStartTimeFallback bool) MetricsAdjuster { | ||
| var fallbackStartTime *time.Time | ||
| if useCollectorStartTimeFallback { | ||
| now := time.Now() |
There was a problem hiding this comment.
This won't be the collector start time, but the start time of this instance of the component. If the pipeline is stopped and restarted then this will result in a different timestamp even though the process has not restarted. This is a subtlety, but could be very confusing for someone using OpAMP to manage collector instances. I'm not sure whether there's a way to get a reliable process start time from the collector host. The system uptime is almost certainly not the correct value to use here. Perhaps the best option is just to clarify in the description of the new configuration field that the approximated start time will be relative to the component start time and not necessarily the collector start time.
There was a problem hiding this comment.
can we populate it using an init function, or a variable outside of metric adjuster?
There was a problem hiding this comment.
Populating a variable in an init function could work. It would be executed once near the start of the process.
| if stma.fallbackStartTime == nil { | ||
| return err | ||
| } | ||
| stma.logger.Warn("Couldn't get start time for metrics. Using fallback start time.", zap.Error(err)) |
There was a problem hiding this comment.
Does this need to be at the Warn level? Wouldn't this be a fairly high-rate log entry if none of the processed metrics have a start time?
This is the prometheus receiver, so the concern here is prometheus metrics not having a start time and we want to ensure that the |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
Description
This change adds an option to the metric adjuster to use an approximation of the collector starttime as a fallback for the start time of scraped cumulative metrics. This is useful when no start time is found and when the collector starts up alongside its targets (like in serverless environments or sidecar approaches).
Link to tracking issue
Fixes #36364
Testing
Added unit test for this config option
Documentation
Config option added to the README.