Start running checks based on the time of creation, not when the agent receives them #739
Labels
enhancement
New feature or request
Comments
|
After thinking about this more, I realized that the problem is that if we wait until the next time the check should have run, we might end up with gaps in the data. We still have some situations where we can wait, but not all of them. |
mem
added a commit
that referenced
this issue
Jul 3, 2024
When the agent restarts, we might end up in a situation where we are running a check too often. For example, if a check is configured to run once every 10 minutes, and the check ran 1 minute ago, if we run immediately, we will have two executions within that 10 minute window. Since we have to publish samples once every two minutes, we cannot wait for 9 minutes, because we end up with a gap. Instead, do run immediately to avoid that. But if the check ran 9 minutes ago, we can align with the expectation by waiting for 1 minute instead of a random value. Presumably the check ran 9 minutes ago, and the sample was replicated 7, 5, 3 and 1 minutes ago. If we wait for 1 minute, we would end up running the check when it was expected to run. In order to actually fix this issue the agents would have to persist data across runs. It might be possible to do this by offloading publishing to another service. Fixes: #739 Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently checks start running based on when the agent received them, not when they were defined.
If the check is transferred to a different agent instance (Grafana Labs issue), or if the agent restarts (Grafana Labs issue + user-visible issue) this resets. This means that the check might end up running slightly more often than expected.
To fix this, look at the creation time (modification?) of the check and compute the next time the check should run. This would preserve the frequency of the runs.
The very first time the check runs is close to the creation time, but it's never going to be the creation time. For checks with low frequencies, this means the first time is "far" in the future.
If C is the creation time and R > C is the time the agent received the check definition, right now the check starts to run at R + random(2 minutes).
In order to get that first check running as soon as possible, we should look at R - C, and if that's less than some threshold, we should run the check immediately. Otherwise we wait until C + f, where f is the frequency found in the check's definition.
The text was updated successfully, but these errors were encountered: