How to query inconsistent data points?

[Knud3]

I am testing Mimir if it could replace our Carbon TSDB.

So far it seems adequate, but I'm not familiar with Prometheus queries and have this strange problem with it.

First, I have set scrape interval as one second in Grafana data source settings, because it is our shortest used interval.

Second, when querying using last_over_time(foo[$__interval]) I can get those inconsistent datapoints.

TL;DR: Problem now is, that each data point is followed by duplicate data point one second apart from it as you can see here in screenshot. If I set longer scrape interval it got more far apart.

Top panel is query from Carbon: groupByNodes(removeEmptySeries(foo.bar.info.http_server_requests.*.GET.200.SUCCESS.None.value_mean), 'average', 2, 4, 5)

And bottom one from Mimir: avg(last_over_time(graphite_untagged{__n000__="foo", __n001__="bar", __n002__="info", __n003__="http_server_requests", __n005__="GET", __n006__="200", __n007__="SUCCESS", __n008__="None", __n009__="value_mean"}[$__interval])) by(__n002__, __n004__, __n005__)

Any hints how to solve this problem?

These metrics are coming from carbon-relay-ng through graphite-proxy-writes if thats matter.

Grafana's representative confirmed the problem and recommended moving this to an issue.
Originally posted by @Knud3 in #6483

[pracucci]

First, I have set scrape interval as one second in Grafana data source settings, because it is our shortest used interval.

The scrape interval in Grafana datasource should be set to the highest scrape interval you have, and not the shortest one. The problem is that if you set it on the shortest one but then you query metrics which have an higher scraping interval, then you may experience side effects.

last_over_time(foo[$__interval])

In the Grafana panel (or the Explore) you should be able to inspect the actual query executed. What's the interval value calculated by Grafana?

Top panel is query from Carbon

I'm not familiar with Carbon support in Grafana. However, looking at the "dots" in the two panels, I've the feeling they're not configured the exact same. My feeling is that the top panel (Carbon) has been configured with the option to draw a line between 2 points, while the lower one doesn't. For example, the light blue line in the top panel has just 2 dots and a straight line connecting it, while the green series has multiple dots for the same time range in which the light blue line has only 2 dots. In a Prometheus/Mimir panel, each dot would be displayed for each "query step" and if for a step there's a missing data point (no dot) then the dots on the left/right of that step wouldn't be connected unless the feature is enabled for the Grafana panel.

[Knud3]

The scrape interval in Grafana datasource should be set to the highest scrape interval you have, and not the shortest one.

How it is so? If I set it higher, then it also set every panel query interval to that. It needs to be shortest possible. At least for the default.

you may experience side effects.

Any examples of this?

What's the interval value calculated by Grafana?

[1s]

I've the feeling they're not configured the exact same

Mimir panel is copy of Carbon panel. Only query is configure different.

while the lower one doesn't

You can see lines between "ghost" data points in Mimir panel.

[Knud3]

Any update? @pracucci

[Knud3]

I found a dirty workaround...

First, I tried to set the Mimir/Prometheus data source Scrape interval to 500ms and the panel Min step to 1 second. It worked, but setting Min step to every panel would not be efficient.

Then I tried a 999ms Scrape interval just for fun, and it seemed to work without needing to touch the Min step setting.

Of course, the Connect null values setting needs to be on.

Now... Anyway to get this working properly without this dirty workaround? @pracucci @npazosmendez