on-change-repeat feature for prometheus output

[hellt]

Prometheus doesn't play well with data that is not monotonously exported. ON_CHANGE subscriptions is that kind of data.

Here I asked how should prometheus user get the last value of a metric that appears irregularly (on_change event) and the reply was that it is against prom principles and the data that is present, should be exported all the time.

That got me thinking if gnmic can get the best from both worlds

• support the gnmi on_change to save routers cpu and netw bandwidth
• and make prometheus happy by exporting the data monotonously

and I am interested to see your opinion @karimra if something like this makes sense and is implementable:

Design

when gnmic creates an event format, it adds additional label to stream-mode=sample|on_change. The prom output when takes this event message checks if the stream-mode is on_change. If it is, gnmic will check if on-change-repeat is set, and if both conditions are true gnmic will export the value periodically using the value from the last received on_change event

[karimra]

That is how the prometheus output works now.
Regardless if the value was received from a sample or on-change subscription, prom output stores the last received value ( until expiration , default 1 minute ) and exports it whenever there is a scrape request from the prometheus client.
The stored value is overwritten if a value with the same hash is received and has a newer timestamp.

What needs to be added is exporting without timestamp, but that can be fixed on the prometheus client side as well.

[hellt]

tested with a long expiration timer.
The metrics indeed do exist on a scrape request, but as you mentioned, because the TS is fixed, Prom will eventually (after 5m by default) will report the metric as stale and wont return its value.

exporting without TS might do the trick, but I wonder if all the metrics should inherit that property, or only the ones that come from on_change subscription?

[karimra]

I believe it should be all metrics, let the prom client set the ts to $scrape_time, that is the expected behavior of prometheus

[hellt]

it seems that making indefinite expiration timeout for all metrics is worse than having it only for on-change metrics
without having expiration for sampled data we cut off the chance to detect stale metrics for sampled data, which is desirable.
I think end goal should be to support non-expirable metrics for on-change only

[karimra]

if you want to detect stale metrics, combining no-ts and no-expiry is not the way to go.
In an ideal (gnmi) world, the target would support heartbeat-interval for an on-change subscription ( I don't know any vendor which supports it now )

Prometheus pulls the metrics from the prom output, so using on-change metrics vs sampled won't matter.
It saves some bandwidth, but is it worth the added complexity on the collector side?

Making on-change metrics non-expirable is probably possible, need to expose more details about the metrics to the prom output.

What is harder to do is deleting metrics per target if that specific target is down.
The above will require to:

• cache metrics not only by hash but also add a second cache that maps a target to its metrics hash.
• find a way to signal to the output, that a certain target connection is down ( or being retried )
• once that signal is received, delete that target's metrics by getting their hashes and go thought the metrics cache to delete them

[hellt]

as I mentioned in the first message, to detect stale metrics all is needed is to make only on-change metrics non expirable, the sampled ones will follow the ouputs configured expiration setting.

on-change subscriptions can be detected if the event format will carry a label with the subscription info.
I think deletion of metrics from coll cache is an orthogonal feature