Custom counter sometimes not tracked

[kasszz]

Dear reader,

I have an issue that it seems random if the counter is actually counting every time the server re-deploys. Have you experienced something like this? For me this is a real head scratcher.

import { collectDefaultMetrics, Counter, register } from 'prom-client'

const PREFIX = 'test_'

// Need to clear the register for HMR
register.clear()
collectDefaultMetrics({ prefix: PREFIX })

const counter1 = new Counter({
  name: `${PREFIX}counter_1`,
  help: 'Counter 1',
})


export { counter1, register }

Context
We are running a NextJS server in Kubernetes with multiple pods. So every pod is running the exact same code.

The default collectDefaultMetrics metrics are working every time for every pod.

We use this counter to count the amount of SSR pages we have to render. So we counter1.inc() every time the getServersideProps() is invoked.

In Grafana you can see the problem happening, this is a graph where it's summed by pod, you can see every new deployment that we do, the amount of pods that are reporting is different:

I see this also happen when requesting the /metrics route from the website. Sometimes I only see the #HELP and #TYPE line of the custom counter, and sometimes the values are popping up. This is due to me hitting different pods every refresh, and some pods are not counting. So it's not a Grafana problem.

Hopefully I provided enough information and you see I do something stupid in this setup.
Thanks in advance, Casper

[zbjornson]

I don't have any advice, sorry. Probably need to add debugging code to see which pods/processes are reporting and if there's a pattern.