Return an http error during scraping if metrics collide when escaped to underscores #1641

ywwg · 2024-10-08T18:27:10Z

This change prevents a possible issue when UTF-8 metrics are scraped by a system that does not support UTF-8. The metric names will be escaped, and if two metrics escape to the same legacy name, they will appear in the exposition "twice", causing collision problems. Therefore, during scraping we return an http error if two metrics escape to the same legacy string.

This change creates a new uint64 per metric and two new maps in the Registry. The new maps are only populated if the hash of the original metric data contains UTF-8 data. This means memory usage will be lower in the near term and will go up in the long term or for OTLP-heavy users. Eventually (i.e. Prom 4.0 or later) we could deprecate the compatibility modes with legacy systems to save this memory, but it's also not clear to me that the memory usage will be that high.

To avoid performance issues checking for collisions on every scrape, the actual collision detection is done at registration time but only reported at scrape time.

ywwg · 2024-10-09T14:43:26Z

So the edge case here is:

collision mode is Compat by default
a UTF-8 metric is created in an init function
the collision mode is flipped to UTF-8
an identical UTF-8 metric is created

This should collide, but because the first metric's hash was Compat mode, it doesn't match the hash of the second metric, and we don't detect the collision.

I think this is another case where we can say "don't do that", but I'm not sure how best to define what "that" is. In general, I think users should not define metrics in init() functions. OR, we could say, don't define UTF-8 metrics in init functions.

ywwg · 2024-10-09T16:02:46Z

Another option is to store hashes for both UTF-8 and compat. This adds 64bits per metric and another collectorsByID map (since we'll need to store both the compat and utf8 collector ids depending on the mode). don't love that either

ywwg · 2024-10-09T16:04:10Z

I am leaning towards having a second map in the registerer that only checks for compat fqname+labelname collisions. It's a memory increase but it gets around all these issues.

ywwg · 2024-10-09T18:32:37Z

prometheus/desc.go

@@ -140,15 +142,45 @@ func (v2) NewDesc(fqName, help string, variableLabels ConstrainableLabels, const
 		d.err = fmt.Errorf("duplicate label names in constant and variable labels for metric %q", fqName)
 		return d
 	}
+
+	d.id, d.dimHash = makeHashes(labelNames, labelValues, help, true)
+	d.compatID, d.compatDimHash = makeHashes(labelNames, labelValues, help, false)


this mutates labelNames so it's not pretty

ywwg · 2024-10-09T18:35:02Z

prometheus/registry.go

 	uncheckedCollectors   []Collector
 	pedanticChecksEnabled bool
+	utf8Collision         bool


So we're adding a uint64, empty struct, and Collector interface obj for every metric registered. This could be a lot of memory with many millions of metrics. Perhaps we could only store the compat IDs if they are different from the default?

(I have now implemented this)

ywwg · 2024-10-09T18:35:55Z

prometheus/registry.go

@@ -391,6 +481,10 @@ func (r *Registry) Unregister(c Collector) bool {
 	defer r.mtx.Unlock()

 	delete(r.collectorsByID, collectorID)
+	if _, exists := r.collectorsByCompatID[compatID]; !exists {


there is other cleanup needed

ywwg · 2024-10-09T20:10:54Z

I think this new approach is a decent solution. We can flip UTF-8 mode on and off at will and we don't pay a huge memory or performance penalty (that I can see)

prometheus/registry_test.go

ArthurSens · 2024-10-10T13:03:28Z

prometheus/desc.go

@@ -57,6 +57,8 @@ type Desc struct {
 	// must be unique among all registered descriptors and can therefore be
 	// used as an identifier of the descriptor.
 	id uint64
+	// compatID is similar to id, but is a hash of all the relevant names escaped with underscores.
+	compatID uint64


When I first saw this variable I thought it was a typo for compactID, but I see you're are using compact is other places as well.

What does compat means? 😅

And still related to this variable, have we thought about using id to store the hash of the already normalized descriptor(instead of creating a new hash)?

"compatible." as in, "compatible with legacy systems." I need to pick better names for sure

have we thought about using id to store the hash of the already normalized descriptor?

do you mean only storing the normalized descriptor? We need both in case the user switches the mode to UTF-8 comparison with no legacy compatibility. Or do you mean something else?

Yes, I meant just storing the normalized descriptor. Good point that's it's possible to change the validation in-flight!

Not sure why someone would like to do that though 🤔

for the record (as discussed in slack), it's because metrics are often created in init() functions before the user has a chance to actually configure the correct escaping scheme. So we have to be ready for both scenarios.

ywwg · 2024-10-15T18:00:49Z

prometheus/promhttp/http.go

+		if hasEscapedCollisions {
+			switch contentType.ToEscapingScheme() {
+			case model.UnderscoreEscaping, model.DotsEscaping:
+				opts.ErrorLog.Println("error: one or more metrics collide when escaped")


need a better error message?

maybe something like

"One or more metrics collide when UTF-8 characters in name/label are translated to underscores"?

ArthurSens

Looking good!

Since we always allow registration regardless of collision, I would just move the test closer to the HTTP package. We can keep the same test cases you've added, but the test would look similar to this:

func TestEscapedCollisions(t *testing.T) {
	reg := prometheus.NewRegistry()
	reg.MustRegister(prometheus.NewCounter(prometheus.CounterOpts{
		Name: "test_metric",
		Help: "A test metric with underscores",
	}))
	reg.MustRegister(prometheus.NewCounter(prometheus.CounterOpts{
		Name: "test.metric",
		Help: "A test metric with dots",
	}))


	
	handler := HandlerFor(reg, HandlerOpts{})
	writer := httptest.NewRecorder()
	request, _ := http.NewRequest("GET", "/metrics", nil)
	request.Header.Add(acceptEncodingHeader, "the header that requires escaping")
	handler.ServeHTTP(writer, request)
	require.Equal(t, "some http error", writer.Code)

	request, _ := http.NewRequest("GET", "/metrics", nil)
	request.Header.Add(acceptEncodingHeader, "the header that allows utf8")
	handler.ServeHTTP(writer, request)
	require.Equal(t, "no erro error", writer.Code)
}

prometheus/registry_test.go

ywwg · 2024-10-16T15:05:32Z

I would just move the test closer to the HTTP package.

I'd like to keep the current unit test since the implementation is a little weird, but I will add this too.

ywwg · 2024-10-16T15:38:41Z

I'll squash this down one we're done with notes

ywwg · 2024-10-16T15:40:43Z

prometheus/promhttp/http_test.go

+		Help: "A test metric with dots",
+	}))
+
+	handler := HandlerFor(reg, HandlerOpts{


this is the sucky part, it's necessary to specify the registry here or else we don't detect the collision case (because Gatherer does not know about collisions)

ArthurSens

LGTM!

Just one comment about adding another dependency, sorry about that 😬

prometheus/registry.go

ArthurSens · 2024-10-16T20:26:33Z

prometheus/registry_test.go

+			require.NoError(t, err)
+			err = reg.Register(tc.counterB())
+			if !tc.expectErr {
+				require.NoError(t, err)
+			} else {
+				require.Error(t, err)
+			}
+			require.Equal(t, tc.expectLegacyCollision, reg.HasEscapedCollision())


Sorry for creating the confusion here, that's my bad! Bartek made a good point for not using require here, would you mind reverting the change? 🙈

We always have problems when we introduce new dependencies, let's avoid it if we can

…to underscores Signed-off-by: Owen Williams <owen.williams@grafana.com>

Signed-off-by: Owen Williams <owen.williams@grafana.com>

ywwg · 2024-10-17T17:57:03Z

prometheus/promhttp/http.go

@@ -134,6 +136,7 @@ func HandlerForTransactional(reg prometheus.TransactionalGatherer, opts HandlerO
 			}
 		}
 	}
+	hasEscapedCollisions = reg.HasEscapedCollision()


hmm... should I use the singular or the plural? :)

ywwg · 2024-10-17T17:57:11Z

prometheus/promhttp/http_test.go

+		Help: "A test metric with dots",
+	}))
+
+	handler := HandlerFor(reg, HandlerOpts{})


ywwg · 2024-10-17T17:58:26Z

prometheus/registry.go

@@ -194,6 +205,10 @@ func (gf GathererFunc) Gather() ([]*dto.MetricFamily, error) {
 	return gf()
 }

+func (gf GathererFunc) HasEscapedCollision() bool {


this is the most risky change. both mimir and avalanche use gathererfunc

nevermind, avalanche doesn't use it

https://github.com/grafana/mimir/blob/main/pkg/api/handlers.go#L279

bwplotka · 2024-10-18T07:36:07Z

prometheus/desc.go

@@ -57,6 +57,9 @@ type Desc struct {
 	// must be unique among all registered descriptors and can therefore be
 	// used as an identifier of the descriptor.
 	id uint64
+	// escapedID is similar to id, but with the metric and label names escaped
+	// with underscores.
+	escapedID uint64


Why not use id for escaped ID?

bwplotka · 2024-10-18T07:36:34Z

prometheus/desc.go

 	}
 	d.id = xxh.Sum64()
+	d.escapedID = escapedXXH.Sum64()


Again, why not use id as escaped ID always?

bwplotka

Thanks for this!

I think I would love to understand this change more before committing here. Asked some likely stupid questions on the issue #1638 (comment)

We never did those kind of checks on HTTP handler only, only on registration layer, I wonder if it's worth to keep that. Having your app working fine and suddenly /metrics are erroring only because 2 series collide is bit unexpected (especially by default) and wasteful, scrape could just ignore one problematic metric.

If anything we could start with a special Gatherer that checks that, without changes to descriptor ever, and see who would actually use it (who would like to opt-in for extra safety here). I would also be happy to consider adding this logic on-registry check/error when escaped metric name is colliding OR only check escaped ones (is there a case when escaped don't collide, but non-escaped collide?)

Is it worth to double check those first @ArthurSens ?

ywwg · 2024-10-18T14:36:41Z

We are going to put this on hold and have a meeting to discuss the various approaches and their pros and cons

ywwg · 2024-11-14T20:58:02Z

https://docs.google.com/document/d/1KS7RuhHZZbxPLoNmRwxCG1csPGPcO5FwDnmgTwv9x9M/edit?tab=t.0

@bwplotka @ArthurSens @beorn7

ywwg changed the title ~~WIP / Sacrificial: By default, return an error if metrics collide when escaped to underscores~~ WIP / strawman: By default, return an error if metrics collide when escaped to underscores Oct 8, 2024

ywwg commented Oct 9, 2024

View reviewed changes

ywwg requested a review from ArthurSens October 10, 2024 13:28

ArthurSens reviewed Oct 10, 2024

View reviewed changes

ywwg changed the title ~~WIP / strawman: By default, return an error if metrics collide when escaped to underscores~~ By default, return an error if metrics collide when escaped to underscores Oct 11, 2024

ywwg force-pushed the owilliams/collide branch from b08de89 to 8e9ef8c Compare October 11, 2024 14:41

ywwg marked this pull request as ready for review October 11, 2024 14:41

ywwg force-pushed the owilliams/collide branch from 97cad93 to 118a3c1 Compare October 11, 2024 14:55

ywwg commented Oct 15, 2024

View reviewed changes

ArthurSens reviewed Oct 16, 2024

View reviewed changes

prometheus/registry_test.go Outdated Show resolved Hide resolved

prometheus/registry_test.go Outdated Show resolved Hide resolved

ywwg commented Oct 16, 2024

View reviewed changes

ywwg changed the title ~~By default, return an error if metrics collide when escaped to underscores~~ Return an http error during scraping if metrics collide when escaped to underscores Oct 16, 2024

ywwg force-pushed the owilliams/collide branch 2 times, most recently from aad8c01 to 7f3763c Compare October 16, 2024 15:51

ywwg mentioned this pull request Oct 16, 2024

Epic: Implement UTF-8 character support for label and metric names prometheus/prometheus#13095

Open

42 tasks

ArthurSens approved these changes Oct 16, 2024

View reviewed changes

Return an http error during scraping if metrics collide when escaped …

42825b6

…to underscores Signed-off-by: Owen Williams <owen.williams@grafana.com>

ywwg force-pushed the owilliams/collide branch from 7f3763c to 42825b6 Compare October 17, 2024 15:20

move func to gatherer

6514f6e

Signed-off-by: Owen Williams <owen.williams@grafana.com>

ywwg commented Oct 17, 2024

View reviewed changes

bwplotka reviewed Oct 18, 2024

View reviewed changes

prometheus/desc.go

}

d.id = xxh.Sum64()

d.escapedID = escapedXXH.Sum64()

Copy link

Member

bwplotka Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, why not use id as escaped ID always?

bwplotka mentioned this pull request Oct 18, 2024

Optionally return error if two metrics would collide under escaping schemes #1638

Open

bwplotka requested changes Oct 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return an http error during scraping if metrics collide when escaped to underscores #1641

Return an http error during scraping if metrics collide when escaped to underscores #1641

ywwg commented Oct 8, 2024 •

edited

Loading

ywwg commented Oct 9, 2024

ywwg commented Oct 9, 2024

ywwg commented Oct 9, 2024

ywwg Oct 9, 2024

ywwg Oct 9, 2024

ywwg Oct 9, 2024

ywwg Oct 9, 2024

ywwg commented Oct 9, 2024

ArthurSens Oct 10, 2024

ArthurSens Oct 10, 2024 •

edited

Loading

ywwg Oct 10, 2024

ywwg Oct 10, 2024

ArthurSens Oct 10, 2024

ArthurSens Oct 10, 2024

ywwg Oct 10, 2024

ywwg Oct 15, 2024

ArthurSens Oct 16, 2024

ArthurSens left a comment

ywwg commented Oct 16, 2024

ywwg commented Oct 16, 2024

ywwg Oct 16, 2024

ArthurSens left a comment

ArthurSens Oct 16, 2024

ywwg Oct 17, 2024

ywwg Oct 17, 2024

ywwg Oct 17, 2024

ywwg Oct 17, 2024

ywwg Oct 17, 2024

bwplotka Oct 18, 2024

bwplotka Oct 18, 2024

bwplotka left a comment

ywwg commented Oct 18, 2024

ywwg commented Nov 14, 2024

Return an http error during scraping if metrics collide when escaped to underscores #1641

Are you sure you want to change the base?

Return an http error during scraping if metrics collide when escaped to underscores #1641

Conversation

ywwg commented Oct 8, 2024 • edited Loading

ywwg commented Oct 9, 2024

ywwg commented Oct 9, 2024

ywwg commented Oct 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywwg commented Oct 9, 2024

Choose a reason for hiding this comment

ArthurSens Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArthurSens left a comment

Choose a reason for hiding this comment

ywwg commented Oct 16, 2024

ywwg commented Oct 16, 2024

Choose a reason for hiding this comment

ArthurSens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

ywwg commented Oct 18, 2024

ywwg commented Nov 14, 2024

ywwg commented Oct 8, 2024 •

edited

Loading

ArthurSens Oct 10, 2024 •

edited

Loading