Sync upstream up to `5fdec3140188808bb9c2bf48e94b8533dbf75ac5` #818

julienduchesne · 2025-01-08T20:49:32Z

I will integrate this version into Mimir

Diff: prometheus/prometheus@9823a93...5fdec31

No conflicts!

PRs possibly relevant to Mimir:

TSDB: Optimization: Merge postings using concrete type prometheus/prometheus#14518
feat(nhcb): support custom buckets in native histograms in the WAL/WBL prometheus/prometheus#15467
otlp/translator: Use separate function for metric names with UTF8 characters prometheus/prometheus#15664
template: Use cases.Title instead of deprecated strings.Title prometheus/prometheus#15721
OTLP receiver: Optimize by initializing regexps at program start prometheus/prometheus#15733
perf(nhcbparse): unroll recursion prometheus/prometheus#15776

Irrelevant PRs:

# Conflicts: # tsdb/docs/format/wal.md

The segment size was too low for the additional NHCB data, thus it created more segments then expected. This meant that less were in the lower numbered segments, which meant more was kept. FAIL: TestCheckpoint (4.05s) FAIL: TestCheckpoint/compress=none (0.22s) checkpoint_test.go:361: Error Trace: /home/krajo/go/github.com/prometheus/prometheus/tsdb/wlog/checkpoint_test.go:361 Error: "0.8586956521739131" is not less than "0.8" Test: TestCheckpoint/compress=none Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Remove the 2 minute timeout as the default is 2 hours and wouldn't interfere. With the test. Otherwise the extra samples combined with race detection can push the test over 2 minutes and make it fail. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

…erge speed We need to create more postings entries so the merger has some work to do. Not material for the regexp ones as they match so few series. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

As long as we run small and big sizes, we don't need all the sizes inbetween. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Now we can call it with more specific types which is more efficient than making everything go through the `Postings` interface. Benchmark the concrete type. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

This enables it to take advantage of a more compact data structure since all postings are known to be `*ListPostings`. Remove the `Get` member which was not used for anything else, and fix up tests. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

It was bumped during 3.0 with the adoption of log/slog and other dep updates. ``` ~/go/src/github.com/prometheus/prometheus (main [ ]) -> grep '^go' go.mod go 1.22.0 ``` Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

fix(main.go): avoid closing the query engine until it is guaranteed to no longer be in use.

…733) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

TSDB: Optimization: Merge postings using concrete type

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

# Conflicts: # tsdb/tsdbutil/histogram.go

prometheus/prometheus#15467 (comment) Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

feat(nhcb): support custom buckets in native histograms in the WAL/WBL

…l (#15786) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com>

…racters (#15664) BuildCompliantName was renamed to BuildCompliantMetricName, and it no longer takes UTF8 support into consideration. It focuses on building a metric name that follows Prometheus conventions. A new function, BuildMetricName, was added to optionally add unit and type suffixes to OTLP metric names without translating any characters to underscores(_).

CLAassistant · 2025-01-08T20:49:44Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 7 committers have signed the CLA.

✅ krajorama
✅ bboreham
✅ aknuds1
✅ carrieedwards
❌ bwplotka
❌ tjhop
❌ heliapb
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…chesne/upstream-pt-3

Version synced here: grafana/mimir-prometheus#818

krajorama · 2025-01-09T10:12:06Z

The NHCB related changes are fine, we cannot ingest NHCB anyway , depends on grafana/mimir#9072 in Mimir.

I'm less sure about the UTF-8 stuff. How will it affect receiving OTLP from customers ? Are there new ways we reject some OTLP input? @ArthurSens

bboreham · 2025-01-09T10:46:14Z

I thought we had not turned on UTF8 handling in production.
Certainly I would have expected to notice the performance impact from recompiling regexps all the time.

ArthurSens · 2025-01-09T12:35:21Z

Yeah, my understanding is that Grafana Cloud still doesn't support keeping UTF-8 characters yet, so it should be fine

bboreham

no flaws detected

krajorama

approved, I mostly looked at NHCB

* Bump to latest mimir-prometheus Version synced here: grafana/mimir-prometheus#818 * Pull MQE functions tests Fix everything related to prometheus/prometheus#15686 We don't support `double_exponential_smoothing` and `predict_linear`, but `deriv` needs to emit a new annotation * Add CHANGELOG * Add back deriv to test gauntlet * Update to `mimir-prometheus`' main branch * Apply suggestions from code review Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> --------- Co-authored-by: Joshua Hesketh <joshua.hesketh@grafana.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>

carrieedwards and others added 30 commits December 5, 2024 09:21

Handle custom buckets in WAL and WBL

aa144b7

Use histogram records for custom value handling

6d413fa

Remove references to custom values record

cfcd515

Attempt for record type

37df50a

Add separate handling for histograms and custom bucket histograms

454f6d3

Rename old histogram record type, use old names for new records

6684344

Update WAL doc to include native histogram encodings

f8a3976

Fix comment and histogram record string

6b44c14

Extend tsdb agent tests with custom bucket histograms

45944c1

Use new record type only for NHCB

a046417

Fix test

1933ccc

Merge branch 'main' into cedwards/nhcb-wal-wbl

efdd088

# Conflicts: # tsdb/docs/format/wal.md

fix(lint): linter errors

8f572fe

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

fix(test): if we are dereferencing a slice we should check its len

07276ae

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Benchmark encoding classic and nhcb

d64d1c4

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Fix lint errors

fdb1516

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Fix unused import

cf36792

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Fix lint for real

df88de5

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

TSDB: extend BenchmarkMemPostings_PostingsForLabelMatching to check m…

e630ffd

…erge speed We need to create more postings entries so the merger has some work to do. Not material for the regexp ones as they match so few series. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

TSDB BenchmarkMerge: run fewer sizes

1b22242

As long as we run small and big sizes, we don't need all the sizes inbetween. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

TSDB: Make mergedPostings generic

0a8779f

Now we can call it with more specific types which is more efficient than making everything go through the `Postings` interface. Benchmark the concrete type. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Update benchmark test and comment

1508149

docs: update required go version in readme to 1.22 (#15447)

4cbd9ff

It was bumped during 3.0 with the adoption of log/slog and other dep updates. ``` ~/go/src/github.com/prometheus/prometheus (main [ ]) -> grep '^go' go.mod go 1.22.0 ``` Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

Merge pull request #15723 from machine424/eng-clos

2dfb4fd

fix(main.go): avoid closing the query engine until it is guaranteed to no longer be in use.

OTLP receiver: Optimize by initializing regexps at program start (#15…

f37b5ad

…733) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Merge pull request #14518 from bboreham/faster-listpostings-merge

096e2aa

TSDB: Optimization: Merge postings using concrete type

record_test.go: avoid captures, simply return test refs

a7ccc8e

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

krajorama and others added 7 commits January 2, 2025 12:50

Merge branch 'main' into cedwards/nhcb-wal-wbl

1e420ef

# Conflicts: # tsdb/tsdbutil/histogram.go

perf(nhcbparse): unroll recursion (#15776)

cfcb00a

prometheus/prometheus#15467 (comment) Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

Merge 3.1 into main (#15775)

a6947a0

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

template: Use cases.Title instead of deprecated strings.Title (#15721)

4f67a38

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Merge pull request #15467 from prometheus/cedwards/nhcb-wal-wbl

a441ad7

feat(nhcb): support custom buckets in native histograms in the WAL/WBL

[Docs] Note that scrape_timeout cannot be greater than scrape_interva…

5609419

…l (#15786) Signed-off-by: Hélia Barroso <helia_barroso@hotmail.com>

Merge commit '5fdec3140188808bb9c2bf48e94b8533dbf75ac5' into juliendu…

d1ab530

…chesne/upstream-pt-3

julienduchesne force-pushed the julienduchesne/upstream-pt-3 branch from 62b9e7a to d1ab530 Compare January 8, 2025 20:50

julienduchesne requested review from bboreham, carrieedwards, krajorama, aknuds1 and ArthurSens January 8, 2025 21:01

julienduchesne marked this pull request as ready for review January 8, 2025 21:02

julienduchesne added a commit to grafana/mimir that referenced this pull request Jan 8, 2025

Bump to latest mimir-prometheus

3328b33

Version synced here: grafana/mimir-prometheus#818

julienduchesne mentioned this pull request Jan 8, 2025

Bump to latest mimir-prometheus grafana/mimir#10383

Merged

4 tasks

bboreham approved these changes Jan 9, 2025

View reviewed changes

krajorama approved these changes Jan 9, 2025

View reviewed changes

julienduchesne merged commit 114aaaa into main Jan 9, 2025
7 of 8 checks passed

julienduchesne deleted the julienduchesne/upstream-pt-3 branch January 9, 2025 13:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync upstream up to `5fdec3140188808bb9c2bf48e94b8533dbf75ac5` #818

Sync upstream up to `5fdec3140188808bb9c2bf48e94b8533dbf75ac5` #818

julienduchesne commented Jan 8, 2025 •

edited

Loading

CLAassistant commented Jan 8, 2025 •

edited

Loading

krajorama commented Jan 9, 2025

bboreham commented Jan 9, 2025

ArthurSens commented Jan 9, 2025

bboreham left a comment

krajorama left a comment

Sync upstream up to 5fdec3140188808bb9c2bf48e94b8533dbf75ac5 #818

Sync upstream up to 5fdec3140188808bb9c2bf48e94b8533dbf75ac5 #818

Conversation

julienduchesne commented Jan 8, 2025 • edited Loading

CLAassistant commented Jan 8, 2025 • edited Loading

krajorama commented Jan 9, 2025

bboreham commented Jan 9, 2025

ArthurSens commented Jan 9, 2025

bboreham left a comment

Choose a reason for hiding this comment

krajorama left a comment

Choose a reason for hiding this comment

Sync upstream up to `5fdec3140188808bb9c2bf48e94b8533dbf75ac5` #818

Sync upstream up to `5fdec3140188808bb9c2bf48e94b8533dbf75ac5` #818

julienduchesne commented Jan 8, 2025 •

edited

Loading

CLAassistant commented Jan 8, 2025 •

edited

Loading