Skip to content

Commit e75bb9f

Browse files
committed
Added mtypes CLI for generating realistic avalanche metric type distributions.
Initially added in bwplotka/prombenchy#12, but it might belong here more. Signed-off-by: bwplotka <bwplotka@gmail.com>
1 parent 5bc0599 commit e75bb9f

File tree

9 files changed

+1360
-21
lines changed

9 files changed

+1360
-21
lines changed

.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
avalanche
1+
./avalanche
2+
.build/
3+
.idea/

Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ LABEL maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com
66
ARG ARCH="amd64"
77
ARG OS="linux"
88
COPY .build/${OS}-${ARCH}/avalanche /bin/avalanche
9+
COPY .build/${OS}-${ARCH}/mtypes /bin/mtypes
910

1011
EXPOSE 9101
1112
USER nobody

README.md

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,29 +9,41 @@ This allows load testing services that can scrape (e.g. Prometheus, OpenTelemetr
99

1010
Metric names and unique series change over time to simulate series churn.
1111

12-
Checkout the [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).
12+
Checkout the (old-ish) [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).
1313

14-
## configuration flags
14+
## Installing
15+
16+
### Locally
1517

1618
```bash
17-
avalanche --help
19+
go install github.com/prometheus-community/avalanche/cmd/avalanche@latest
20+
${GOPATH}/bin/avalanche --help
1821
```
1922

20-
## run Docker image
23+
### Docker
2124

2225
```bash
23-
docker run quay.io/prometheuscommunity/avalanche:main --help
26+
docker run quay.io/prometheuscommunity/avalanche:latest --help
2427
```
2528

26-
## Endpoints
29+
NOTE: We recommend using pinned image to a certain version (see all tags [here](https://quay.io/repository/prometheuscommunity/avalanche?tab=tags&tag=latest))
30+
31+
## Using
32+
33+
See [example](example/kubernetes-deployment.yaml) k8s manifest for deploying avalanche as an always running scrape target.
34+
35+
### Configuration
36+
37+
See `--help` for all flags and their documentation.
38+
39+
Notably, from 0.6.0 version, `avalanche` allows specifying various counts per various metric types.
40+
41+
You can choose you own distribution, but usually it makes more sense to mimic realistic distribution used by your example targets. Feel free to use a [handy `mtypes` Go CLI](./cmd/mtypes) to gather type distributions from a target and generate avalanche flags from it.
42+
43+
On top of scrape target functionality, avalanche is capable of Remote Write client load simulation, following the same, configured metric distribution via `--remote*` flags.
44+
45+
### Endpoints
2746

2847
Two endpoints are available :
2948
* `/metrics` - metrics endpoint
3049
* `/health` - healthcheck endpoint
31-
32-
## build and run go binary
33-
34-
```bash
35-
go install github.com/prometheus-community/avalanche/cmd@latest
36-
go/bin/cmd --help
37-
```
File renamed without changes.

cmd/mtypes/README.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# mtypes
2+
3+
Go CLI gathering statistics around the distribution of types, average number of buckets (and more) across your Prometheus metrics/series.
4+
5+
## Usage
6+
7+
The main usage allows to take resource (from stdin, file or HTTP /metrics endpoint) and calculate type statistics e.g.:
8+
9+
```bash
10+
$ mtypes -resource=http://localhost:9090/metrics
11+
$ mtypes -resource=./metrics.prometheus.txt
12+
$ cat ./metrics.prometheus.txt | mtypes
13+
```
14+
15+
```bash
16+
Metric Type Metric Families Series Series % Series % (complex type adjusted) Average Buckets/Objectives
17+
GAUGE 77 94 30.618893 15.112540 -
18+
COUNTER 104 167 54.397394 26.848875 -
19+
HISTOGRAM 11 19 6.188925 39.710611 11.000000
20+
SUMMARY 15 27 8.794788 18.327974 2.222222
21+
```
22+
23+
> NOTE: "Adjusted" series, means actual number of individual series stored in Prometheus. Classic histograms and summaries are stored as a set of counters. This is relevant as the cost of indexing new series is higher than storing complex values (this is why we slowly move to native histograms).
24+
25+
Additionally, you can pass `--avalanche-flags-for-adjusted-series=10000` to print Avalanche v0.6.0+ flags to configure, for avalanche to generate metric target with the given amount of adjusted series, while maintaining a similar distribution e.g.
26+
27+
```bash
28+
cat ../../manifests/load/exampleprometheustarget.txt | go run main.go --avalanche-flags-for-adjusted-series=10000
29+
Metric Type Metric Families Series (adjusted) Series (adjusted) % Average Buckets/Objectives
30+
GAUGE 77 94 (94) 30.921053 (15.719064) -
31+
COUNTER 104 166 (166) 54.605263 (27.759197) -
32+
HISTOGRAM 11 17 (224) 5.592105 (37.458194) 11.176471
33+
SUMMARY 15 27 (114) 8.881579 (19.063545) 2.222222
34+
--- --- --- --- ---
35+
* 207 304 (598) 100.000000 (100.000000) -
36+
37+
Avalanche flags for the similar distribution to get to the adjusted series goal of: 10000
38+
--gauge-metric-count=157
39+
--counter-metric-count=277
40+
--histogram-metric-count=28
41+
--histogram-metric-bucket-count=10
42+
--native-histogram-metric-count=0
43+
--summary-metric-count=47
44+
--summary-metric-objective-count=2
45+
--series-count=10
46+
--value-interval=300 # Changes values every 5m.
47+
--series-interval=3600 # 1h series churn.
48+
--metric-interval=0
49+
This should give the total adjusted series to: 9860
50+
```

cmd/mtypes/main.go

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
// Package main implements mtypes CLI, see README for details.
2+
package main
3+
4+
import (
5+
"errors"
6+
"flag"
7+
"fmt"
8+
"io"
9+
"log"
10+
"net/http"
11+
"net/url"
12+
"os"
13+
"strings"
14+
"text/tabwriter"
15+
16+
dto "github.com/prometheus/client_model/go"
17+
"github.com/prometheus/common/expfmt"
18+
)
19+
20+
type stats struct {
21+
families, series, buckets, objectives int
22+
23+
// adjustedSeries represents series that would result in "series" in Prometheus data model
24+
// (includes _bucket, _count, _sum, _quantile).
25+
adjustedSeries int
26+
}
27+
28+
var metricType_NATIVE_HISTOGRAM dto.MetricType = 999
29+
30+
func main() {
31+
resource := flag.String("resource", "", "Path or URL to the resource (file, <url>/metrics) containing Prometheus metric format.")
32+
avalancheFlagsForTotal := flag.Int("avalanche-flags-for-adjusted-series", 0, "If more than zero, it additionally prints flags for the avalanche 0.6.0 command line to generate metrics for the similar type distribution; to get the total number of adjusted series to the given value.")
33+
flag.Parse()
34+
35+
var input io.Reader = os.Stdin
36+
if *resource != "" {
37+
switch {
38+
case strings.HasPrefix(*resource, "https://"), strings.HasPrefix(*resource, "http://"):
39+
if _, err := url.Parse(*resource); err != nil {
40+
log.Fatalf("error parsing HTTP URL to the resource %v; got %v", *resource, err)
41+
}
42+
resp, err := http.Get(*resource)
43+
if err != nil {
44+
log.Fatalf("http get against %v failed", err)
45+
}
46+
defer resp.Body.Close()
47+
input = resp.Body
48+
default:
49+
// Open the input file.
50+
file, err := os.Open(*resource)
51+
if err != nil {
52+
log.Fatalf("Error opening file: %v", err) //nolint:gocritic
53+
}
54+
defer file.Close()
55+
input = file
56+
}
57+
}
58+
statistics, err := calculateTargetStatistics(input)
59+
if err != nil {
60+
log.Fatal(err)
61+
}
62+
var total stats
63+
for _, s := range statistics {
64+
total.families += s.families
65+
total.series += s.series
66+
total.adjustedSeries += s.adjustedSeries
67+
}
68+
69+
writeStatistics(os.Stdout, total, statistics)
70+
71+
if *avalancheFlagsForTotal > 0 {
72+
// adjustedGoal is tracking the # of adjusted series we want to generate with avalanche.
73+
adjustedGoal := float64(*avalancheFlagsForTotal)
74+
fmt.Println()
75+
fmt.Println("Avalanche flags for the similar distribution to get to the adjusted series goal of:", adjustedGoal)
76+
77+
adjustedGoal /= 10.0 // Assuming --series-count=10
78+
// adjustedSum is tracking the total sum of series so far (at the end hopefully adjustedSum ~= adjustedGoal)
79+
adjustedSum := 0
80+
for _, mtype := range allTypes {
81+
s := statistics[mtype]
82+
83+
// adjustedSeriesRatio is tracking the ratio of this type in the input file.
84+
// We try to get similar ratio, but with different absolute counts, given the total sum of series we are aiming for.
85+
adjustedSeriesRatio := float64(s.adjustedSeries) / float64(total.adjustedSeries)
86+
87+
// adjustedSeriesForType is tracking (per metric type), how many unique series of that
88+
// metric type avalanche needs to create according to the ratio we got from our input.
89+
adjustedSeriesForType := int(adjustedGoal * adjustedSeriesRatio)
90+
91+
switch mtype {
92+
case dto.MetricType_GAUGE:
93+
fmt.Printf("--gauge-metric-count=%v\n", adjustedSeriesForType)
94+
adjustedSum += adjustedSeriesForType
95+
case dto.MetricType_COUNTER:
96+
fmt.Printf("--counter-metric-count=%v\n", adjustedSeriesForType)
97+
adjustedSum += adjustedSeriesForType
98+
case dto.MetricType_HISTOGRAM:
99+
avgBkts := s.buckets / s.series
100+
adjustedSeriesForType /= 2 + avgBkts
101+
fmt.Printf("--histogram-metric-count=%v\n", adjustedSeriesForType)
102+
fmt.Printf("--histogram-metric-bucket-count=%v\n", avgBkts-1) // -1 is due to caveat of additional +Inf not added by avalanche.
103+
adjustedSum += adjustedSeriesForType * (2 + avgBkts)
104+
case metricType_NATIVE_HISTOGRAM:
105+
fmt.Printf("--native-histogram-metric-count=%v\n", adjustedSeriesForType)
106+
adjustedSum += adjustedSeriesForType
107+
case dto.MetricType_SUMMARY:
108+
avgObjs := s.objectives / s.series
109+
adjustedSeriesForType /= 2 + avgObjs
110+
fmt.Printf("--summary-metric-count=%v\n", adjustedSeriesForType)
111+
fmt.Printf("--summary-metric-objective-count=%v\n", avgObjs)
112+
adjustedSum += adjustedSeriesForType * (2 + avgObjs)
113+
default:
114+
if s.series > 0 {
115+
log.Fatalf("not supported %v metric in avalanche", mtype)
116+
}
117+
}
118+
}
119+
fmt.Printf("--series-count=10\n")
120+
fmt.Printf("--value-interval=300 # Changes values every 5m.\n")
121+
fmt.Printf("--series-interval=3600 # 1h series churn.\n")
122+
fmt.Printf("--metric-interval=0\n")
123+
124+
fmt.Println("This should give the total adjusted series to:", adjustedSum*10)
125+
}
126+
}
127+
128+
var allTypes = []dto.MetricType{dto.MetricType_GAUGE, dto.MetricType_COUNTER, dto.MetricType_HISTOGRAM, metricType_NATIVE_HISTOGRAM, dto.MetricType_GAUGE_HISTOGRAM, dto.MetricType_SUMMARY, dto.MetricType_UNTYPED}
129+
130+
func writeStatistics(writer io.Writer, total stats, statistics map[dto.MetricType]stats) {
131+
w := tabwriter.NewWriter(writer, 0, 0, 4, ' ', 0)
132+
fmt.Fprintln(w, "Metric Type\tMetric Families\tSeries (adjusted)\tSeries (adjusted) %\tAverage Buckets/Objectives")
133+
134+
for _, mtype := range allTypes {
135+
s, ok := statistics[mtype]
136+
if !ok {
137+
continue
138+
}
139+
140+
mtypeStr := mtype.String()
141+
if mtype == metricType_NATIVE_HISTOGRAM {
142+
mtypeStr = "HISTOGRAM (native)"
143+
}
144+
145+
seriesRatio := 100 * float64(s.series) / float64(total.series)
146+
adjustedSeriesRatio := 100 * float64(s.adjustedSeries) / float64(total.adjustedSeries)
147+
switch {
148+
case s.buckets > 0:
149+
fmt.Fprintf(w, "%s\t%d\t%d (%d)\t%f (%f)\t%f\n", mtypeStr, s.families, s.series, s.adjustedSeries, seriesRatio, adjustedSeriesRatio, float64(s.buckets)/float64(s.series))
150+
case s.objectives > 0:
151+
fmt.Fprintf(w, "%s\t%d\t%d (%d)\t%f (%f)\t%f\n", mtypeStr, s.families, s.series, s.adjustedSeries, seriesRatio, adjustedSeriesRatio, float64(s.objectives)/float64(s.series))
152+
default:
153+
fmt.Fprintf(w, "%s\t%d\t%d (%d)\t%f (%f)\t-\n", mtypeStr, s.families, s.series, s.adjustedSeries, seriesRatio, adjustedSeriesRatio)
154+
}
155+
}
156+
fmt.Fprintf(w, "---\t---\t---\t---\t---\n")
157+
fmt.Fprintf(w, "*\t%d\t%d (%d)\t%f (%f)\t-\n", total.families, total.series, total.adjustedSeries, 100.0, 100.0)
158+
_ = w.Flush()
159+
}
160+
161+
func calculateTargetStatistics(r io.Reader) (statistics map[dto.MetricType]stats, _ error) {
162+
// Parse the Prometheus Text format.
163+
parser := expfmt.NewDecoder(r, expfmt.NewFormat(expfmt.TypeProtoText))
164+
165+
statistics = map[dto.MetricType]stats{}
166+
nativeS := statistics[metricType_NATIVE_HISTOGRAM]
167+
for {
168+
var mf dto.MetricFamily
169+
if err := parser.Decode(&mf); err != nil {
170+
if errors.Is(err, io.EOF) {
171+
break
172+
}
173+
return nil, fmt.Errorf("parsing %w", err)
174+
}
175+
176+
s := statistics[mf.GetType()]
177+
178+
var mfAccounted, mfAccountedNative bool
179+
switch mf.GetType() {
180+
case dto.MetricType_GAUGE_HISTOGRAM, dto.MetricType_HISTOGRAM:
181+
for _, m := range mf.GetMetric() {
182+
if m.GetHistogram().GetSchema() == 0 {
183+
// classic one.
184+
s.series++
185+
s.buckets += len(m.GetHistogram().GetBucket())
186+
s.adjustedSeries += 2 + len(m.GetHistogram().GetBucket())
187+
188+
if !mfAccounted {
189+
s.families++
190+
mfAccounted = true
191+
}
192+
} else {
193+
// native one.
194+
nativeS.series++
195+
nativeS.buckets += len(m.GetHistogram().GetNegativeDelta())
196+
nativeS.buckets += len(m.GetHistogram().GetNegativeCount())
197+
nativeS.buckets += len(m.GetHistogram().GetPositiveDelta())
198+
nativeS.buckets += len(m.GetHistogram().GetPositiveCount())
199+
nativeS.adjustedSeries++
200+
201+
if !mfAccountedNative {
202+
nativeS.families++
203+
mfAccountedNative = true
204+
}
205+
}
206+
}
207+
case dto.MetricType_SUMMARY:
208+
s.series += len(mf.GetMetric())
209+
s.families++
210+
for _, m := range mf.GetMetric() {
211+
s.objectives += len(m.GetSummary().GetQuantile())
212+
s.adjustedSeries += 2 + len(m.GetSummary().GetQuantile())
213+
}
214+
default:
215+
s.series += len(mf.GetMetric())
216+
s.families++
217+
s.adjustedSeries += len(mf.GetMetric())
218+
}
219+
statistics[mf.GetType()] = s
220+
}
221+
if nativeS.series > 0 {
222+
statistics[metricType_NATIVE_HISTOGRAM] = nativeS
223+
}
224+
return statistics, nil
225+
}

0 commit comments

Comments
 (0)