Skip to content

Commit 17c6214

Browse files
committed
Added mtypes CLI for generating realistic avalanche metric type distributions.
Initially added in bwplotka/prombenchy#12, but it might belong here more. Signed-off-by: bwplotka <bwplotka@gmail.com>
1 parent 5bc0599 commit 17c6214

File tree

10 files changed

+2303
-21
lines changed

10 files changed

+2303
-21
lines changed

.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
1-
avalanche
1+
./avalanche
2+
.build/
3+
.idea/

Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ LABEL maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com
66
ARG ARCH="amd64"
77
ARG OS="linux"
88
COPY .build/${OS}-${ARCH}/avalanche /bin/avalanche
9+
COPY .build/${OS}-${ARCH}/mtypes /bin/mtypes
910

1011
EXPOSE 9101
1112
USER nobody

README.md

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,29 +9,41 @@ This allows load testing services that can scrape (e.g. Prometheus, OpenTelemetr
99

1010
Metric names and unique series change over time to simulate series churn.
1111

12-
Checkout the [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).
12+
Checkout the (old-ish) [blog post](https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c).
1313

14-
## configuration flags
14+
## Installing
15+
16+
### Locally
1517

1618
```bash
17-
avalanche --help
19+
go install github.com/prometheus-community/avalanche/cmd/avalanche@latest
20+
${GOPATH}/bin/avalanche --help
1821
```
1922

20-
## run Docker image
23+
### Docker
2124

2225
```bash
23-
docker run quay.io/prometheuscommunity/avalanche:main --help
26+
docker run quay.io/prometheuscommunity/avalanche:latest --help
2427
```
2528

26-
## Endpoints
29+
NOTE: We recommend using pinned image to a certain version (see all tags [here](https://quay.io/repository/prometheuscommunity/avalanche?tab=tags&tag=latest))
30+
31+
## Using
32+
33+
See [example](example/kubernetes-deployment.yaml) k8s manifest for deploying avalanche as an always running scrape target.
34+
35+
### Configuration
36+
37+
See `--help` for all flags and their documentation.
38+
39+
Notably, from 0.6.0 version, `avalanche` allows specifying various counts per various metric types.
40+
41+
You can choose you own distribution, but usually it makes more sense to mimic realistic distribution used by your example targets. Feel free to use a [handy `mtypes` Go CLI](./cmd/mtypes) to gather type distributions from a target and generate avalanche flags from it.
42+
43+
On top of scrape target functionality, avalanche is capable of Remote Write client load simulation, following the same, configured metric distribution via `--remote*` flags.
44+
45+
### Endpoints
2746

2847
Two endpoints are available :
2948
* `/metrics` - metrics endpoint
3049
* `/health` - healthcheck endpoint
31-
32-
## build and run go binary
33-
34-
```bash
35-
go install github.com/prometheus-community/avalanche/cmd@latest
36-
go/bin/cmd --help
37-
```
File renamed without changes.

cmd/mtypes/README.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# mtypes
2+
3+
Go CLI gathering statistics around the distribution of types, average number of buckets (and more) across your Prometheus metrics/series.
4+
5+
## Usage
6+
7+
The main usage allows to take resource (from stdin, file or HTTP /metrics endpoint) and calculate type statistics e.g.:
8+
9+
```bash
10+
go install github.com/prometheus-community/avalanche/cmd/mtypes@latest # or locally: alias mtypes="go run ./cmd/mtypes"
11+
$ mtypes -resource=http://localhost:9090/metrics
12+
$ mtypes -resource=./cmd/mtypes/exampleprometheustarget.txt
13+
$ cat ./cmd/mtypes/exampleprometheustarget.txt | mtypes
14+
```
15+
16+
```bash
17+
Metric Type Metric Families Series (adjusted) Series (adjusted) % Average Buckets/Objectives
18+
GAUGE 76 93 (93) 31.958763 (16.909091) -
19+
COUNTER 96 157 (157) 53.951890 (28.545455) -
20+
HISTOGRAM 8 14 (186) 4.810997 (33.818182) 11.285714
21+
SUMMARY 15 27 (114) 9.278351 (20.727273) 2.222222
22+
--- --- --- --- ---
23+
* 195 291 (550) 100.000000 (100.000000) -
24+
```
25+
26+
> NOTE: "Adjusted" series, means actual number of individual series stored in Prometheus. Classic histograms and summaries are stored as a set of counters. This is relevant as the cost of indexing new series is higher than storing complex values (this is why we slowly move to native histograms).
27+
28+
Additionally, you can pass `--avalanche-flags-for-adjusted-series=10000` to print Avalanche v0.6.0+ flags to configure, for avalanche to generate metric target with the given amount of adjusted series, while maintaining a similar distribution e.g.
29+
30+
```bash
31+
cat ./cmd/mtypes/exampleprometheustarget.txt | mtypes --avalanche-flags-for-adjusted-series=1000
32+
Metric Type Metric Families Series (adjusted) Series (adjusted) % Average Buckets/Objectives
33+
GAUGE 76 93 (93) 31.958763 (16.909091) -
34+
COUNTER 96 157 (157) 53.951890 (28.545455) -
35+
HISTOGRAM 8 14 (186) 4.810997 (33.818182) 11.285714
36+
SUMMARY 15 27 (114) 9.278351 (20.727273) 2.222222
37+
--- --- --- --- ---
38+
* 195 291 (550) 100.000000 (100.000000) -
39+
40+
Avalanche flags for the similar distribution to get to the adjusted series goal of: 1000
41+
--gauge-metric-count=16
42+
--counter-metric-count=28
43+
--histogram-metric-count=2
44+
--histogram-metric-bucket-count=10
45+
--native-histogram-metric-count=0
46+
--summary-metric-count=5
47+
--summary-metric-objective-count=2
48+
--series-count=10
49+
--value-interval=300 # Changes values every 5m.
50+
--series-interval=3600 # 1h series churn.
51+
--metric-interval=0
52+
This should give the total adjusted series to: 900
53+
```

0 commit comments

Comments
 (0)