Skip to content

Latest commit

 

History

History
171 lines (147 loc) · 6.42 KB

benchtool.md

File metadata and controls

171 lines (147 loc) · 6.42 KB
Raw

Benchtool

The benchtool is a small load testing utility to generate load for Prometheus remote-write and query API endpoints. It uses a statically configured YAML workload file to describe the series and queries used for the generate load.

warning: the specifications outlined here are subject to change. It is likely the specs in this file will require some adjustment to accomodate expanded functionality and this should not be consided stable.

Workload file

The benchtool is designed to generate Prometheus remote-write and PromQL API query requests to a configured endpoint. The characteristics of the data used for these requests is driven by a YAML workload configuration file. The workload file can be configured as a follows:

replicas: "<int>"
queries:
  - expr_template: "<string>"
    interval: "<duration>"
    num_queries: "<int>"
    series_type: "<enum series-type>"
    regex: "<bool>"
series:
  - name: "<string>"
    type: "<enum series-type>"
    static_labels:
      "<string>": "<string>"
    labels:
      - name: "<string>"
        unique_values: "<int>"
        value_prefix: "<string>"
write_options:
  batch_size: "<int>"
  interval: "<duration>"

Series

The series configuration section of the workload is used to configure the metrics generated by the workload. It is a list of series objects that allow the workload author to declare name and characteristics of each series the workload will generate.

  • name is the metric name of the series that will generated. The name will be used as the reserved __name__ label for each series generated using this series config.
  • type references the type of series that will be generated and the characteristics of it's underlying data. See the series type section for more information.
  • static_labels is a map of string values to string keys. These will be added as labels to each series generated from this series description.
  • labels is field that allows for the input of a list of label descriptions. The series generated from this series description will be based on the result of all possible combinations of the provided label descriptions.
    • name refers to the label name of the generated label.
    • unique_values referes to the number of values to generate for this label.
    • value_prefix refers to the prefix that will used for each generated label value. The resultant label value will be the prefix appended with -<integer>, where the integer is a value between 0 and the configured unique_values.

Series types

Series types are used in both the series and queries section of the workload. The denote the character and properties of the data of the generated or queried series. The following series types are supported.

  • gauge-zero: correlates to a gauge series that will contantly be zero. This is a commonly seen pattern in Prometheus info metrics.
  • gauge-random: correlates to a gauge series that constant changes to a random value. The values is chosen using rand.Float64.
  • counter-one correlates to a counter that increases by 1 at every interval.
  • counter-random correlates to a counter that increases by a random amount every interval. The random amount is not constant and is currently chosen using rand.Int().

Global options

  • replicas: Replicas is meant to be a stand in for the host label. For each value between 0 and the configured replica value, a bench_replica label will be added and appended to each generated series. This label will also be used for queries configured to use regular expressions.

Queries

The query workload is made up of a list of configured queries. Queries have a variety of options that can be configured to get the desired behavior.

  • series_type: Series type to use for this query. A series name generated from the series section of the Cortex config will be chosen and used for each generated query.
  • expr_template is used to configure the query expression that will be run. It uses a modified Go text template to allow for the name of a series and a matcher to be injected into the query. The name will be sourced from the series name and the matcher will be a match on the bench_replicas label.
  • regex is a boolean value that, if enabled, will cause the injected bench_replica matcher on the query to be a regex match on a single replica instead of an exact matcher.
  • interval: Interval at which each instantiation of this query description will run. A random jitter no greater than half the configured interval is applied before the first run of each query to ensure that queries are not all scheduled to run simultaneously.
  • num_queries: Number of replicas to run of this configured query. Each instance of the generated query will select a random series with the same series type. Each query will also apply a unique random jitter on the interval before it's first run.

Write options

  • batch_size determines the number of samples send in each remote-write request.
  • interval is the duration period between when each batch of remote-write requests are generated and queued to be sent.

Example workload file

---
queries:
  - expr_template: sum(<<.Name>>{<<.Matchers>>})
    interval: 1m
    num_queries: 3
    series_type: gauge-random
  - expr_template: sum(<<.Name>>{<<.Matchers>>})
    interval: 1m
    num_queries: 5
    series_type: gauge-zero
    time_range: 2h
replicas: 5
series:
  - labels:
      - name: label_01
        unique_values: 5
        value_prefix: label_value_01
      - name: label_02
        unique_values: 20
        value_prefix: label_value_02
    name: metric_gauge_random_01
    static_labels:
        static: "true"
    type: gauge-random
  - labels:
      - name: label_01
        unique_values: 5
        value_prefix: label_value_01
      - name: label_02
        unique_values: 20
        value_prefix: label_value_02
    name: metric_gauge_zero_01
    static_labels:
        static: "true"
    type: gauge-zero
write_options:
    batch_size: 1000
    interval: 15s

Consistency

To ensure results are consistent between runs, the seed of the random number generator used by the workload is determined by the configured -bench.id flag value of the benchtool instance.

This will ensure two benchtool processes run with the same id and workload config file will result in the same behavior between runs.