2.9.0

This release contains 252 PRs from 46 authors, including new contributors Alex R, Alexander Soelberg Heidarsson, Alexander Weaver, Benjamin Lazarecki, Dhanu Saputra, Dominik Süß, Fiona Liao, Jonathan Halterman, Kristian Bremberg, MattiasSegerdahl, Salva Corts, Stephanie Closson, willychrisza. Thank you!

Grafana Mimir version 2.9.0 release notes

Grafana Labs is excited to announce version 2.9 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Reduced store-gateway memory utilization on fetching series from long-term storage For queries that include broad label matchers (e.g. datacenter="dc1"), Mimir 2.9 will fetch a reduced volume of index data, which leads to a significant reduction in memory allocations in the store-gateway.
• Reduced CPU utilisation for some shuffle sharding scenarios Mimir queriers will now use significantly less CPU in cases where shuffle sharding is enabled for tenants with a shard size that's large but lower than the total number of ingesters.
• Reduced object storage API calls in compactors and rulers Mimir 2.9 comes with optimizations that will reduce the amount of times compactors and rulers need to access rules stored in object storage.
  • This release adds experimental support for a ruler storage cache. This cache should reduce the number of "list objects" API calls issued to the object storage when there are 2+ ruler replicas running in a Mimir cluster. The cache can be configured by setting the -ruler-storage.cache.* CLI flags or their respective YAML config options.
  • We also introduced a new feature to trigger a synchronization of tenant's rule groups as soon as changes to the rule configuration are made via API. This synchronization is in addition of the periodic syncing done every -ruler.poll-interval, which has then been relaxed from every 1m to every 10m. The new behaviour is enabled globally by default but can be disabled with -ruler.sync-rules-on-changes-enabled=false or tuned at a per-tenant level.
• Experimental support for streaming chunks from ingester to querier This is expected to greatly reduce querier memory consumption when evaluating queries that select a large number of series, because chunks streamed from the querier can now be read into memory as needed.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.9 we have removed the following previously deprecated or experimental metrics:

• cortex_bucket_store_chunk_pool_requested_bytes_total
• cortex_bucket_store_chunk_pool_returned_bytes_total

The following configuration options are deprecated and will be removed in Grafana Mimir 2.11:

• The CLI flag -querier.query-ingesters-within. This configuration is moved to per-tenant overrides.
• The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
• The CLI flags -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes, -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes and -blocks-storage.bucket-store.max-chunk-pool-bytes.
• The CLI flags -querier.iterators and -query.batch-iterators.

The following configuration options that were deprecated in 2.7 are removed:

• The CLI flag -blocks-storage.bucket-store.chunks-cache.subrange-size. A fixed value of 16000 is now always used.
• The CLI flag -blocks-storage.bucket-store.consistency-delay.
• The CLI flag -compactor.consistency-delay.
• The CLI flag -ingester.ring.readiness-check-ring-health.

The following experimental options and features are now stable:

• The CLI flag -query-frontend.query-sharding-max-regexp-size-bytes.
• The CLI flag -query-scheduler.max-used-instances.
• The CLI flags -(alertmanager|blocks|ruler)-storage.storage-prefix.
• The CLI flag -compactor.first-level-compaction-wait-period.
• The CLI flags -usage-stats.enabled and -usage-stats.installation-mode.
• The CLI flag -query-frontend.query-sharding-target-series-per-shard.

The following configuration option defaults were changed:

• The default value for the CLI flag -query-frontend.query-sharding-max-regexp-size-bytes was changed from 0 to 4096. As a result, queries with regex matchers exceeding this limit will not be sharded by default.
• The default value for the CLI flag -compactor.partial-block-deletion-delay was changed from 0s to 1d. As a result, partial blocks resulting from a failed block upload or deletion will be cleaned up automatically.
• The default value for the CLI flag -ruler.poll-interval was changed from 1m to 10m.

Bug fixes

• Store-gateway: Detect collisions in the postings cache. PR 4770
• Store-gateway: Fix panic caused by cached LabelValues responses with more than 655360 values. PR 5021

Changelog

2.9.0

Grafana Mimir

• [CHANGE] Store-gateway: change expanded postings, postings, and label values index cache key format. These caches will be invalidated when rolling out the new Mimir version. #4770 #4978 #5037
• [CHANGE] Distributor: remove the "forwarding" feature as it isn't necessary anymore. #4876
• [CHANGE] Query-frontend: Change the default value of -query-frontend.query-sharding-max-regexp-size-bytes from 0 to 4096. #4932
• [CHANGE] Querier: -querier.query-ingesters-within has been moved from a global flag to a per-tenant override. #4287
• [CHANGE] Querier: Use -blocks-storage.tsdb.retention-period instead of -querier.query-ingesters-within for calculating the lookback period for shuffle sharded ingesters. Setting -querier.query-ingesters-within=0 no longer disables shuffle sharding on the read path. #4287
• [CHANGE] Block upload: /api/v1/upload/block/{block}/files endpoint now allows file uploads with no Content-Length. #4956
• [CHANGE] Store-gateway: deprecate configuration parameters for chunk pooling, they will be removed in Mimir 2.11. The following options are now also ignored: #4996
  • -blocks-storage.bucket-store.max-chunk-pool-bytes
  • -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
  • -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
• [CHANGE] Store-gateway: remove metrics cortex_bucket_store_chunk_pool_requested_bytes_total and cortex_bucket_store_chunk_pool_returned_bytes_total. #4996
• [CHANGE] Compactor: change default of -compactor.partial-block-deletion-delay to 1d. This will automatically clean up partial blocks that were a result of failed block upload or deletion. #5026
• [CHANGE] Compactor: the deprecated configuration parameter -compactor.consistency-delay has been removed. #5050
• [CHANGE] Store-gateway: the deprecated configuration parameter -blocks-storage.bucket-store.consistency-delay has been removed. #5050
• [CHANGE] The configuration parameter -blocks-storage.bucket-store.bucket-index.enabled has been deprecated and will be removed in Mimir 2.11. Mimir is running by default with the bucket index enabled since version 2.0, and starting from the version 2.11 it will not be possible to disable it. #5051
• [CHANGE] The configuration parameters -querier.iterators and -query.batch-iterators have been deprecated and will be removed in Mimir 2.11. Mimir runs by default with -querier.batch-iterators=true, and starting from version 2.11 it will not be possible to change this. #5114
• [CHANGE] Compactor: change default of -compactor.first-level-compaction-wait-period to 25m. #5128
• [CHANGE] Ruler: changed default of -ruler.poll-interval from 1m to 10m. Starting from this release, the configured rule groups will also be re-synced each time they're modified calling the ruler configuration API. #5170
• [FEATURE] Query-frontend: add -query-frontend.log-query-request-headers to enable logging of request headers in query logs. #5030
• [ENHANCEMENT] Add per-tenant limit -validation.max-native-histogram-buckets to be able to ignore native histogram samples that have too many buckets. #4765
• [ENHANCEMENT] Store-gateway: reduce memory usage in some LabelValues calls. #4789
• [ENHANCEMENT] Store-gateway: add a stage label to the metric cortex_bucket_store_series_data_touched. This label now applies to data_type="chunks" and data_type="series". The stage label has 2 values: processed - the number of series that parsed - and returned - the number of series selected from the processed bytes to satisfy the query. #4797 #4830
• [ENHANCEMENT] Distributor: make __meta_tenant_id label available in relabeling rules configured via metric_relabel_configs. #4725
• [ENHANCEMENT] Compactor: added the configurable limit compactor.block-upload-max-block-size-bytes or compactor_block_upload_max_block_size_bytes to limit the byte size of uploaded or validated blocks. #4680
• [ENHANCEMENT] Querier: reduce CPU utilisation when shuffle sharding is enabled with large shard sizes. #4851
• [ENHANCEMENT] Packaging: facilitate configuration management by instructing systemd to start mimir with a configuration file. #4810
• [ENHANCEMENT] Store-gateway: reduce memory allocations when looking up postings from cache. #4861 #4869 #4962 #5047
• [ENHANCEMENT] Store-gateway: retain only necessary bytes when reading series from the bucket. #4926
• [ENHANCEMENT] Ingester, store-gateway: clear the shutdown marker after a successful shutdown to enable reusing their persistent volumes in case the ingester or store-gateway is restarted. #4985
• [ENHANCEMENT] Store-gateway, query-frontend: Reduc...

2.9.0-rc.1

This release contains 260 PRs from 46 authors. Thank you!

Grafana Mimir version 2.9 release notes

Grafana Labs is excited to announce version 2.9 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Reduced store-gateway memory utilization on fetching series from long-term storage For queries that include broad label matchers (e.g. datacenter="dc1"), Mimir 2.9 will fetch a reduced volume of index data, which leads to a significant reduction in memory allocations in the store-gateway.
• Reduced CPU utilisation for some shuffle sharding scenarios Mimir queriers will now use significantly less CPU in cases where shuffle sharding is enabled for tenants with a shard size that's large but lower than the total number of ingesters.
• Reduced object storage API calls in compactors and rulers Mimir 2.9 comes with optimizations that will reduce the amount of times compactors and rulers need to access rules stored in object storage.
  • This release adds experimental support for a ruler storage cache. This cache should reduce the number of "list objects" API calls issued to the object storage when there are 2+ ruler replicas running in a Mimir cluster. The cache can be configured by setting the -ruler-storage.cache.* CLI flags or their respective YAML config options.
  • We also introduced a new feature to trigger a synchronization of tenant's rule groups as soon as changes to the rule configuration are made via API. This synchronization is in addition of the periodic syncing done every -ruler.poll-interval and allows to increase the polling interval. The new behavior is enabled globally by default but can be disabled with -ruler.sync-rules-on-changes-enabled=false or tuned at a per-tenant level.
• Experimental support for streaming chunks from ingester to querier This is expected to greatly reduce querier memory consumption when evaluating queries that select a large number of series, because chunks streamed from the querier can now be read into memory as needed.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.9 we have removed the following previously deprecated or experimental metrics:

• cortex_bucket_store_chunk_pool_requested_bytes_total
• cortex_bucket_store_chunk_pool_returned_bytes_total

The following configuration options are deprecated and will be removed in Grafana Mimir 2.11:

• The CLI flag -querier.query-ingesters-within. This configuration is moved to per-tenant overrides.
• The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
• The CLI flags -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes, -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes and -blocks-storage.bucket-store.max-chunk-pool-bytes.
• The CLI flags querier.iterators and -query.batch-iterators.

The following configuration options that were deprecated in 2.7 are removed:

• The CLI flag -blocks-storage.bucket-store.chunks-cache.subrange-size. A fixed value of 16000 is now always used.
• The CLI flag -blocks-storage.bucket-store.consistency-delay.
• The CLI flag -compactor.consistency-delay.
• The CLI flag -ingester.ring.readiness-check-ring-health.

The following experimental options and features are now stable:

• The CLI flag -query-frontend.query-sharding-max-regexp-size-bytes.
• The CLI flag -query-scheduler.max-used-instances.
• The CLI flags -(alertmanager|blocks|ruler)-storage.storage-prefix.
• The CLI flag -compactor.first-level-compaction-wait-period.
• The CLI flags -usage-stats.enabled and -usage-stats.installation-mode.
• The CLI flag -query-frontend.query-sharding-target-series-per-shard.

The following configuration option defaults were changed:

• The default value for the CLI flag -query-frontend.query-sharding-max-regexp-size-bytes was changed from 0 to 4096. As a result, queries with regex matchers exceeding this limit will not be sharded by default.
• The default value for the CLI flag -compactor.partial-block-deletion-delay was changed from 0s to 1d. As a result, partial blocks resulting from a failed block upload or deletion will be cleaned up automatically.
• The default value for the CLI flag -ruler.poll-interval was changed from 1m to 10m.

Bug fixes

• Store-gateway: Detect collisions in the postings cache. PR 4770
• Store-gateway: Fix panic caused by cached LabelValues responses with more than 655360 values. PR 5021

Changelog

2.9.0-rc.1

Grafana Mimir

• [CHANGE] Store-gateway: change expanded postings, postings, and label values index cache key format. These caches will be invalidated when rolling out the new Mimir version. #4770 #4978 #5037
• [CHANGE] Distributor: remove the "forwarding" feature as it isn't necessary anymore. #4876
• [CHANGE] Query-frontend: Change the default value of -query-frontend.query-sharding-max-regexp-size-bytes from 0 to 4096. #4932
• [CHANGE] Querier: -querier.query-ingesters-within has been moved from a global flag to a per-tenant override. #4287
• [CHANGE] Querier: Use -blocks-storage.tsdb.retention-period instead of -querier.query-ingesters-within for calculating the lookback period for shuffle sharded ingesters. Setting -querier.query-ingesters-within=0 no longer disables shuffle sharding on the read path. #4287
• [CHANGE] Block upload: /api/v1/upload/block/{block}/files endpoint now allows file uploads with no Content-Length. #4956
• [CHANGE] Store-gateway: deprecate configuration parameters for chunk pooling, they will be removed in Mimir 2.11. The following options are now also ignored: #4996
  • -blocks-storage.bucket-store.max-chunk-pool-bytes
  • -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
  • -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
• [CHANGE] Store-gateway: remove metrics cortex_bucket_store_chunk_pool_requested_bytes_total and cortex_bucket_store_chunk_pool_returned_bytes_total. #4996
• [CHANGE] Compactor: change default of -compactor.partial-block-deletion-delay to 1d. This will automatically clean up partial blocks that were a result of failed block upload or deletion. #5026
• [CHANGE] Compactor: the deprecated configuration parameter -compactor.consistency-delay has been removed. #5050
• [CHANGE] Store-gateway: the deprecated configuration parameter -blocks-storage.bucket-store.consistency-delay has been removed. #5050
• [CHANGE] The configuration parameter -blocks-storage.bucket-store.bucket-index.enabled has been deprecated and will be removed in Mimir 2.11. Mimir is running by default with the bucket index enabled since version 2.0, and starting from the version 2.11 it will not be possible to disable it. #5051
• [CHANGE] The configuration parameters -querier.iterators and -query.batch-iterators have been deprecated and will be removed in Mimir 2.11. Mimir runs by default with -querier.batch-iterators=true, and starting from version 2.11 it will not be possible to change this. #5114
• [CHANGE] Compactor: change default of -compactor.first-level-compaction-wait-period to 25m. #5128
• [CHANGE] Ruler: changed default of -ruler.poll-interval from 1m to 10m. Starting from this release, the configured rule groups will also be re-synced each time they're modified calling the ruler configuration API. #5170
• [FEATURE] Query-frontend: add -query-frontend.log-query-request-headers to enable logging of request headers in query logs. #5030
• [ENHANCEMENT] Add per-tenant limit -validation.max-native-histogram-buckets to be able to ignore native histogram samples that have too many buckets. #4765
• [ENHANCEMENT] Store-gateway: reduce memory usage in some LabelValues calls. #4789
• [ENHANCEMENT] Store-gateway: add a stage label to the metric cortex_bucket_store_series_data_touched. This label now applies to data_type="chunks" and data_type="series". The stage label has 2 values: processed - the number of series that parsed - and returned - the number of series selected from the processed bytes to satisfy the query. #4797 #4830
• [ENHANCEMENT] Distributor: make __meta_tenant_id label available in relabeling rules configured via metric_relabel_configs. #4725
• [ENHANCEMENT] Compactor: added the configurable limit compactor.block-upload-max-block-size-bytes or compactor_block_upload_max_block_size_bytes to limit the byte size of uploaded or validated blocks. #4680
• [ENHANCEMENT] Querier: reduce CPU utilisation when shuffle sharding is enabled with large shard sizes. #4851
• [ENHANCEMENT] Packaging: facilitate configuration management by instructing systemd to start mimir with a configuration file. #4810
• [ENHANCEMENT] Store-gateway: reduce memory allocations when looking up postings from cache. #4861 #4869 #4962 #5047
• [ENHANCEMENT] Store-gateway: retain only necessary bytes when reading series from the bucket. #4926
• [ENHANCEMENT] Ingester, store-gateway: clear the shutdown marker after a successful shutdown to enable reusing their persistent volumes in case the ingester or store-gateway is restarted. #4985
• [ENHANCEMENT] Store-gateway, query-frontend: Reduced memory allocations when looking up cached entries from Memcached. #4862
• [ENHANCEMENT] Alertmanager: Add additional template function queryFromGeneratorURL returning query URL decoded query from the GeneratorURL field of an alert. #4301
• [ENHANCEMENT] Rul...

2.8.0

This release contains 223 PRs from 53 authors, including new contributors Abdurrahman J. Allawala, Ashray Jain, Cyrill N, Daniel Barnes, Dave, David van der Spek, day4me, Devin Trejo, Dmitriy Okladin, Gabriel Santos, inbarpatashnik, Johannes Tandler, Julien Girard, KingJ, Miller, Rafał Boniecki, Raphael Ferreira, Raúl Marín, Ruslan Kovalov, Shagit Ziganshin, shanmugara, Wilfried ROSET. Thank you!

Grafana Mimir version 2.8.0 release notes

Grafana Labs is excited to announce version 2.8 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Experimental support for using Redis as cache Mimir now can use Redis for caching results, chunks, index and metadata.
• Experimental support for fetching secret from Vault for TLS configuration.
• Experimental support for querying native histograms. This support is not finalized as the related Prometheus API is also experimental, thus the exact behavior might change in future releases.
• Query-frontend and ruler now use protobuf internal query result payload format by default This reduces the CPU and memory utilisation of the querier, query-frontend and ruler, as well as reducing network bandwidth consumed between these components.
• Query-frontend cached results now contain timestamp This allows Mimir to check if cached results are still valid based on current TTL configured for tenant. Results cached by previous Mimir version are used until they expire from cache, which can take up to 7 days. If you need to use per-tenant TTL sooner, please flush results cache manually.
• Optimized regular expression label matchers This reduces CPU utilisation in ingesters and store-gateways when running queries containing regular expression label matchers.
• Store-gateway now use streaming for LabelNames RPC This improves memory utilization in store-gateway when calling LabelNames RPC.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.8 we have removed the following previously deprecated or experimental metrics:

• cortex_bucket_store_series_get_all_duration_seconds
• cortex_bucket_store_series_merge_duration_seconds
• cortex_ingester_tsdb_wal_replay_duration_seconds

The following configuration options are deprecated and will be removed in Grafana Mimir 2.10:

• The CLI flag -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup and its respective YAML configuration option tsdb.max_tsdb_opening_concurrency_on_startup.

The following configuration options that were deprecated in 2.6 are removed:

• The CLI flag -store.max-query-length and its respective YAML configuration option limits.max_query_length.

The following configuration options that were deprecated in 2.5 are removed:

• The CLI flag -azure.msi-resource.

The following experimental options and features are now stable:

• The protobuf internal query result payload format, which is now enabled by default

We changed default value of block storage retention period. The default value for -blocks-storage.tsdb.retention-period was 24h and now is 13h

Bug fixes

• Querier: Streaming remote read will now continue to return multiple chunks per frame after the first frame. PR 4423
• Query-frontend: don't retry queries which error inside PromQL. PR 4643
• Store-gateway & query-frontend: report more consistent statistics for fetched index bytes. PR 4671

Changelog

2.8.0

Grafana Mimir

• [CHANGE] Ingester: changed experimental CLI flag from -out-of-order-blocks-external-label-enabled to -ingester.out-of-order-blocks-external-label-enabled #4440
• [CHANGE] Store-gateway: The following metrics have been removed: #4332
  • cortex_bucket_store_series_get_all_duration_seconds
  • cortex_bucket_store_series_merge_duration_seconds
• [CHANGE] Ingester: changed default value of -blocks-storage.tsdb.retention-period from 24h to 13h. If you're running Mimir with a custom configuration and you're overriding -querier.query-store-after to a value greater than the default 12h then you should increase -blocks-storage.tsdb.retention-period accordingly. #4382
• [CHANGE] Ingester: the configuration parameter -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup has been deprecated and will be removed in Mimir 2.10. #4445
• [CHANGE] Query-frontend: Cached results now contain timestamp which allows Mimir to check if cached results are still valid based on current TTL configured for tenant. Results cached by previous Mimir version are used until they expire from cache, which can take up to 7 days. If you need to use per-tenant TTL sooner, please flush results cache manually. #4439
• [CHANGE] Ingester: the cortex_ingester_tsdb_wal_replay_duration_seconds metrics has been removed. #4465
• [CHANGE] Query-frontend and ruler: use protobuf internal query result payload format by default. This feature is no longer considered experimental. #4557 #4709
• [CHANGE] Ruler: reject creating federated rule groups while tenant federation is disabled. Previously the rule groups would be silently dropped during bucket sync. #4555
• [CHANGE] Compactor: the /api/v1/upload/block/{block}/finish endpoint now returns a 429 status code when the compactor has reached the limit specified by -compactor.max-block-upload-validation-concurrency. #4598
• [CHANGE] Compactor: when starting a block upload the maximum byte size of the block metadata provided in the request body is now limited to 1 MiB. If this limit is exceeded a 413 status code is returned. #4683
• [CHANGE] Store-gateway: cache key format for expanded postings has changed. This will invalidate the expanded postings in the index cache when deployed. #4667
• [FEATURE] Cache: Introduce experimental support for using Redis for results, chunks, index, and metadata caches. #4371
• [FEATURE] Vault: Introduce experimental integration with Vault to fetch secrets used to configure TLS for clients. Server TLS secrets will still be read from a file. tls-ca-path, tls-cert-path and tls-key-path will denote the path in Vault for the following CLI flags when -vault.enabled is true: #4446.
  • -distributor.ha-tracker.etcd.*
  • -distributor.ring.etcd.*
  • -distributor.forwarding.grpc-client.*
  • -querier.store-gateway-client.*
  • -ingester.client.*
  • -ingester.ring.etcd.*
  • -querier.frontend-client.*
  • -query-frontend.grpc-client-config.*
  • -query-frontend.results-cache.redis.*
  • -blocks-storage.bucket-store.index-cache.redis.*
  • -blocks-storage.bucket-store.chunks-cache.redis.*
  • -blocks-storage.bucket-store.metadata-cache.redis.*
  • -compactor.ring.etcd.*
  • -store-gateway.sharding-ring.etcd.*
  • -ruler.client.*
  • -ruler.alertmanager-client.*
  • -ruler.ring.etcd.*
  • -ruler.query-frontend.grpc-client-config.*
  • -alertmanager.sharding-ring.etcd.*
  • -alertmanager.alertmanager-client.*
  • -memberlist.*
  • -query-scheduler.grpc-client-config.*
  • -query-scheduler.ring.etcd.*
  • -overrides-exporter.ring.etcd.*
• [FEATURE] Distributor, ingester, querier, query-frontend, store-gateway: add experimental support for native histograms. Requires that the experimental protobuf query result response format is enabled by -query-frontend.query-result-response-format=protobuf on the query frontend. #4286 #4352 #4354 #4376 #4377 #4387 #4396 #4425 #4442 #4494 #4512 #4513 #4526
• [FEATURE] Added -<prefix>.s3.storage-class flag to configure the S3 storage class for objects written to S3 buckets. #4300
• [FEATURE] Add freebsd to the target OS when generating binaries for a Mimir release. #4654
• [FEATURE] Ingester: Add prepare-shutdown endpoint which can be used as part of Kubernetes scale down automations. #4718
• [ENHANCEMENT] Add timezone information to Alpine Docker images. #4583
• [ENHANCEMENT] Ruler: Sync rules when ruler JOINING the ring instead of ACTIVE, In order to reducing missed rule iterations during ruler restarts. #4451
• [ENHANCEMENT] Allow to define service name used for tracing via JAEGER_SERVICE_NAME environment variable. #4394
• [ENHANCEMENT] Querier and query-frontend: add experimental, more performant protobuf query result response format enabled with -query-frontend.query-result-response-format=protobuf. #4304 #4318 #4375
• [ENHANCEMENT] Compactor: added experimental configuration parameter -compactor.first-level-compaction-wait-period, to configure how long the compactor should wait before compacting 1st level blocks (uploaded by ingesters). This configuration option allows to reduce the chances compactor begins compacting blocks before all ingesters have uploaded their blocks to the storage. #4401
• [ENHANCEMENT] Store-gateway: use more efficient chunks fetching and caching. #4255
• [ENHANCEMENT] Query-frontend and ruler: add experimental, more performant protobuf internal query result response format enabled with -ruler.query-frontend.query-result-response-format=protobuf. #4331
• [ENHANCEMENT] Ruler: increased tolerance for missed iterations on alerts, reducing the chances of flapping firing alerts during ruler restarts. #4432
• [ENHANCEMENT] Optimized .* and .+ regular expression label matchers. #4432
• [ENHANCEMENT] Optimized regular expression label matchers with alternates (e.g. a|b|c). #4647
• [ENHANCEMENT] Added an in-memory cache for regular expression mat...

2.7.3

2.7.3

Grafana Mimir

• [BUGFIX] Security: updates Go to version 1.20.4 to fix CVE-2023-24539, CVE-2023-24540, CVE-2023-29400. #4905

Full Changelog: mimir-2.7.2...mimir-2.7.3

2.6.2

Changelog

2.6.2

Grafana Mimir

• [BUGFIX] Security: updates Go to version 1.20.4 to fix CVE-2023-24539, CVE-2023-24540, CVE-2023-29400. #4903

Full Changelog: mimir-2.6.1...mimir-2.6.2

2.8.0-rc.2

This release contains 2 PRs from 2 authors. Thank you!

Changelog

2.8.0-rc.2

Grafana Mimir

• [ENHANCEMENT] Ruler: Improve rule upload performance when not enforcing per-tenant rule group limits. #4828

All changes in this release: mimir-2.8.0-rc.1...mimir-2.8.0-rc.2

2.8.0-rc.1

This release contains 8 PRs from 2 authors. Thank you!

Changelog

2.8.0-rc.1

Grafana Mimir

• [ENHANCEMENT] Improved memory limit on the in-memory cache used for regular expression matchers. #4751
• [ENHANCEMENT] Go: update to 1.20.3. #4773
• [BUGFIX] Packaging: fix preremove script preventing upgrades. #4801

All changes in this release: mimir-2.8.0-rc.0...mimir-2.8.0-rc.1

2.6.1

This release contains 3 PRs from 2 authors. Thank you!

Changelog

2.6.1

Grafana Mimir

• [BUGFIX] Security: updates Go to version 1.20.3 to fix CVE-2023-24538 #4798

All changes in this release: mimir-2.6.0...mimir-2.6.1

2.7.2

This release contains 2 PRs from 2 authors. Thank you!

Changelog

2.7.2

Grafana Mimir

• [BUGFIX] Security: updated Go version to 1.20.3 to fix CVE-2023-24538 #4795

All changes in this release: mimir-2.7.1...mimir-2.7.2

2.8.0-rc.0

This release contains 210 PRs from 53 authors, including new contributors Abdurrahman J. Allawala, Ashray Jain, Cyrill N, Daniel Barnes, Dave, David van der Spek, day4me, Devin Trejo, Dmitriy Okladin, Gabriel Santos, inbarpatashnik, Johannes Tandler, Julien Girard, KingJ, Miller, Rafał Boniecki, Raphael Ferreira, Raúl Marín, Ruslan Kovalov, Shagit Ziganshin, shanmugara, Wilfried ROSET. Thank you!

Grafana Mimir version 2.8.0-rc.0 release notes

Grafana Labs is excited to announce version 2.8 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Changed default value of block storage retention period The default value for -blocks-storage.tsdb.retention-period was 24h and now is 13h.
• Query-frontend cached results now contain timestamp This allows Mimir to check if cached results are still valid based on current TTL configured for tenant. Results cached by previous Mimir version are used until they expire from cache, which can take up to 7 days. If you need to use per-tenant TTL sooner, please flush results cache manually.
• Experimental support for using Redis as cache Mimir now can use Redis for caching results, chunks, index and metadata.
• Experimental support for fetching secret from Vault for TLS configuration.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.8 we have removed the following previously deprecated or experimental configuration options or metrics.

The following metrics have been removed cortex_bucket_store_series_get_all_duration_seconds, cortex_bucket_store_series_merge_duration_seconds,
cortex_ingester_tsdb_wal_replay_duration_seconds.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.10:

• The CLI flag -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup and its respective YAML configuration option tsdb.max_tsdb_opening_concurrency_on_startup.

The following experimental options and features are now stable:

• Use protobuf internal query result payload format by default.

Bug fixes

• Querier: Streaming remote read will now continue to return multiple chunks per frame after the first frame. PR 4423
• Query-frontend: don't retry queries which error inside PromQL. PR 4643
• Store-gateway & query-frontend: report more consistent statistics for fetched index bytes. PR 4671
• Native histograms: fix how IsFloatHistogram determines if mimirpb.Histogram is a float histogram. PR 4706
• Query-frontend: fix query sharding for native histograms. PR 4666

Changelog

2.8.0-rc.0

Grafana Mimir

• [CHANGE] Ingester: changed experimental CLI flag from -out-of-order-blocks-external-label-enabled to -ingester.out-of-order-blocks-external-label-enabled #4440
• [CHANGE] Store-gateway: The following metrics have been removed: #4332
  • cortex_bucket_store_series_get_all_duration_seconds
  • cortex_bucket_store_series_merge_duration_seconds
• [CHANGE] Ingester: changed default value of -blocks-storage.tsdb.retention-period from 24h to 13h. If you're running Mimir with a custom configuration and you're overriding -querier.query-store-after to a value greater than the default 12h then you should increase -blocks-storage.tsdb.retention-period accordingly. #4382
• [CHANGE] Ingester: the configuration parameter -blocks-storage.tsdb.max-tsdb-opening-concurrency-on-startup has been deprecated and will be removed in Mimir 2.10. #4445
• [CHANGE] Query-frontend: Cached results now contain timestamp which allows Mimir to check if cached results are still valid based on current TTL configured for tenant. Results cached by previous Mimir version are used until they expire from cache, which can take up to 7 days. If you need to use per-tenant TTL sooner, please flush results cache manually. #4439
• [CHANGE] Ingester: the cortex_ingester_tsdb_wal_replay_duration_seconds metrics has been removed. #4465
• [CHANGE] Query-frontend and ruler: use protobuf internal query result payload format by default. This feature is no longer considered experimental. #4557 #4709
• [CHANGE] Ruler: reject creating federated rule groups while tenant federation is disabled. Previously the rule groups would be silently dropped during bucket sync. #4555
• [CHANGE] Compactor: the /api/v1/upload/block/{block}/finish endpoint now returns a 429 status code when the compactor has reached the limit specified by -compactor.max-block-upload-validation-concurrency. #4598
• [CHANGE] Compactor: when starting a block upload the maximum byte size of the block metadata provided in the request body is now limited to 1 MiB. If this limit is exceeded a 413 status code is returned. #4683
• [CHANGE] Store-gateway: cache key format for expanded postings has changed. This will invalidate the expanded postings in the index cache when deployed. #4667
• [FEATURE] Cache: Introduce experimental support for using Redis for results, chunks, index, and metadata caches. #4371
• [FEATURE] Vault: Introduce experimental integration with Vault to fetch secrets used to configure TLS for clients. Server TLS secrets will still be read from a file. tls-ca-path, tls-cert-path and tls-key-path will denote the path in Vault for the following CLI flags when -vault.enabled is true: #4446.
  • -distributor.ha-tracker.etcd.*
  • -distributor.ring.etcd.*
  • -distributor.forwarding.grpc-client.*
  • -querier.store-gateway-client.*
  • -ingester.client.*
  • -ingester.ring.etcd.*
  • -querier.frontend-client.*
  • -query-frontend.grpc-client-config.*
  • -query-frontend.results-cache.redis.*
  • -blocks-storage.bucket-store.index-cache.redis.*
  • -blocks-storage.bucket-store.chunks-cache.redis.*
  • -blocks-storage.bucket-store.metadata-cache.redis.*
  • -compactor.ring.etcd.*
  • -store-gateway.sharding-ring.etcd.*
  • -ruler.client.*
  • -ruler.alertmanager-client.*
  • -ruler.ring.etcd.*
  • -ruler.query-frontend.grpc-client-config.*
  • -alertmanager.sharding-ring.etcd.*
  • -alertmanager.alertmanager-client.*
  • -memberlist.*
  • -query-scheduler.grpc-client-config.*
  • -query-scheduler.ring.etcd.*
  • -overrides-exporter.ring.etcd.*
• [FEATURE] Distributor, ingester, querier, query-frontend, store-gateway: add experimental support for native histograms. Requires that the experimental protobuf query result response format is enabled by -query-frontend.query-result-response-format=protobuf on the query frontend. #4286 #4352 #4354 #4376 #4377 #4387 #4396 #4425 #4442 #4494 #4512 #4513 #4526
• [FEATURE] Added -<prefix>.s3.storage-class flag to configure the S3 storage class for objects written to S3 buckets. #4300
• [FEATURE] Add freebsd to the target OS when generating binaries for a Mimir release. #4654
• [FEATURE] Ingester: Add prepare-shutdown endpoint which can be used as part of Kubernetes scale down automations. #4718
• [ENHANCEMENT] Add timezone information to Alpine Docker images. #4583
• [ENHANCEMENT] Ruler: Sync rules when ruler JOINING the ring instead of ACTIVE, In order to reducing missed rule iterations during ruler restarts. #4451
• [ENHANCEMENT] Allow to define service name used for tracing via JAEGER_SERVICE_NAME environment variable. #4394
• [ENHANCEMENT] Querier and query-frontend: add experimental, more performant protobuf query result response format enabled with -query-frontend.query-result-response-format=protobuf. #4304 #4318 #4375
• [ENHANCEMENT] Compactor: added experimental configuration parameter -compactor.first-level-compaction-wait-period, to configure how long the compactor should wait before compacting 1st level blocks (uploaded by ingesters). This configuration option allows to reduce the chances compactor begins compacting blocks before all ingesters have uploaded their blocks to the storage. #4401
• [ENHANCEMENT] Store-gateway: use more efficient chunks fetching and caching. #4255
• [ENHANCEMENT] Query-frontend and ruler: add experimental, more performant protobuf internal query result response format enabled with -ruler.query-frontend.query-result-response-format=protobuf. #4331
• [ENHANCEMENT] Ruler: increased tolerance for missed iterations on alerts, reducing the chances of flapping firing alerts during ruler restarts. #4432
• [ENHANCEMENT] Optimized .* and .+ regular expression label matchers. #4432
• [ENHANCEMENT] Optimized regular expression label matchers with alternates (e.g. a|b|c). #4647
• [ENHANCEMENT] Added an in-memory cache for regular expression matchers, to avoid parsing and compiling the same expression multiple times when used in recurring queries. #4633
• [ENHANCEMENT] Query-frontend: results cache TTL is now configurable by using -query-frontend.results-cache-ttl and -query-frontend.results-cache-ttl-for-out-of-order-time-window options. These values can also be specified per tenant. Default values are unchanged (7 days and 10 minutes respectively). #4385
• [ENHANCEMENT] Ingester: added advanced configuration parameter -blocks-storage.tsdb.wal-replay-concurrency representing the maximum number of CPUs used during WAL replay. #4445
• [ENHANCEMENT] Ingester: added metrics cortex_ingester_tsdb_open_duration_seconds_total to measure the total time it takes to open all existing TSDBs. T...