Cortex 1.11.0

[bboreham]

This release contains 76 contributions from 31 authors. Thank you!

A broad range of improvements, including support for cloud services such as Memcached auto-discovery and Amazon SNS.

Cortex

• [CHANGE] Memberlist: Expose default configuration values to the command line options. Note that setting these explicitly to zero will no longer cause the default to be used. If the default is desired, then do set the option. The following are affected: Expose default configuration values for memberlist. #4276
  • -memberlist.stream-timeout
  • -memberlist.retransmit-factor
  • -memberlist.pull-push-interval
  • -memberlist.gossip-interval
  • -memberlist.gossip-nodes
  • -memberlist.gossip-to-dead-nodes-time
  • -memberlist.dead-node-reclaim-time
• [CHANGE] -querier.max-fetched-chunks-per-query previously applied to chunks from ingesters and store separately; now the two combined should not exceed the limit. max chunks per query limit shared between ingesters and storage gateways #4260
• [CHANGE] Memberlist: the metric memberlist_kv_store_value_bytes has been removed due to values no longer being stored in-memory as encoded bytes. Optimise memberlist kv store access by storing data unencoded. #4345
• [CHANGE] Some files and directories created by Cortex components on local disk now have stricter permissions, and are only readable by owner, but not group or others. Upgrade thanos version #4394
• [CHANGE] The metric cortex_deprecated_flags_inuse_total has been renamed to deprecated_flags_inuse_total as part of using grafana/dskit functionality. Use flagext module from grafana/dskit #4443
• [FEATURE] Ruler: Add new -ruler.query-stats-enabled which when enabled will report the cortex_ruler_query_seconds_total as a per-user metric that tracks the sum of the wall time of executing queries in the ruler in seconds. Add a new config and metric for reporting ruler query execution wall time. #4317
• [FEATURE] Query Frontend: Add cortex_query_fetched_series_total and cortex_query_fetched_chunks_bytes_total per-user counters to expose the number of series and bytes fetched as part of queries. These metrics can be enabled with the -frontend.query-stats-enabled flag (or its respective YAML config option query_stats_enabled). Add per-user query metrics for series and bytes returned #4343
• [FEATURE] AlertManager: Add support for SNS Receiver. Adding AlertManager SNS Receiver Implementation #4382
• [FEATURE] Distributor: Add label status to metric cortex_distributor_ingester_append_failures_total Send new label 'status' for ingester failures #4442
• [FEATURE] Queries: Added present_over_time PromQL function, also some TSDB optimisations. Update Thanos to v0.19.1-0.20210923155558-c15594a03c45. #4505
• [ENHANCEMENT] Add timeout for waiting on compactor to become ACTIVE in the ring. Add timeout for waiting on compactor to become ACTIVE in the ring. #4262
• [ENHANCEMENT] Reduce memory used by streaming queries, particularly in ruler. MergeIterator: allocate less memory at first #4341
• [ENHANCEMENT] Ring: allow experimental configuration of disabling of heartbeat timeouts by setting the relevant configuration value to zero. Applies to the following: Allow setting ring heartbeat timeout to zero to disable timeout check. #4342
  • -distributor.ring.heartbeat-timeout
  • -ring.heartbeat-timeout
  • -ruler.ring.heartbeat-timeout
  • -alertmanager.sharding-ring.heartbeat-timeout
  • -compactor.ring.heartbeat-timeout
  • -store-gateway.sharding-ring.heartbeat-timeout
• [ENHANCEMENT] Ring: allow heartbeats to be explicitly disabled by setting the interval to zero. This is considered experimental. This applies to the following configuration options: Allow disabling of ring heartbeats by setting relevant options to zero. #4344
  • -distributor.ring.heartbeat-period
  • -ingester.heartbeat-period
  • -ruler.ring.heartbeat-period
  • -alertmanager.sharding-ring.heartbeat-period
  • -compactor.ring.heartbeat-period
  • -store-gateway.sharding-ring.heartbeat-period
• [ENHANCEMENT] Memberlist: optimized receive path for processing ring state updates, to help reduce CPU utilization in large clusters. Optimise memberlist kv store access by storing data unencoded. #4345
• [ENHANCEMENT] Memberlist: expose configuration of memberlist packet compression via -memberlist.compression=enabled. Expose configuration of memberlist packet compression. #4346
• [ENHANCEMENT] Update Go version to 1.16.6. Update go version in build image to 1.16.6 #4362
• [ENHANCEMENT] Updated Prometheus to include changes from LabelNames API with matchers  prometheus/prometheus#9083. Now whenever /labels API calls include matchers, blocks store is queried for LabelNames with matchers instead of Series calls which was inefficient. Upgrade Prometheus to LabelNames with matchers #4380
• [ENHANCEMENT] Querier: performance improvements in socket and memory handling. querier: exhaust response body before closing #4429 Reuse the byte buffer from GRPC response in the frontend. #4377
• [ENHANCEMENT] Exemplars are now emitted for all gRPC calls and many operations tracked by histograms. Chore: Upgrade weaveworks/common #4462
• [ENHANCEMENT] New options -server.http-listen-network and -server.grpc-listen-network allow binding as 'tcp4' or 'tcp6'. Chore: Upgrade weaveworks/common #4462
• [ENHANCEMENT] Rulers: Using shuffle sharding subring on GetRules API. Using subring on getShardedRules when shuffle sharding #4466
• [ENHANCEMENT] Support memcached auto-discovery via auto-discovery flag, introduced by thanos in query: add memcached autodiscovery support thanos-io/thanos#4487. Both AWS and Google Cloud memcached service support auto-discovery, which returns a list of nodes of the memcached cluster. add memcached auto-discovery support from thanos #4412
• [BUGFIX] Fixes a panic in the query-tee when comparing result. Fixes a panic in the query-tee. #4465
• [BUGFIX] Frontend: Fixes @ modifier functions (start/end) when splitting queries by time. Fixes @ modifier when splitting queries by time. #4464
• [BUGFIX] Compactor: compactor will no longer try to compact blocks that are already marked for deletion. Previously compactor would consider blocks marked for deletion within -compactor.deletion-delay / 2 period as eligible for compaction. Don't try to compact blocks marked for deletion. #4328
• [BUGFIX] HA Tracker: when cleaning up obsolete elected replicas from KV store, tracker didn't update number of cluster per user correctly. Fix bug with counting of clusters for deleted elected replicas. #4336
• [BUGFIX] Ruler: fixed counting of PromQL evaluation errors as user-errors when updating cortex_ruler_queries_failed_total. Fix ruler query failure reporting #4335
• [BUGFIX] Ingester: When using block storage, prevent any reads or writes while the ingester is stopping. This will prevent accessing TSDB blocks once they have been already closed. Prevent read on TSDB once closeAllTSDB function has been called #4304
• [BUGFIX] Ingester: fixed ingester stuck on start up (LEAVING ring state) when -ingester.heartbeat-period=0 and -ingester.unregister-on-shutdown=false. Fix Ingesters unable to re-join the cluster then unregister_on_shutdown=false + -ingester.heartbeat-period=0 #4366
• [BUGFIX] Ingester: panic during shutdown while fetching batches from cache. Ingesters panic during shutdown causing data loss #4397
• [BUGFIX] Querier: After query-frontend restart, querier may have lower than configured concurrency. Fix bug where querier may not be able to achieve max-concurrent #4417
• [BUGFIX] Memberlist: forward only changes, not entire original message. Memberlist now forwards only changes, not full original received message #4419
• [BUGFIX] Memberlist: don't accept old tombstones as incoming change, and don't forward such messages to other gossip members. Memberlist ignore old tombstones #4420
• [BUGFIX] Querier: fixed panic when querying exemplars and using -distributor.shard-by-all-labels=false. fix querier panics when query exemplars #4473
• [BUGFIX] Querier: honor querier minT,maxT if nil SelectHints are passed to Select(). [querier] honor querier mint,maxt if no SelectHints are passed to Select #4413
• [BUGFIX] Compactor: fixed panic while collecting Prometheus metrics. Fixed panic while collecting Prometheus metrics #4483
• [BUGFIX] Update go-kit package to fix spurious log messages Update go-kit/kit to 0.12.0 to fix spurious debug logs #4544

---

This discussion was created from the release Cortex 1.11.0.

[mvkrishna86]

Is exemplars supported by cortex now?

[bboreham]

Waiting for someone to do the work. #4364

[mvkrishna86]

Yes, have been since 1.10.

I am not able to see any exemplars in Grafana with cortex (prometheus) data source. I am using cortex 1.11.0, and pushing the metrics using grafana agent.
With prometheus, I was able to see the metrics with exemplars in grafana.

[mvkrishna86]

Got this: blocks_storage.tsdb.max_exemplars. Can you tell me what is the recommended value for this?

[bboreham]

There is no recommended value. Each one takes some memory, and the amount depends on how many labels are on your exemplars.
Decide how much memory you want to devote to exemplars, and set the max accordingly.

[mvkrishna86]

Thanks @bboreham , looks like they will be saved only in ingesters and not be pushed to block storage. May I know if you have any plan for storing them in block storage?
Ref: https://cortexmetrics.io/docs/configuration/v1guarantees/
Exemplar storage, currently in-memory only within the Ingester based on Prometheus exemplar storage (-blocks-storage.tsdb.max-exemplars)