Querier not working with Block storage #3596
-
|
I am working on implementing block storage cortex solution and in the process of building a cortex cluster. I have hit a roadblock when installing/deploying querier. So far i have installed the following
Problem: Not able to install querier service, as it is throwing the following error. There is nothing stated in the querier config for providing explicit etcd endpoints or so.. But out of curiosity i provided the etcd endpoints via the CLI to see if that could do something. But then i started seeing some other error. My Deployment code apiVersion: apps/v1
kind: Deployment
metadata:
name: cortex-querier
namespace: default
labels:
app: cortex-querier
chart: cortex-0.2.0
release: cortex
heritage: Tiller
annotations:
{}
spec:
replicas: 3
selector:
matchLabels:
app: cortex-querier
release: cortex
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: cortex-querier
name: cortex-querier
target: querier
release: cortex
annotations:
checksum/config: 6dd406f10b037c8f1727c9703989ae342b09846df41aea9dacbd8d76a7c6fde4
prometheus.io/port: http-metrics
prometheus.io/scrape: "true"
spec:
securityContext:
{}
initContainers:
[]
containers:
- name: cortex
image: "quay.io/cortexproject/cortex:v1.5.0"
imagePullPolicy: IfNotPresent
args:
- "-target=querier"
- "-config.file=/etc/cortex/cortex.yaml"
- "-etcd.endpoints=http://etcd-client.default.svc.cluster.local:2379"
volumeMounts:
- name: config
mountPath: /etc/cortex
- name: storage
mountPath: "/data"
subPath:
ports:
- name: http-metrics
containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
resources:
limits:
cpu: 600m
memory: 4Gi
requests:
cpu: 600m
memory: 2Gi
securityContext:
readOnlyRootFilesystem: true
env:
nodeSelector:
{}
affinity:
{}
tolerations:
[]
terminationGracePeriodSeconds: 180
volumes:
- name: config
secret:
secretName: cortex
- name: storage
# emptyDir: {}
hostPath:
path: "/data"
type: DirectoryOrCreate`My overall config config:
auth_enabled: false
ingester_client:
grpc_client_config:
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
use_gzip_compression: false
ingester:
lifecycler:
join_after: 0s
final_sleep: 0s
num_tokens: 256
ring:
replication_factor: 2
kvstore:
store: "etcd"
querier:
default_evaluation_interval: 15s
store_gateway_addresses: cortex-store-gateway-headless.default.svc.cluster.local:8080
limits:
ingestion_rate: 750000
ingestion_rate_strategy: local
ingestion_burst_size: 1000000
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
server:
http_listen_port: 8080
grpc_listen_port: 9095
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
grpc_server_max_concurrent_streams: 1000
store_gateway:
sharding_enabled: false
storage:
engine: blocks
blocks_storage:
tsdb:
dir: /tmp/cortex/tsdb
block_ranges_period: ['0h1m0s']
bucket_store:
sync_dir: /tmp/cortex/tsdb-sync
backend: azure
azure:
account_key: <key>
account_name: cortexblockstorage
container_name: cortex-poc
distributor:
ring:
kvstore:
store: "etcd"
shard_by_all_labels: true
pool:
health_check_ingesters: true |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 2 replies
-
|
@vicmania2007 FYI: I updated you comment to include proper formatting for your code blocks azure:
account_key: cI/hhGjWOfkNo5xeP8Fvn524unLN2N9wf6dDhh8+zA8HO2Qq1Tac5v8X9iARZWvlid9avMwu9ni5QNksVtZ/Pg==
account_name: cortexblockstorage
container_name: cortex-pocAlso you pasted a key into your comment, I highly recommend you invalidate this key ASAP |
Beta Was this translation helpful? Give feedback.
-
|
@jtlisi i was still formatting on the side, but thanks for doing that to me. Yes i didnt realize about the azure key. I will immediately invalidate it. |
Beta Was this translation helpful? Give feedback.
-
|
@vicmania2007 I think the following snippet is responsible for your issue: securityContext:
readOnlyRootFilesystem: trueThis means the cortex pod can't create files locally unless they are scoped to a temporary volume. This is going to cause an issue when running Cortex because it does create a number of temporary files in certain cases. You can either remove this config or ensure each path that requires file system access is set to a mounted volume. To update the path the query tracker uses for a temporary file, you need to change the following config in the active_query_tracker_dir: <string> | default = "./active-query-tracker |
Beta Was this translation helpful? Give feedback.
-
|
Note that you can also set this to “”, which disables active query tracker. Unfortunately it also disables max PromQL concurrency limit, which is why it doesn’t default to “”. |
Beta Was this translation helpful? Give feedback.
-
|
I removed the below config, and then the querier was working. Thanks for the help guys. |
Beta Was this translation helpful? Give feedback.
@vicmania2007 I think the following snippet is responsible for your issue:
This means the cortex pod can't create files locally unless they are scoped to a temporary volume. This is going to cause an issue when running Cortex because it does create a number of temporary files in certain cases. You can either remove this config or ensure each path that requires file system access is set to a mounted volume.
To update the path the query tracker uses for a temporary file, you need to change the following config in the
querierconfig section: