Issue after drain nodes k8s - s3 chunks reset

[ericfolch]

Hi, after doing a drain and an uncord on my k8s cluster, when cortex started again, I lost all my historic data.
I assume it's not a K8s PV issue because it's not the first time this deployment has been rebooted.
I am using s3 as block storage.
The connection to s3 is OK, actually chunks are being loagind after the uncordon.
In s3 I see the all historic chunks and I see that every time I try something the bucket index (last update) is regenerated, but even so in grafana, the queries do not bring me data prior to the drain / uncordon.

This is my configuration

cat /etc/cortex/cortex.yaml
alertmanager:
enable_api: false
external_url: /api/prom/alertmanager
storage: {}
api:
prometheus_http_prefix: /prometheus
response_compression_enabled: true
auth_enabled: true
blocks_storage:
backend: s3
bucket_store:
bucket_index:
enabled: true
sync_dir: /data/tsdb-sync
s3:
access_key_id: ***
bucket_name: ***
endpoint: ***
secret_access_key: ***
tsdb:
dir: /data/tsdb
distributor:
pool:
health_check_ingesters: true
shard_by_all_labels: true
frontend:
log_queries_longer_than: 10s
ingester:
lifecycler:
final_sleep: 30s
join_after: 10s
num_tokens: 512
observe_period: 10s
ring:
kvstore:
store: memberlist
replication_factor: 3
ingester_client:
grpc_client_config:
max_recv_msg_size: 10485760
max_send_msg_size: 10485760
limits:
enforce_metric_name: true
max_query_lookback: 0s
reject_old_samples: true
reject_old_samples_max_age: 168h
memberlist:
bind_port: 7946
join_members:

• 'cortex-memberlist'
  querier:
  active_query_tracker_dir: /data/active-query-tracker
  store_gateway_addresses: |-
  dns+cortex-store-gateway-headless:9095
  query_range:
  align_queries_with_step: true
  cache_results: true
  results_cache:
  cache:
  memcached:
  expiration: 1h
  memcached_client:
  timeout: 1s
  split_queries_by_interval: 24h
  ruler:
  enable_alertmanager_discovery: false
  enable_api: true
  storage: {}
  runtime_config:
  file: /etc/cortex-runtime-config/runtime_config.yaml
  server:
  grpc_listen_port: 9095
  grpc_server_max_concurrent_streams: 10000
  grpc_server_max_recv_msg_size: 10485760
  grpc_server_max_send_msg_size: 10485760
  http_listen_port: 8080
  storage:
  engine: blocks
  index_queries_cache_config:
  memcached:
  expiration: 1h
  memcached_client:
  timeout: 1s
  store_gateway:

Any suggestions?