Unable to successfully connect to configured object storage - issue with bucket connectivity for ap-south-2 region, URGENT!!!!!!!!!!

[Vaibhav-1995]

Issue
Trying to deploy mimir helm chart in EKS cluster which is in ap-south-2 region - also created s3 bucket for block storage in ap-south-2 region for the same and configure bucket for block storage and updated the s3 configuration but pods showing below error log -

ts=2023-09-01T14:05:41.314666075Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: The ap-south-2 location constraint is incompatible for the region specific endpoint this request was sent to."

Steps to reproduce the behavior:

• create s3 bucket in ap-south-2 region and try to configure it
• use IRSA with s3 full access policy for giving permissions
• mimir version v2.9.0
• infrastructure: EKS

Expected behavior
Bucket should get configured in ap-south-2 region as same setup is working fine for the ap-south-1 region

[aknuds1]

Converting to a discussion as it looks more like a configuration issue than a Mimir bug.

[Vaibhav-1995]

@aknuds1
Thanks for your reply on this issue.

[charleskorn]

Please don't close and re-open issues or create new issues for the same issue @Vaibhav-1995.

Can you please confirm the S3 buckets your configuration refers to exist? If they do exist, what region have they been created in? I suspect the buckets have been created in the ap-south-1 region, so connecting to them through the ap-south-2 S3 endpoint fails.

[Vaibhav-1995]

Hi @charleskorn
Thanks for your reply

Yes @charleskorn - I was not suppose to close and re-open issue for the same one but was not getting any reply since couple of months so had to do it to get it at top of list - I apologize for it

[Vaibhav-1995]

Can you please confirm the S3 buckets your configuration refers to exist? If they do exist, what region have they been created in? I suspect the buckets have been created in the ap-south-1 region, so connecting to them through the ap-south-2 S3 endpoint fails.

@charleskorn bucket has been created in ap-south-2 region only

[charleskorn]

Could you please try modifying your config to specify the region directly on the blocks storage config? Something like this:

    blocks_storage:
      backend: s3
      bucket_store:
        max_chunk_pool_bytes: 12884901888
        sync_dir: /data/tsdb-sync
      s3:
        bucket_name: as2-qa-ss-mimir-blocks
        endpoint: s3.ap-south-2.amazonaws.com
        region: ap-south-2
      tsdb:
        dir: /data/tsdb
    common:
      storage:
        backend: s3
        s3:
          endpoint: s3.ap-south-2.amazonaws.com
          region: ap-south-2

[Vaibhav-1995]

Hi

We tried as suggested but still facing same issue.

apiVersion: v1
data:
  mimir.yaml: |

    activity_tracker:
      filepath: /active-query-tracker/activity.log
    alertmanager:
      data_dir: /data
      enable_api: true
      external_url: /alertmanager
      fallback_config_file: xxxx
    blocks_storage:
      backend: s3
      bucket_store:
        max_chunk_pool_bytes: xxxxxxxx
        sync_dir: /data/tsdb-sync
      s3:
        bucket_name: xxxxxxxx
        endpoint: s3.ap-south-2.amazonaws.com
        region: ap-south-2
      tsdb:
        dir: /data/tsdb
    common:
      storage:
        backend: s3
        s3:
          endpoint: s3.ap-south-2.amazonaws.com
          region: ap-south-2
    compactor:
      compaction_interval: xxx
      data_dir: xxx
      deletion_delay: xxx
      max_closing_blocks_concurrency: xxx
      max_opening_blocks_concurrency: xxx
      sharding_ring:
        wait_stability_min_duration: xxx
      symbols_flushers_concurrency: xxx
    frontend:
      parallelize_shardable_queries: true
      scheduler_address: xxxxxx-query-scheduler-headless.xxx.svc:9095
    frontend_worker:
      grpc_client_config:
        max_send_msg_size: xxxxx
      scheduler_address: xxxxxx-query-scheduler-headless.xxx.svc:9095
    ingester:
      ring:
        final_sleep: xxx
        num_tokens: xxx
        tokens_file_path: xxx
        unregister_on_shutdown: false
        zone_awareness_enabled: true
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: xxxxx
        max_send_msg_size: xxxxx
    limits:
      ingestion_rate: xxx
      max_cache_freshness: xxx
      max_global_series_per_user: xxx
      max_label_names_per_series: xxx
      max_query_parallelism: xxx
      max_total_query_length: xxx
    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - dns+xxx-gossip-ring.xxx.svc.cluster.local:7946
    multitenancy_enabled: false
    querier:
      max_concurrent: xx
    query_scheduler:
      max_outstanding_requests_per_tenant: xxx
    ruler:
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.xxx-alertmanager-headless.xxx.svc.cluster.local/alertmanager
      enable_api: true
      evaluation_interval: xx
      rule_path: xxx
    runtime_config:
      file: xxx
    server:
      grpc_server_max_concurrent_streams: xxx
      grpc_server_max_connection_age: xxx
      grpc_server_max_connection_age_grace: xxx
      grpc_server_max_connection_idle: xxx
    store_gateway:
      sharding_ring:
        kvstore:
          prefix: multi-zone/
        tokens_file_path: xxx
        wait_stability_min_duration: xxx
        zone_awareness_enabled: true
    usage_stats:
      installation_mode: helm
kind: ConfigMap

[charleskorn]

Could you please try with Mimir v2.10? Looks like this was an issue that was fixed in minio/minio-go#1781, which is included in Mimir v2.10 onwards.

[Vaibhav-1995]

Ok Thanks!
I will try with Mimir v2.10 and let you know

[get2aakash]

Did it work?