Bug: Old Data Unreadable After Migrating Mimir to Kubernetes with Helm and Upgrading to v2.14

[Zqsdoo]

What is the bug?

Issue Description
We encountered an issue after migrating two Mimir instances from ubuntu virtual machines to a distributed Mimir setup using Helm charts on a Kubernetes cluster.

Details

We kept the same S3 bucket to store all logs and metrics during the migration.
The Kubernetes-based Mimir setup can correctly write and read data it creates in the bucket.
However, our problem is that old data (from before the migration) appears to be inaccessible. The data still exists in the S3 bucket but cannot be read by Mimir.

Additional Context

We are aware that sometimes it takes time for old data to be indexed or processed, but the migration was completed three days ago, and we still cannot see any of the old logs.
We upgraded Mimir from version 2.12 to version 2.14 (latest) during the migration.
There are no error logs on any of the Mimir pods.

How to reproduce it?

Set up a Mimir instance (version 2.12) on a virtual machine.
Configure the instance to store logs and metrics in an S3 bucket.
Migrate the setup to a distributed Mimir using Helm charts on a Kubernetes cluster.
Upgrade Mimir from version 2.12 to version 2.14 during the migration.
Ensure that the same S3 bucket is used to store logs and metrics after the migration.
Observe that new data is being written and read correctly from the S3 bucket, but old data (prior to the migration) is not accessible.

What did you think would happen?

After migrating Mimir from version 2.12 to 2.14 and moving to a Kubernetes-based distributed setup, we expected the following:
The S3 bucket would continue to be used for storing logs and metrics
Mimir would be able to both read and write data correctly from the S3 bucket, including old data (prior to the migration)
Both new and old logs and metrics would be accessible without issues

What was your environment?

Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

mimir installed from : Dec 22 2023 mimir-2.11.0_amd64.deb

Any additional context to share?

No response