The bucket index is too old

[shindebshekhar]

Describe the bug

We are getting above error while querying data from Grafana. We are using backed as s3 storage and everything at backend storage seems to be working fine.

Compactor logs shows below error -

ts=2024-03-14T02:18:10.338216072Z caller=blocks_cleaner.go:208 level=error component=cleaner run_id=1710382678 task=clean_up_users msg="failed to run blocks cleanup and maintenance" err="3 errors: failed to delete blocks for user: clxxx-m: get file: 01HMSXTRQ31RRZWZ695ZE3J6NP/deletion-mark.json: The server is temporarily unavailable. Please try again later.; failed to delete blocks for user: clxxx-m: get file: 01HMT06H4MKJQ7PKZ1Z6A65906/deletion-mark.json: The server is temporarily unavailable. Please try again later.; failed to delete blocks for user: clxxx: get file: 01HMSZRNS3DNTKPYAWSA6BCR1C/deletion-mark.json: The server is temporarily unavailable. Please try again later."

[charleskorn]

The The server is temporarily unavailable. Please try again later. error is coming from your object storage - are you using S3 provided by AWS, or a S3-compatible object storage?

[shindebshekhar]

Hello, We are using S3-compatible object storage. Connectivity is fine and we are able to access backend storage as well. The bucket index is not getting updated.
Is there any temporary workaround to solve this issue. We are unable to see dat in grafana?

[charleskorn]

Which object storage are you using?

Unfortunately there is no workaround for this problem: if your object storage is not working correctly, then Mimir can't work around that.

It seems there is an issue either with the object storage or with how Mimir is configured to connect to your object storage. I'd start by investigating why the object storage is returning the The server is temporarily unavailable. Please try again later. error. There may be more information available in the object storage logs, for example.

[shindebshekhar]

We are using cloudian s3 object storage. Data is still getting written correctly to backend s3 storage but only bucket index file is not getting updated. Please see below output.

2024-03-14 05:03:42 628 Bytes clxorgid124/01HRXNG1REXYMAK4CJ8JRKWZMY/meta.json
2024-03-14 05:04:34 5.5 MiB clxorgid124/01HRXNH1KTC954V5A5A4VDG9S7/chunks/000001
2024-03-14 05:04:35 3.1 MiB clxorgid124/01HRXNH1KTC954V5A5A4VDG9S7/index
2024-03-14 05:04:35 628 Bytes clxorgid124/01HRXNH1KTC954V5A5A4VDG7S7/meta.json
2024-03-05 23:51:16 3.8 KiB clxorgid124/bucket-index.json.gz

[charleskorn]

If the object storage is returning errors like those, then the compactor won't update the bucket index - it needs those read operations to succeed (or return a not found error, if the objects don't exist) before it writes an updated bucket index.

I'd suggest contacting the object storage provider to understand why it is returning those errors.

[shindebshekhar]

We were not able able to find the root cause of the issue. Storage end we checked all the logs along with the vendor and they say that all the other blocks are written then they are not sure why only bucket-index file is not been updated. They had asked to check at application side.

As of now we have disabled the bucket-index so that querier reads the data by performing ls operation on bucket and not route through bucket-index file from where it gets the stale timestamp

But still the root cause is not known...

[charleskorn]

The issue isn't writing the bucket index file: the issue is reading the files listed in the error message, such as 01HMSXTRQ31RRZWZ695ZE3J6NP/deletion-mark.json.

As part of updating the bucket index, the compactor needs to first read those deletion-mark.josn files (or get a not-found error) and will then write a new bucket index.