Thanos store request fails trying to access deleted blocks

[olivet-r]

Hello,

We are using Thanos and S3 storage for long time retention of the data. A compactor is running for downsampling and retention purpose. Everything works fine, but after a while (~12h or more) we are encountering errors on request trying to access deleted block of data.

Errors comes with this message on store logs :

level=debug name=info ts=2023-06-14T06:37:54.982024592Z caller=bucket.go:1290 msg="stats query processed" request="min_time:1686548160000 max_time:1686724560000 matchers:<type:RE name:\"cluster_name\" value:\"*****\" > matchers:<name:\"__name__\" value:\"*******\" > max_resolution_window:3600000 aggregates:COUNT aggregates:SUM " stats="&{blocksQueried:32 postingsTou 
......
onSum:132.017705ms GetAllDuration:1.490377ms mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Unknown desc = receive series from 01H2T6K8X7S6TXNJ9MD786ZDXP: load chunks: get range reader: The specified key does not exist."

Referenced block 01H2T6K8X7S6TXNJ9MD786ZDXP is no longer existing in the bucket

But it is still referenced on the local store cache :
/store/meta-syncer/01H2T6K8X7S6TXNJ9MD786ZDXP sitll present with meta.json file

From compator logs, the block is deleted :

level=info ts=2023-06-13T21:48:52.642475752Z caller=block.go:203 group="0@{prometheus=\"****\"}" groupKey=0@16653103604663073225 msg="block has been marked for deletion" block=01H2T6K8X7S6TXNJ9MD786ZDXP
...
level=info ts=2023-06-14T00:50:33.300713635Z caller=blocks_cleaner.go:54 msg="deleted block marked for deletion" block=01H2T6K8X7S6TXNJ9MD786ZDXP

The main store settings includes the following parameters :
--ignore-deletion-marks-delay=15m
--sync-block-duration=30m \

type: IN-MEMORY
chunk_object_attrs_ttl: 30m
chunk_subrange_ttl: 30m
metafile_exists_ttl: 30m
metafile_doesnt_exist_ttl: 15m
metafile_content_ttl: 30m

In the store log file some messages "dropped outdated block" are present, but after a while there is no more message regarding dropped block in the log.
5 hours has elapsed between the deletion of the block by the compactor and the error encountered in the store query.
Any idea on which parameter could help in clearing more frequently the outdated block in the cache?
How can I check if the Block Synchronization is still running ?

Thanos version : 0.30.1

[olivet-r]

After upgrade to latest Thanos version, issue has gone