How to run compactor so stored blocks are in FIFO model?

[zulh-civo]

I'm running Cortex to store multi-tenant metrics and I configured a S3-compatible storage system/back-end as long term storage so the compactor can compact these metrics and make them available for long retention or when I need to go -2h (or longer) back in time metrics.

The storage bucket size/limit is 500GB and I don't want to make it any bigger. Because of this limitation, I'm seeing a lot of these quota errors from that S3/storage system:

cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.578615077Z caller=blocks_cleaner.go:144 component=cleaner msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867598335Z caller=blocks_cleaner.go:301 component=cleaner org_id=ghi-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867650464Z caller=blocks_cleaner.go:301 component=cleaner org_id=jkl-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867664582Z caller=blocks_cleaner.go:301 component=cleaner org_id=abc-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867684256Z caller=blocks_cleaner.go:301 component=cleaner org_id=def-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867694777Z caller=blocks_cleaner.go:301 component=cleaner org_id=mno-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=info ts=2023-02-17T04:14:07.867716709Z caller=blocks_cleaner.go:301 component=cleaner org_id=pqr-cluster msg="started blocks cleanup and maintenance"
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.369969281Z caller=updater.go:108 component=cleaner org_id=def-cluster msg="skipped partial block when updating bucket index" block=01GR8JEBZQ4Y1V172470CGHDWB
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.450240834Z caller=updater.go:108 component=cleaner org_id=def-cluster msg="skipped partial block when updating bucket index" block=01GR9AXEW184445AD0Z4HYZB01
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.530288868Z caller=updater.go:200 component=cleaner org_id=ghi-cluster msg="skipped missing block deletion mark when updating bucket index" block=01GQ1573WZM74KPHK96ZMGE4JK
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.539810401Z caller=blocks_cleaner.go:304 component=cleaner org_id=ghi-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.60621109Z caller=blocks_cleaner.go:304 component=cleaner org_id=jkl-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.722806106Z caller=blocks_cleaner.go:304 component=cleaner org_id=mno-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.752128461Z caller=blocks_cleaner.go:304 component=cleaner org_id=pqr-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.895118237Z caller=blocks_cleaner.go:304 component=cleaner org_id=def-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.911554689Z caller=updater.go:108 component=cleaner org_id=abc-cluster msg="skipped partial block when updating bucket index" block=01GQQMATH36K9NHFXPTQS6K1H9
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:08.995127852Z caller=updater.go:108 component=cleaner org_id=abc-cluster msg="skipped partial block when updating bucket index" block=01GR8JE1S7K6M441KYA8GNYM2V
cortex-compactor-0 compactor level=warn ts=2023-02-17T04:14:09.105537978Z caller=blocks_cleaner.go:304 component=cleaner org_id=abc-cluster msg="failed blocks cleanup and maintenance" err="upload bucket index: upload s3 object: Error response code QuotaExceeded."
cortex-compactor-0 compactor level=error ts=2023-02-17T04:14:09.105658738Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="6 errors: failed to delete blocks for user: ghi-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded.; failed to delete blocks for user: jkl-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded.; failed to delete blocks for user: mno-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded.; failed to delete blocks for user: pqr-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded.; failed to delete blocks for user: def-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded.; failed to delete blocks for user: abc-cluster: upload bucket index: upload s3 object: Error response code QuotaExceeded."

My question here is, how do I configure Cortex so the metrics/blocks stored in the S3/storage system is in FIFO model e.g. in the event compactor wants to upload 10GB of latest/newest blocks, it will replace the oldest 10GB of blocks in the same bucket? Upsizing the bucket size/limit isn't an option for me and I'm looking for solutions to get rid of these quota errors.

P/S: FWIW, I have another question opened which is related to compactor too but I think they both are not related to each other.

[friedrichg]

The compactor will delete blocks based on the retention period configured.
The default configuration does not delete any blocks. Which is not ideal for any prod environment.

# Delete blocks containing samples older than the specified retention period. 0
# to disable.
# CLI flag: -compactor.blocks-retention-period
[compactor_blocks_retention_period: <duration> | default = 0s]

If a retention period in the compactor is configured, the usage in object storage should not greatly differentiate from day to day with same actives series ingestion after the retention period has been passed. The reasonable minimum is probably '1d' (one day).

---

The idea of being able to delete based on size would require the compactor to actually know the current usage. Which is not currently the case. Making that alone is a challenge when it is considered that a tenant could be shared among multiple compactors. Add to that making it support any s3 compatible storage.
Still that won't help you entirely because you are looking at a limit that is shared among all tenants.
Such implementation would require a proposal

But supposing we could do all, it's going be very surprising for a tenant, that has not changed its cortex ingestion usage, suddenly discover they cannot see older metrics they could see before due to a limit reached by other's tenant usage change. It breaks multi-tenancy.

[zulh-civo]

Understood. Thank you for the clarification, @friedrichg. Does it mean I can set '1d' for tenant A and '7d' for tenant B in the current version of Cortex?

[friedrichg]

yes, create a runtime config and put a different compactor_blocks_retention_period for each tenant