Successfully uploaded TSDB block data via mimirtool, but no data on Grafana dashboard

[nhyphamnt]

Hi everyone,
I am currently having issue with the mimir backfill data. I successfully uploaded all the tsdb block to mimir, and can confirmed it in our Object Storage (nfs server via minio). But data was not shown in our Grafana metrics explore. I tried to restart the memory cache pod and also waited almost a day following this post (#3735) but it did not help. Could you please advise if I am doing wrong in some points. I put details in the comment. Many thanks.
Condition:

• Grafana, Prometheus, Mimir are hosted on a Kubernetes cluster
• Using minio Object storage to have mimir data in our nfs server

Steps done:

1. Created data as Openmetric format

data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="a"} 170364955325 1726876808
data_created{project="A",customer="b"} 170364955325 1726876808

2. Using promtool to create tsdb data block

Data$ promtool tsdb list tsdb_output                                                                                                                                              
BLOCK ULID                  MIN TIME       MAX TIME       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE                                                                                                 
01J8MGZ11ZB190S52MC0BJ22FP  1726876808000  1726876808001  1ms          3            3            3            904                                                                                                  
01J8MGZ16KT48W8GMH07QDDTSC  1726876808000  1726876808001  1ms          1            1            1            684 

3. Uploaded this tsdb block to mimir using mimir tool. Got the successful message

./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  
INFO[0000] Backfilling                                   blocks=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  user=anonymous
INFO[0000] making request to start block upload          block=01J8MGZ11ZB190S52MC0BJ22FP  file=meta.json path=./tsdb_output1/01J8Q12KH2HCC58DKP2YEKAYQ7
INFO[0000] uploading block file                          block=01J8Q12KH2HCC58DKP2YEKAYQ7 file=index path=./tsdb_output1/01J8Q12KH2HCC58DKP2YEKAYQ7 size=904
INFO[0000] uploading block file                          block=01J8Q12KH2HCC58DKP2YEKAYQ7 file=chunks/000001 path=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  size=904
INFO[0020] block uploaded successfully                   block=01J8MGZ11ZB190S52MC0BJ22FP  path=./tsdb_output1/01J8MGZ11ZB190S52MC0BJ22FP  
INFO[0020] finished uploading blocks                     already_exists=0 failed=0 succeeded=1

After uploading, the folder has this data hiar:

  rwxr-xr-x 2 root root 29 Sep 26 23:44 uploading-meta.json
drwxr-xr-x 2 root root 29 Sep 26 23:44 index
drwxr-xr-x 3 root root 28 Sep 26 23:44 chunks
drwxr-xr-x 2 root root 29 Sep 26 23:44 validation.json
drwxr-xr-x 2 root root 29 Sep 26 23:44 meta.json

Currently the status of these blocks was changed to delete-mark.

drwxr-xr-x 2 root root 29 Sep 25 22:08 uploading-meta.json
drwxr-xr-x 2 root root 29 Sep 25 22:08 index
drwxr-xr-x 3 root root 28 Sep 25 22:08 chunks
drwxr-xr-x 2 root root 29 Sep 25 22:08 validation.json
drwxr-xr-x 2 root root 29 Sep 25 22:08 meta.json
drwxr-xr-x 2 root root 29 Sep 25 22:56 deletion-mark.json

I am not sure what I did wrong here, data did not appear on Grafana metric explore. Please kindly help. Many thanks.

[pstibrany]

Hello. There are two possible reasons why block may be marked for deletion:

• either it fell out of retention window
• or block was merged (compacted) with another block from the same time range

Based on your

promtool tsdb list tsdb_output
BLOCK ULID                  MIN TIME       MAX TIME       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01J8MGZ11ZB190S52MC0BJ22FP  1726876808000  1726876808001  1ms          3            3            3            904
01J8MGZ16KT48W8GMH07QDDTSC  1726876808000  1726876808001  1ms          1            1            1            684 

and mimirtool backfill output, block 01J8MGZ11ZB190S52MC0BJ22FP was uploaded. This block has 3 series, each with exactly 1 sample at Sat Sep 21 00:00:08 UTC 2024. Did you query the right time range to see this one sample?

[nhyphamnt]

Many thanks @pstibrany for your prompt response. The idea is every day we will upload the data usage then we want to sum the data for every month. That's why we want to upload it just once a day, as specific time. This will be reflected using textfile_collector once a day later. Currently we want to put the historic data first.
Thus, we have data for just 1 sample for one series at the same time range, with different labels. Is this ok to have data like that?

# HELP dmf_data_volume_created My metric.
# TYPE dmf_data_volume_created counter
data_created{project="A",vg="v1_8",pool="a"} 1477410031051 1725321608
data_created{project="A",vg="v_8",pool="a"} 519875 1725321608
data_created{project="A",vg="v2_8",pool="b"} 519875 1725321608
data_created{project="A",vg="v_8",pool="b"} 1477410031051 1725321608
data_created{project="A",vg="v_8",pool="c"} 1477410031051 1725321608
data_created{project="C",vg="v3_8",pool="a"} 2073645102173 1725321608
data_created{project="C",vg="vc_8",pool="c"} 2073645102173 1725321608
data_created{project="R",vg="v_8",pool="a"} 9546910033 1725321608
data_created{project="R",vg="v_8",pool="b"} 9546910033 1725321608
data_created{project="R",vg="v_6",pool="c"} 9546910033 1725321608
# EOF

Thank you again and I am waiting for your advice.

[pstibrany]

Hello. It's fine to have data like this. As mentioned earlier your block was uploaded correctly, but you have trouble querying it and block was also marked for deletion.

To figure out why block is marked for deletion, I'd try to find compactor logs for given block ID, to see which of the above mentioned cases (retention or compaction) made compactor to mark block for deletion.

For querying, what is your query and time range?

[nhyphamnt]

Thank you @pstibrany and I apologise for the delayed response. I am currently on vacation and will be back in 10 days. I will collect logs and posted here as soon as I have access again to my work computer.

[nhyphamnt]

Many thanks @pstibrany for your comment.
I tried again to upload the new blocks and below is the compactor logs

ts=2024-10-22T11:20:41.772776818Z caller=block_upload.go:105 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W102WK9TW1Z1XWSNDB4F operation="start block upload" request_id=1800c26fa9ce5528 msg="started block upload"
ts=2024-10-22T11:20:41.823291129Z caller=block_upload.go:172 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W102WK9TW1Z1XWSNDB4F operation="complete block upload" request_id=1800c26fad1eb07b msg="validation process started"
ts=2024-10-22T11:20:41.872229979Z caller=block_upload.go:389 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W102WK9TW1Z1XWSNDB4F operation="complete block upload" request_id=1800c26fad1eb07b msg="successfully completed block upload"
ts=2024-10-22T11:21:02.215297323Z caller=block_upload.go:105 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W125C11KTFZ734WVFSSS operation="start block upload" request_id=1800c2746c40f96f msg="started block upload"
ts=2024-10-22T11:21:02.276994206Z caller=block_upload.go:172 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W125C11KTFZ734WVFSSS operation="complete block upload" request_id=1800c2746ff7cee0 msg="validation process started"
ts=2024-10-22T11:21:02.325706254Z caller=block_upload.go:389 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W125C11KTFZ734WVFSSS operation="complete block upload" request_id=1800c2746ff7cee0 msg="successfully completed block upload"
ts=2024-10-22T11:21:22.58384785Z caller=block_upload.go:105 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W14E7A4BR54G6YDG8T0J operation="start block upload" request_id=1800c2792a6aad42 msg="started block upload"
ts=2024-10-22T11:21:22.635969024Z caller=block_upload.go:172 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W14E7A4BR54G6YDG8T0J operation="complete block upload" request_id=1800c2792daec79c msg="validation process started"
ts=2024-10-22T11:21:22.680930208Z caller=block_upload.go:389 level=info component=compactor user=anonymous feature="block upload" block=01JAS6W14E7A4BR54G6YDG8T0J operation="complete block upload" request_id=1800c2792daec79c msg="successfully completed block upload"
ts=2024-10-22T11:22:51.834362481Z caller=blocks_cleaner.go:207 level=info component=cleaner run_id=1729596171 task=clean_up_users msg="started blocks cleanup and maintenance"
ts=2024-10

I will keep an eye on the logs to check when it is marked as deletion.
In addition, I did not set the retention window in the helm value file( e.g compactor_blocks_retention_period value was not set). Thus, by default, I think it has no retention period.

I did not use any query yet, I used metric explore feature in Grafana dashboard to look for the data_created metrics, it returned nothing.

Thank you.

[nhyphamnt]

Hi @pstibrany
I found the reason why these blocks were marked as deleted. They were compacted into another block:

ts=2024-10-22T11:54:21.052823513Z caller=compact.go:1034 level=info component=compactor msg="Found overlapping blocks during compaction"
ts=2024-10-22T11:54:21.12694065Z caller=compact.go:554 level=info component=compactor msg="compact blocks" count=3 mint=1727654408000 maxt=1727654448001 ulid=01JAT17YF628D8PC2EYCB9X7VG sources="[01JAS6W102WK9TW1Z1XWSNDB4F 01JAS6W14E7A4BR54G6YDG8T0J 01JAS6W125C11KTFZ734WVFSSS]" duration=96.62165ms shard=1_of_1

I am not sure if the message: "Found overlapping blocks during compaction" indicated something weird. Please kindly advise.
I attached the new block data for your reference.
It's almost 15 hours from the time I uploaded tsdb data in. I still cannot see the metrics via Grafana Explore feature. Please help. Thank you.

01JAT17YF628D8PC2EYCB9X7VG.tar.gz