[Understanding CLP IR] How can Fluentbit generated IRs be turned to Archives

[shadowshot-x]

Hi CLP Team.

I have been exploring CLP and trying to understand more about the technology. I am able to generate zstd compressed IRs and upload to S3.

The piece which feels missing is how to convert these IRs to archives so I can Query them using Presto.

I am using CLP 0.7.0 and latest Fluentbit clp plugin

Command: ./clp-s c /mnt/logs/archives/ --timestamp-key '@time' /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst
Error

2026-01-08T16:54:54.040+00:00 [error] Encountered curl error while ingesting /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst - Code: 3 - Message: 
2026-01-08T16:54:54.041+00:00 [error] Could not deduce content type for input /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst
2026-01-08T16:54:54.050+00:00 [error] Encountered error while parsing input.

Also tried decompressing the ZST
Command: ./clp-s c /mnt/logs/archives/ --timestamp-key '@time' /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir
Error

2026-01-08T16:56:55.684+00:00 [error] Encountered error while creating kv-ir deserializer: (93) - Protocol not supported
2026-01-08T16:56:55.694+00:00 [error] Encountered error while parsing input.

If we can convert fluentbit generated IRs to archives, that will be a huge win for the platform!

[gibber9809]

Hey @shadowshot-x,

I think the issue here is that the fluent-bit plugin is generating an older version of our IR designed for unstructured log-text, whereas clp-s currently only supports ingesting an updated version of the IR format that we call KV-IR.

We do have a version of our fluentbit plugin that emits KV-IR directly, so if switching which fluentbit plugin you use is an option that should get things working for you.

That said, I agree that it would be good to support ingesting the older IR format into clp-s directly. It should be feasible, but we probably won't be able to get it into the release we're trying to finish up right now. If this is a blocker for you though, let us know and we can definitely try to prioritize it!

[shadowshot-x]

Hi @gibber9809 , Thank you. I can try with the V2 version for Fluentbit that you suggested.

Moving forward, KV-IR new version will be good for us so its not exactly a Blocker.

I do see the Fluentbit new version is in Beta (as per the branch name). Let me test it out. I am happy to contribute too if any open issues are there!

[davemarco]

Hi @shadowshot-x,

Quick update: we’ve added documentation for the KV-IR branch. Note that this was a beta built for a specific user. After some internal discussion, we’ll work on adding KV-IR support to the main branch, which is our fault-tolerant branch. I’ll follow up once that work is complete — likely in February.