Automatic Chunking

[Material-Scientist]

Hi,

I recently had a problem where I got indexCacheBuf corrupt record state: dlen 46134798 slen 21 index 0 rl 46134820 lbuf 46134820 when trying to upload a 40MB file to the KV store, despite setting max_payload to 64MB in the NATS server config.

But maybe instead of increasing the maximum msg size, it would be a better idea to add a way for NATS to automatically chunk the object into 8MB fragments, and only store the references in the KV store.

AFAIK, that's how NATS object store works. So, why not just use the same automatic chunking everywhere (even pub/sub)?

It would push the burden of checking file-sizes and chunking from the user to the messaging system.

Thoughts?

[ripienaar]

That’s what the object store is for :)

Nats isn’t optimised for big payloads 2 or 3MB max is a good rule

[Material-Scientist]

I know. I mentioned it in the post.

I.e. KV or pub/sub could add the NATS object store interface to automatically chunk and store only the references & automatically fetch the data, rather than let the user deal with that.

[Jarema]

pub/sub does not persist any data, is at-most-once,
Using it with JetStream layer is not a way to go.

On the other hand, chunking messages for KV would bring a lot of challenges to KV API.
Maybe we can revisit some of those ideas in the future when batched publishes are implemented.

Meanwhile, you can use request many pattern from our extension libraries

It does not automatically chunk messages, but provides API for requesting chunked responses and strategies to handle chunks termination.

[jgaskins]

why not just use the same automatic chunking everywhere (even pub/sub)?

It all comes down to optimizing for large vs optimizing for small. Object storage (not just in NATS, but cloud object storage like S3, as well) is designed to accommodate arbitrarily sized objects, but that design has significant cost. Getting a value from the NATS object store is much heavier than getting a value from KV which, in turn, is heavier than getting a value via pub/sub.

Getting an object from the object store involves the following overhead:

• sending a request for object metadata, which is functionally equivalent to fetching a value from KV
• creating a temporary subscription for the object's contents
• sending a second request for the object's contents
• running the entire response through a pipe (implemented in terms of file descriptors, with 3x the I/O overhead per chunk)
• some kind of signal that the value is complete

Getting a value from KV, on the other hand, involves only a single NATS request, equivalent to just the first of those 5 bullet points. Comparatively, the overhead is quite low, so you can perform far more KV operations than Object operations in NATS on the same hardware (both client and server).

If it helps, you can think of the constraint as a guarantee. You can count on values fetched from KV fitting into RAM. If your KV entries could be arbitrary sizes, this guarantee goes away and you will have to account for the possibility of massive KV entries and make the return values streams that you can read from. However, if you get that far with it, you've reinvented the object store.