StructArray or Arc<StructArray>

[potter420]

Hi,

I'm doing some kind of streaming applications in Rust at the moment to do some batch streaming into Redshift. Basically we have a kafka consumer, processors which will turn events into StructArray, and async writer to write them to S3, then call load manager API to queue the file in order to be upload to Redshift. All queue using tokio::sync::mpsc or async_channel depend on the situation

From the processors to async writers, should we parse the StructArray into the queue or Arc<StructArray>, since I saw that the StructArray is basically a collection of Vec<Field> and Arc<dyn Array>.

If my understanding are correct, both StructArray and Arc<StructArray> should just move a collection of array pointers across the channels rather than the actual array data themselves, the struct array is actually . Is it correct and which should I choose?

Thanks

[jorgecarleitao]

Cool scenario :)

I would go for Arc<StructArray>.

They are both Send+Sync. The Arc offers two things:

• allows the StructArray to be a trait object Array.
• cheap clone

Since you mentioned a streaming scenario where sometimes the size of the metadata matters, and there is a small overhead in cloning a StructArray due to the clone of the metadata Vec<Field>, I would go for Arc<StructArray>.