How to go from original data in Vec<T> to arrow (and parquet)?

[nbigaouette]

I'm trying to wrap my head around Parquet and Arrow. What I'd like to have is a way to serialize Rust data to a binary on disk format. I though parquet could be this binary format but I'm having trouble trying to properly do this serialization.

My Rust data can be described as a group of two (or more) vectors. For example:

struct MyData<'a> {
    names: Vec<String>,
    values: Vec<Vec<f32>>,

Most of the time, I have a single values vector, so I can simplify to:

struct MyData {
    names: Vec<String>,
    values: Vec<f32>,

And since memory is managed elsewhere, the following would be ideal to serialize without copy:

struct MyData<'a> {
    names: &'a [String],
    values:  &'a [f32],

Now everywhere I look it seems that the serialization expects something like a "row" instead of a "column". This means I need to create a struct like that:

struct Row {
    name: String,
    value: f32,
}

Since I already have my Vec in memory, do I really need to "chunk" some rows together before saving to parquet? Can't I just point to the already allocated memory and say "here are two slices, save them to disk"?

[adsick]

Hi, I have the same question, particularly about Vec<Vec<T>>. I think it is really not trivial to construct a ListArray from an existing Vec, there are some implementations for primitive arrays like UInt32Array::from_vec<vec![1, 2, 3]> but I don't see any example on how to construct (or better convert) nested types like ListArray