can we use rust engine for z order

[djouallah]

I am getting disk full after I tried to zorder a delta table, around 400 M, datasize is around 40 MB
is there plan to use rust engine instead ?

[ion-elgreco]

@djouallah all operations use solely Rust, except the write operation which we can be done with pyarrow or rust engine.

Are you sure you have diskspace left?

[djouallah]

my RAM is 32 GB, so I presumed it is more than enough to sort the file ?

[ion-elgreco]

The error indicates the issue is the storage, not the memory

[djouallah]

I understand, I meant, it should not touch the disk at all as it is a small file

[ion-elgreco]

I can only assume that disk space is required for optionally spilling to disk so you need free space

[djouallah]

anyway as a workaround, I am using compact for now, I suspect maybe it is spark runtime issue , thanks

[djouallah]

is there a way to define a temp directory where i can spill ?

[ion-elgreco]

I don't know..

@rtyler @roeap @wjones127 any clue on this?

[wjones127]

my RAM is 32 GB, so I presumed it is more than enough to sort the file ?

We don't currently detect the RAM on the system. We use a limit specified in the settings max_spill_size. The documentation for this is actually wrong right now; it says it's the maximum that will spill to disk, but it's actually the threshold for memory in the pool at which point it will start to spill to disk.

https://github.com/delta-io/delta-rs/blob/815c965a1c67aef3e26555191ff149a73b7a1a4c/python/deltalake/table.py#L1840C9-L1840C23

You could try increasing that to 30GB.

is there a way to define a temp directory where i can spill ?

I don't think there's any way to control where it spills. AFAIK it just uses the system default to choose a temp directory.