Chunk Size for Async

[dmillerksu]

Hello all,

Is there an ideal method or best practice for determining best chunk size for writing data using the async functions? I had heard from someone at a conference that 30k was an ideal chunk size for TM1. I'm guessing it all depends on the cube's size and the complexity of the rules.

I've been using that 30k chunk size for everything so far. The individual async threads usually complete around 0 to 0.2 seconds. Is it better to try to evenly spread the chunks across the cores to avoid any threads waiting (ie. total records / cores)? Has anyone produced a script for testing out ranges of chunk sizes?

Thanks,
Danny

[MariusWirtz]

Hi @dmillerksu,

thanks for opening this discussion.
We had a conversation about the ideal chunk size when @macsir first introduced the feature
#543
But we never really concluded this if I remember it correctly.

Theoretically 32'000 or multiples of 32'000 should work well as a chunk size.
That's because ultimately TM1py executes unbound TI processes with a maximum of 32'000 CellPut statements in it (16k in the prolog and 16k in the epilog).

So if you need to write 640'000 cells in total, a chunk size of 32'000 would be reasonable.
TM1py would then execute 20 unbound processes (irrespective of how many threads are used) with each the maximum number of statements in it.

If you use a chunk size of less than 32'000 cells, TM1py will execute unbound TI processes with less than the maximum number of possible statements. Theoretically, that should be less efficient.
So in the above example, let's say we would go with a chunk size of 10'000, TM1py would then execute 64 unbound processes with each 10'000 cellput statements.
That should be slightly slower.

If we use a chunk size of 64'000, TM1py breaks the 640'000 updates into 10 chunks of 64'000 cells each.
It then spawns n threads (n being the max_workers that you pass to the function) and works through 10 operations. Each operation would then execute 2 unbound processes because 1 process can only take a maximum of 32'000 statements.
I think this performs as well or slightly better than using 32'000.
However if your total dataset is only 320'000 updates and you choose a chunk_size of 64'000, you can use a maximum of 5 threads. I think that would be a downside of using multiples of 32'000.

[MariusWirtz]

Based on the discussions I had recently with people I think TM1py should provide a default value for chunk_size in the write_async functions. I think the casual TM1py user shouldn't the bothered with the thoughts above.
What do you think?

By the way, if you do investigate further or possibly quantify something, it would be great if you could share your findings.

I would like to provide a sensible default value.