Parallelize runs

[TinoDidriksen]

A pipe often has a single bottleneck (usually a complex CG), so even though the pipe is multi-process, the benefit is reduced. Splitting the input in ~4 and running that many pipes can thus take full advantage of available CPUs.

This should both be per-corpus and across corpora, so runs should internally be changed to a single tasks list.

(quick'n'dirty per-corpus TinoDidriksen/regtest@c18ed0c)

[mr-martian]

I will also note that I think the current setup doesn't even take advantage of pipeline parallelization.

[AsH1605]

Hi,
I’m considering the following approach to optimize performance:
-Split the input file into chunks.
-Run the pipeline on each chunk in parallel.
-Combine outputs into a single result.
Is this the approach valid? Any concerns or edge cases I should consider?

[TinoDidriksen]

I did all that already in https://github.com/TinoDidriksen/regtest - just haven't made it work with Apertium yet.

[AsH1605]

@TinoDidriksen Thanks for sharing the link to regtest. Are there specific issues you're facing with making it work for Apertium? I'd be happy to help troubleshoot or collaborate on this.

[TinoDidriksen]

No issues - just haven't taken the time.

[AsH1605]

Got it! Is there any other area in this project where I can contribute or help further?

[TinoDidriksen]

You're welcome to try to merge the differences between the 2 regtest systems in order to get parallel back into Apertium's copy, or whatever else is interesting.

(when asking on IRC, please ask in the main channel so everyone can see and potentially answer)

[AsH1605]

Sure! I’ll work on proposing a solution for this task and will reach out for further discussion if needed.

I’m currently facing issues with the channel. When I reply, my messages don’t seem to appear, and the channel has crashed a few times. Just wanted to let you know in case it’s a broader issue.