Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per file and per block multithreading don't work correctly #134

Open
K0-RR opened this issue Oct 22, 2023 · 3 comments
Open

Per file and per block multithreading don't work correctly #134

K0-RR opened this issue Oct 22, 2023 · 3 comments

Comments

@K0-RR
Copy link

K0-RR commented Oct 22, 2023

ECT only uses 12,5% of my CPU. I have 16 threads so I used --mt-deflate=16 --mt-file=16 but looking at the system monitor only two threads are being utilized to 100% and the rest are idling.

@fhanau
Copy link
Owner

fhanau commented Oct 23, 2023

ECT uses multithreading, but there are limitations to how many threads it can use based on the input data.
--mt-deflate uses several threads to compress the different blocks in a file. If the block splitting algorithm divides the data into n blocks, up to n threads can be used to compress the data as the data can't be divided arbitrarily without making the compression ratio worse. --mt-file compresses several files in parallel – if there are n files up to n threads will be used at once.
Use care when using both options at once – if you are compressing several larger files with these options you might end up with 16^2=256 threads at once, which will slow down the overall system.

@fideliochan
Copy link

@fhanau does ect use multithreading by default or do I have to pass mt arguments?

@fhanau
Copy link
Owner

fhanau commented Jun 18, 2024

multithreading is off-by-default. Since there are two different multithreading modes, it is non-trivial to define a good default multithreading setting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants