-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
zstd filter: add support for parallel processing #163
Comments
There are 3 main stumbling blocks:
|
Yeah, one might want to use the auto-detection capability of the library:
Heh. I think every time you compress something bigger you want to have it done as fast as possible. That's why using |
My understanding is that the library does not have this capability, only the binary. But if there is a library API to select this mode it would indeed be helpful.
Not if the thing producing the data is producing it at a fairly leisurely pace. It just runs for long enough that you need compression. Also all cases where you have a lot of different things to compress are better handled by parallelism across the items to compress, not by parallelizing the compression. |
Sure, I understand it's not an exciting use case for you. In my case, I stream data to a single file where I would like to apply compression. And that's why boost IO streams come in handy and it would help me to save a lot of time. |
I had a bit of a look. |
Well, ZSTD |
As you probably know,
zstd
is famous for its fast (and stable) parallel compression capability. It would be great ifiostream
would teach the skill.I've got a WIP parallel implementation of the
io::multichar_output_filter
filter:boost-zstd.cc.txt
The text was updated successfully, but these errors were encountered: