Skip to content

Support ZSTD compression method for reading#256

Merged
fintelia merged 2 commits intoimage-rs:masterfrom
stschake:feature/zstd-support
Dec 20, 2024
Merged

Support ZSTD compression method for reading#256
fintelia merged 2 commits intoimage-rs:masterfrom
stschake:feature/zstd-support

Conversation

@stschake
Copy link
Copy Markdown
Contributor

ZSTD is generally recognized as superior to LZW in size and speed, particularly for decompression.
We use the self-assigned tag first defined by libtiff and also used by GDAL and others.

ZSTD is generally recognized as superior to LZW in size and
speed, particularly for decompression.
We use the self-assigned tag first defined by libtiff and
also used by GDAL and others.
@stschake stschake force-pushed the feature/zstd-support branch from 5444ff9 to 663c19c Compare December 19, 2024 18:55
@stschake stschake changed the title Support the ZSTD compression method for reading Support ZSTD compression method for reading Dec 19, 2024
@fintelia
Copy link
Copy Markdown
Contributor

fintelia commented Dec 20, 2024

Thanks for this PR, zstd support is something I've wanted for a while!

My main concern is actually around build time. On my machine, the zstd crate takes around 10 seconds to compile and this change increases the main image crate's compile time by nearly 25%.

I see two possible options:

  • Check whether the ruzstd would work as an alternative.
  • Put zstd support behind a feature flag.

@stschake
Copy link
Copy Markdown
Contributor Author

Thanks for having a look! I looked into ruzstd but it seems to have a 1.5x-2x performance penalty over the reference implementation, which I felt would diminish the benefit of zstd too much. I added a feature flag now for the zstd support.

@fintelia fintelia merged commit a19cf37 into image-rs:master Dec 20, 2024
@fintelia
Copy link
Copy Markdown
Contributor

Thanks! If I have a chance, I might look into using ruzstd decoder by default and having the feature opt into the faster zstd bindings. Even the slower decoder may still significantly outperform LZW

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants