-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Implement ZStandard Stream, Encoder, Decoder #119575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rzikm
wants to merge
103
commits into
dotnet:main
Choose a base branch
from
rzikm:59591-zstd
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
103 commits
Select commit
Hold shift + click to select a range
724ad85
Add zstd lib files to the repo
rzikm 68c7505
Link zstd to System.IO.Compression.Native
rzikm 15ee2fe
Skeleton implementation
rzikm 9b9ef3f
Export native zstd symbols
rzikm 065c073
ZDictionary creation
rzikm 69bc9ac
ZStandardEncoder
rzikm 694be0c
ZStandardDecoder
rzikm e20b035
Roundtrip tests
rzikm bd76d85
Dictionary lifetime management
rzikm 2417c7c
Fix test failures
rzikm 1220608
minor fix
rzikm ff0e08f
Fix unit tests
rzikm f55605b
Minor changes
rzikm e7227d4
move strings to resx
rzikm 8d9306b
Change capitalization to Zstandard
rzikm 9d8dc42
Add support for Zstandard to System.IO.Compression
rzikm f9faf6b
Add ctor overloads
rzikm 48d43b3
Fix build on Linux
rzikm d3b1a0a
Fix runtime build on Windows
rzikm 1689460
Fix build on tvos
rzikm 97b4fd0
Fix AoT builds
rzikm 98081be
Fix Android builds
rzikm 4787d0d
More flexible options
rzikm 28c2125
Add ReferencePrefix
rzikm 66f619f
Fix tests
rzikm 322dbac
Add finalizer test
rzikm e90a175
fixup! Add ReferencePrefix
rzikm ca4ebd7
fixup! More flexible options
rzikm fb77336
Add ability to specify max window during decompression
rzikm b1b2469
Minor changes
rzikm cfa1884
Use ZSTD_decompressBound
rzikm 46a464c
Update src/native/external/zstd-version.txt
rzikm bfb499c
fixup! Use ZSTD_decompressBound
rzikm 144dfb2
Feedback from API review
rzikm beba29f
Update runtime-assets dependencies
rzikm 47537ac
Fix build
rzikm c74a432
Share memory between CDict and DDict
rzikm 6a16f3a
Use specified window in compress
rzikm 8041429
Deduplication
rzikm 2fab299
Nullcheck
rzikm 61c180a
Fix type
rzikm 7bb75f1
TestGetMaxCompressedSize
rzikm 7c352af
SetSourceSize tests
rzikm 12ad8a3
Checkpoint for SetPrefix
rzikm 7e5ec0b
Prefix tests
rzikm 92fed5e
Decoder.SetPrefix
rzikm 3ac2585
Comment changes
rzikm a8d11ca
Test compression options
rzikm 23fb7ee
Code style
rzikm 09ded3c
More tests
rzikm 532ca91
Dictionary training tests
rzikm a2bc5cc
test append checksum
rzikm e1b29f5
Fix linux build
rzikm c8fa1e3
Remove trailing whitespace
rzikm ea44a92
Seal encoder/decoder
rzikm db359cc
Consistent braces
rzikm 29fac4d
fixup! Seal encoder/decoder
rzikm bef2672
Use ZstandardUtils.IsError
rzikm 48d3552
Fix compress empty data
rzikm c286da1
Use PinnedGCHandle<T>
rzikm a8ee411
Use volatile read to check _activeRwOperation
rzikm 26f67f4
Better dispose with concurrent rw op
rzikm 6736eb0
Use ArrayBuffer
rzikm 2655c8c
Fix test case on 32 bit cpu
rzikm ba4fc78
Share tests for Encoder/Decoder with Brotli
rzikm 0218510
remove extra newline
rzikm 4e79e5e
Remove patch files, reference upstream commits instead
rzikm e2802b7
Add test for decompressing corrupted data
rzikm 1afa62b
Convert to xml comments on internal zstd enums
rzikm 9ca15bc
Use byte* instead of IntPtr on interop stubs
rzikm d2d18f8
Code review feedback
rzikm 7015ce0
Merge remote-tracking branch 'upstream/main' into pr/rzikm/119575
rzikm 5ca5639
Remove unnecessary reference
rzikm 15fa022
Implement rewinding to make tests pass
rzikm 864a80e
Don't leave out safe handles for finalization
rzikm f028e01
Workaround 32bit allocation issues
rzikm 0f61f50
Fix build
rzikm 301c8ee
API review feedback
rzikm 6b1fb67
Merge branch 'main' into 59591-zstd
rzikm 79e4401
Fix build on x86 linux
rzikm 1737925
Revert change
rzikm c723a3b
test
rzikm de4ab98
another trial
rzikm 2ce1941
Iterating
rzikm 0bda120
Fix build issues on Linux x86
rzikm 26c1bcf
More code coverage
rzikm ca96633
Fix thrown exception type
rzikm 6c88b5c
Fix platform-dependent max window
rzikm e6c3f8a
Apply suggestions from code review
rzikm 55d2c55
Use quality 1 for "Fastest"
rzikm 3b3bb10
Remove extraneous test
rzikm 3307ee0
Fix test
rzikm 5896899
Merge branch 'main' into 59591-zstd
rzikm 32ce3a9
Fix param type
rzikm fb3cdba
Revert changes to Versions.props
rzikm 713d2f4
Code review feedback
rzikm b32e1d9
Merge remote-tracking branch 'upstream/main' into 59591-zstd
rzikm b10d501
Code review feedback
rzikm 80f39d3
Merge remote-tracking branch 'upstream/main' into 59591-zstd
rzikm 57d0c2d
Add stream truncation tests
rzikm 33e6c43
Remove todo
rzikm b5ee213
Lower stackalloc, add fallback to pooling
rzikm 381b53c
Wrap native errors in IOException
rzikm File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UseSystemXXis paired withCLR_CMAKE_USE_SYSTEM_XXsupport for distros to use system installed lib. That typically happens when we are confident that distro provided version is working on major distros like Fedora and Ubuntu. Seegit grep CLR_CMAKE_USE_SYSTEM_BROTLI. In cases where we have many functionality-related patches to make the lib work (like llvm-libunwind has tons of local / hard-to-upstream patches: #72655), distros cannot use the system provided variant and we use the regular in-tree version.TODOs:
CLR_CMAKE_USE_SYSTEM_ZSTDsupport in src/native/libs and VMR'srepo-projects/runtime.proj(next toCLR_CMAKE_USE_SYSTEM_BROTLI)CLR_CMAKE_USE_SYSTEM_ZSTD) changes.