Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
724ad85
Add zstd lib files to the repo
rzikm Sep 3, 2025
68c7505
Link zstd to System.IO.Compression.Native
rzikm Sep 3, 2025
15ee2fe
Skeleton implementation
rzikm Sep 4, 2025
9b9ef3f
Export native zstd symbols
rzikm Sep 8, 2025
065c073
ZDictionary creation
rzikm Sep 8, 2025
69bc9ac
ZStandardEncoder
rzikm Sep 8, 2025
694be0c
ZStandardDecoder
rzikm Sep 8, 2025
e20b035
Roundtrip tests
rzikm Sep 8, 2025
bd76d85
Dictionary lifetime management
rzikm Sep 8, 2025
2417c7c
Fix test failures
rzikm Sep 9, 2025
1220608
minor fix
rzikm Sep 9, 2025
ff0e08f
Fix unit tests
rzikm Sep 10, 2025
f55605b
Minor changes
rzikm Sep 10, 2025
e7227d4
move strings to resx
rzikm Sep 11, 2025
8d9306b
Change capitalization to Zstandard
rzikm Sep 22, 2025
9d8dc42
Add support for Zstandard to System.IO.Compression
rzikm Sep 22, 2025
f9faf6b
Add ctor overloads
rzikm Sep 22, 2025
48d43b3
Fix build on Linux
rzikm Sep 24, 2025
d3b1a0a
Fix runtime build on Windows
rzikm Sep 24, 2025
1689460
Fix build on tvos
rzikm Sep 25, 2025
97b4fd0
Fix AoT builds
rzikm Sep 26, 2025
98081be
Fix Android builds
rzikm Sep 26, 2025
4787d0d
More flexible options
rzikm Sep 30, 2025
28c2125
Add ReferencePrefix
rzikm Sep 30, 2025
66f619f
Fix tests
rzikm Sep 30, 2025
322dbac
Add finalizer test
rzikm Sep 30, 2025
e90a175
fixup! Add ReferencePrefix
rzikm Sep 30, 2025
ca4ebd7
fixup! More flexible options
rzikm Sep 30, 2025
fb77336
Add ability to specify max window during decompression
rzikm Sep 30, 2025
b1b2469
Minor changes
rzikm Oct 3, 2025
cfa1884
Use ZSTD_decompressBound
rzikm Oct 3, 2025
46a464c
Update src/native/external/zstd-version.txt
rzikm Oct 6, 2025
bfb499c
fixup! Use ZSTD_decompressBound
rzikm Nov 5, 2025
144dfb2
Feedback from API review
rzikm Nov 5, 2025
beba29f
Update runtime-assets dependencies
rzikm Nov 5, 2025
47537ac
Fix build
rzikm Nov 12, 2025
c74a432
Share memory between CDict and DDict
rzikm Nov 12, 2025
6a16f3a
Use specified window in compress
rzikm Nov 12, 2025
8041429
Deduplication
rzikm Nov 12, 2025
2fab299
Nullcheck
rzikm Nov 12, 2025
61c180a
Fix type
rzikm Nov 12, 2025
7bb75f1
TestGetMaxCompressedSize
rzikm Nov 12, 2025
7c352af
SetSourceSize tests
rzikm Nov 13, 2025
12ad8a3
Checkpoint for SetPrefix
rzikm Nov 13, 2025
7e5ec0b
Prefix tests
rzikm Nov 13, 2025
92fed5e
Decoder.SetPrefix
rzikm Nov 13, 2025
3ac2585
Comment changes
rzikm Nov 13, 2025
a8d11ca
Test compression options
rzikm Nov 13, 2025
23fb7ee
Code style
rzikm Nov 13, 2025
09ded3c
More tests
rzikm Nov 13, 2025
532ca91
Dictionary training tests
rzikm Nov 13, 2025
a2bc5cc
test append checksum
rzikm Nov 13, 2025
e1b29f5
Fix linux build
rzikm Nov 13, 2025
c8fa1e3
Remove trailing whitespace
rzikm Nov 13, 2025
ea44a92
Seal encoder/decoder
rzikm Nov 18, 2025
db359cc
Consistent braces
rzikm Nov 18, 2025
29fac4d
fixup! Seal encoder/decoder
rzikm Nov 18, 2025
bef2672
Use ZstandardUtils.IsError
rzikm Nov 18, 2025
48d3552
Fix compress empty data
rzikm Nov 18, 2025
c286da1
Use PinnedGCHandle<T>
rzikm Nov 18, 2025
a8ee411
Use volatile read to check _activeRwOperation
rzikm Nov 18, 2025
26f67f4
Better dispose with concurrent rw op
rzikm Nov 18, 2025
6736eb0
Use ArrayBuffer
rzikm Nov 18, 2025
2655c8c
Fix test case on 32 bit cpu
rzikm Nov 18, 2025
ba4fc78
Share tests for Encoder/Decoder with Brotli
rzikm Nov 19, 2025
0218510
remove extra newline
rzikm Nov 19, 2025
4e79e5e
Remove patch files, reference upstream commits instead
rzikm Nov 19, 2025
e2802b7
Add test for decompressing corrupted data
rzikm Nov 21, 2025
1afa62b
Convert to xml comments on internal zstd enums
rzikm Nov 21, 2025
9ca15bc
Use byte* instead of IntPtr on interop stubs
rzikm Nov 21, 2025
d2d18f8
Code review feedback
rzikm Nov 21, 2025
7015ce0
Merge remote-tracking branch 'upstream/main' into pr/rzikm/119575
rzikm Nov 24, 2025
5ca5639
Remove unnecessary reference
rzikm Nov 24, 2025
15fa022
Implement rewinding to make tests pass
rzikm Nov 24, 2025
864a80e
Don't leave out safe handles for finalization
rzikm Nov 24, 2025
f028e01
Workaround 32bit allocation issues
rzikm Nov 24, 2025
0f61f50
Fix build
rzikm Nov 25, 2025
301c8ee
API review feedback
rzikm Nov 26, 2025
6b1fb67
Merge branch 'main' into 59591-zstd
rzikm Nov 26, 2025
79e4401
Fix build on x86 linux
rzikm Nov 26, 2025
1737925
Revert change
rzikm Nov 26, 2025
c723a3b
test
rzikm Nov 26, 2025
de4ab98
another trial
rzikm Nov 27, 2025
2ce1941
Iterating
rzikm Nov 27, 2025
0bda120
Fix build issues on Linux x86
rzikm Nov 28, 2025
26c1bcf
More code coverage
rzikm Dec 1, 2025
ca96633
Fix thrown exception type
rzikm Dec 2, 2025
6c88b5c
Fix platform-dependent max window
rzikm Dec 2, 2025
e6c3f8a
Apply suggestions from code review
rzikm Dec 2, 2025
55d2c55
Use quality 1 for "Fastest"
rzikm Dec 2, 2025
3b3bb10
Remove extraneous test
rzikm Dec 3, 2025
3307ee0
Fix test
rzikm Dec 3, 2025
5896899
Merge branch 'main' into 59591-zstd
rzikm Dec 4, 2025
32ce3a9
Fix param type
rzikm Dec 9, 2025
fb3cdba
Revert changes to Versions.props
rzikm Dec 9, 2025
713d2f4
Code review feedback
rzikm Dec 9, 2025
b32e1d9
Merge remote-tracking branch 'upstream/main' into 59591-zstd
rzikm Dec 9, 2025
b10d501
Code review feedback
rzikm Dec 10, 2025
80f39d3
Merge remote-tracking branch 'upstream/main' into 59591-zstd
rzikm Dec 10, 2025
57d0c2d
Add stream truncation tests
rzikm Dec 10, 2025
33e6c43
Remove todo
rzikm Dec 10, 2025
b5ee213
Lower stackalloc, add fallback to pooling
rzikm Dec 10, 2025
381b53c
Wrap native errors in IOException
rzikm Dec 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions THIRD-PARTY-NOTICES.TXT
Original file line number Diff line number Diff line change
Expand Up @@ -1167,6 +1167,41 @@ WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWIS
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

License notice for zstd
-----------------------

https://github.com/facebook/zstd/blob/dev/LICENSE

BSD License

For Zstandard software
Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name Facebook, nor Meta, nor the names of its contributors may
be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

License notice for fmtlib/fmt
-------------------------------

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ The .NET Foundation licenses this file to you under the MIT license.
<UseSystemZlib Condition="'$(UseSystemZlib)' == '' and !Exists('$(IlcFrameworkNativePath)libz.a')">true</UseSystemZlib>
<!-- Use libbrotlicommon.a as the sentinel for the three brotli libs. -->
<UseSystemBrotli Condition="'$(UseSystemBrotli)' == '' and !Exists('$(IlcFrameworkNativePath)libbrotlicommon.a')">true</UseSystemBrotli>
<UseSystemZstd Condition="'$(UseSystemZstd)' == '' and !Exists('$(IlcFrameworkNativePath)libzstd.a')">true</UseSystemZstd>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UseSystemXX is paired with CLR_CMAKE_USE_SYSTEM_XX support for distros to use system installed lib. That typically happens when we are confident that distro provided version is working on major distros like Fedora and Ubuntu. See git grep CLR_CMAKE_USE_SYSTEM_BROTLI. In cases where we have many functionality-related patches to make the lib work (like llvm-libunwind has tons of local / hard-to-upstream patches: #72655), distros cannot use the system provided variant and we use the regular in-tree version.

TODOs:

  • Add CLR_CMAKE_USE_SYSTEM_ZSTD support in src/native/libs and VMR's repo-projects/runtime.proj (next to CLR_CMAKE_USE_SYSTEM_BROTLI)
  • Test with system provided libs at least on fedora and ubuntu
  • If the test fails, remove UseSystemZstd in NAOT buildintegration and file an issue to follow up
  • Otherwise, check-in those (CLR_CMAKE_USE_SYSTEM_ZSTD) changes.


<FullRuntimeName>libRuntime.WorkstationGC</FullRuntimeName>
<FullRuntimeName Condition="'$(ServerGarbageCollection)' == 'true' or '$(IlcLinkServerGC)' == 'true'">libRuntime.ServerGC</FullRuntimeName>
Expand Down Expand Up @@ -178,6 +179,11 @@ The .NET Foundation licenses this file to you under the MIT license.
<NativeLibrary Include="$(IlcFrameworkNativePath)libbrotlicommon.a" />
</ItemGroup>

<ItemGroup>
<!-- zstd must be added after System.IO.Compression.Native due to dependency. -->
<NativeLibrary Condition="'$(UseSystemZstd)' != 'true'" Include="$(IlcFrameworkNativePath)libzstd.a" />
</ItemGroup>

<ItemGroup Condition="'$(StaticICULinking)' == 'true' and '$(NativeLib)' != 'Static' and '$(InvariantGlobalization)' != 'true'">
<NativeLibrary Include="$(IntermediateOutputPath)libs/System.Globalization.Native/build/libSystem.Globalization.Native.a" />
<DirectPInvoke Include="System.Globalization.Native" />
Expand Down Expand Up @@ -217,6 +223,7 @@ The .NET Foundation licenses this file to you under the MIT license.
<NativeSystemLibrary Include="swiftFoundation" Condition="'$(_IsApplePlatform)' == 'true'" />
<NativeSystemLibrary Include="z" Condition="'$(UseSystemZlib)' == 'true'" />
<NativeSystemLibrary Include="brotlienc;brotlidec;brotlicommon" Condition="'$(UseSystemBrotli)' == 'true'" />
<NativeSystemLibrary Include="zstd" Condition="'$(UseSystemZstd)' == 'true'" />
<NativeSystemLibrary Include="rt" Condition="'$(_IsApplePlatform)' != 'true' and '$(_linuxLibcFlavor)' != 'bionic'" />
<NativeSystemLibrary Include="log" Condition="'$(_linuxLibcFlavor)' == 'bionic'" />
<NativeSystemLibrary Include="icucore" Condition="'$(_IsApplePlatform)' == 'true'" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ The .NET Foundation licenses this file to you under the MIT license.
<NativeLibrary Include="$(IlcSdkPath)brotlicommon$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)brotlienc$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)brotlidec$(LibFileExt)" />
<NativeLibrary Include="$(IlcSdkPath)zstd_static$(LibFileExt)" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,9 @@
<PlatformManifestFileEntry Include="brotlienc.lib" IsNative="true" />
<PlatformManifestFileEntry Include="brotlidec.lib" IsNative="true" />
<PlatformManifestFileEntry Include="brotlicommon.lib" IsNative="true" />
<!-- zstd-specific files -->
<PlatformManifestFileEntry Include="libzstd.a" IsNative="true" />
<PlatformManifestFileEntry Include="zstd_static.lib" IsNative="true" />
</ItemGroup>

<ItemGroup>
Expand Down
Loading
Loading