Implement ZStandard Stream, Encoder, Decoder

[rzikm]

Implements #59591

TO REVIEWERS:
You can Ignore native code in src/native/external/zstd (verbatim copy of latest zstd release, included in the very first commit)
The actual C# implementation added in this PR is "only" about 4.5k LOC.

There are some patches applied to vendored zstd sources which are listed in src/native/external, neither of them should be necessary once we update to next release (maintainers didn't commit to a release date, but hopefully they will release something early next year).

Benchmarks results (taken from dotnet/performance repo) look great so far

Benchmark results

Method	level	file	Algorithm	Mean	Error	StdDev	Size	Compressed
Compress	Optimal	alice29.txt	Brotli	2,306.40 μs	45.807 μs	71.316 μs	148481	54517
Compress	Optimal	alice29.txt	Zstandard	842.743 μs	8.3719 μs	7.8310 μs	148481	54919
Compress	Optimal	alice29.txt	Deflate	2,421.16 μs	41.274 μs	38.608 μs	148481	53890
Decompress	Optimal	alice29.txt	Brotli	320.82 μs	4.802 μs	4.257 μs
Decompress	Optimal	alice29.txt	Zstandard	139.251 μs	2.0885 μs	1.7440 μs
Decompress	Optimal	alice29.txt	Deflate	275.43 μs	4.849 μs	4.535 μs
Compress	Optimal	sum	Brotli	577.62 μs	7.647 μs	5.971 μs	38240	12527
Compress	Optimal	sum	Zstandard	155.050 μs	3.0367 μs	3.3753 μs	38240	13389
Compress	Optimal	sum	Deflate	404.70 μs	7.732 μs	9.779 μs	38240	13388
Decompress	Optimal	sum	Brotli	101.26 μs	2.781 μs	7.566 μs
Decompress	Optimal	sum	Zstandard	31.333 μs	0.6172 μs	0.5774 μs
Decompress	Optimal	sum	Deflate	54.86 μs	0.676 μs	0.632 μs
Compress	Optimal	TestDocument.pdf	Brotli	763.43 μs	13.851 μs	23.520 μs	121993	115863
Compress	Optimal	TestDocument.pdf	Zstandard	182.115 μs	3.3456 μs	2.9658 μs	121993	119285
Compress	Optimal	TestDocument.pdf	Deflate	2,133.16 μs	42.543 μs	104.358 μs	121993	115369
Decompress	Optimal	TestDocument.pdf	Brotli	267.64 μs	5.281 μs	10.047 μs
Decompress	Optimal	TestDocument.pdf	Zstandard	17.508 μs	0.3336 μs	0.3708 μs
Decompress	Optimal	TestDocument.pdf	Deflate	224.92 μs	2.642 μs	2.342 μs
Compress	Fastest	alice29.txt	Brotli	732.30 μs	14.288 μs	19.074 μs	148481	59265
Compress	Fastest	alice29.txt	Zstandard	509.3 us	5.10 us	4.53 us	148481	59891
Compress	Fastest	alice29.txt	Deflate	612.28 μs	11.300 μs	16.564 μs	148481	83053
Decompress	Fastest	alice29.txt	Brotli	380.92 μs	7.570 μs	9.844 μs
Decompress	Fastest	alice29.txt	Zstandard	111.14 us	1.321 us	1.103 us
Decompress	Fastest	alice29.txt	Deflate	361.78 μs	6.700 μs	14.847 μs
Compress	Fastest	sum	Brotli	125.26 μs	2.489 μs	3.801 μs	38240	14305
Compress	Fastest	sum	Zstandard	108.0 us	1.05 us	0.88 us	38240	13848
Compress	Fastest	sum	Deflate	121.81 μs	2.409 μs	4.699 μs	38240	17222
Decompress	Fastest	sum	Brotli	99.19 μs	1.974 μs	3.243 μs
Decompress	Fastest	sum	Zstandard	29.01 us	0.559 us	0.599 us
Decompress	Fastest	sum	Deflate	58.68 μs	0.999 μs	0.934 μs
Compress	Fastest	TestDocument.pdf	Brotli	300.40 μs	5.753 μs	6.394 μs	121993	117104
Compress	Fastest	TestDocument.pdf	Zstandard	137.3 us	2.00 us	1.67 us	121993	119405
Compress	Fastest	TestDocument.pdf	Deflate	947.61 μs	17.861 μs	18.342 μs	121993	121404
Decompress	Fastest	TestDocument.pdf	Brotli	233.14 μs	3.625 μs	3.391 μs
Decompress	Fastest	TestDocument.pdf	Zstandard	17.22 us	0.260 us	0.244 us
Decompress	Fastest	TestDocument.pdf	Deflate	238.81 μs	2.153 μs	1.908 μs

[am11]

This should be upstreamed. You can link the PR in -version.txt file (see libunwind-version.txt). We've avoided adding patch files like these to the repo in the past.

[am11]

UseSystemXX is paired with CLR_CMAKE_USE_SYSTEM_XX support for distros to use system installed lib. That typically happens when we are confident that distro provided version is working on major distros like Fedora and Ubuntu. See git grep CLR_CMAKE_USE_SYSTEM_BROTLI. In cases where we have many functionality-related patches to make the lib work (like llvm-libunwind has tons of local / hard-to-upstream patches: #72655), distros cannot use the system provided variant and we use the regular in-tree version.

TODOs:

• Add CLR_CMAKE_USE_SYSTEM_ZSTD support in src/native/libs and VMR's repo-projects/runtime.proj (next to CLR_CMAKE_USE_SYSTEM_BROTLI)
• Test with system provided libs at least on fedora and ubuntu
• If the test fails, remove UseSystemZstd in NAOT buildintegration and file an issue to follow up
• Otherwise, check-in those (CLR_CMAKE_USE_SYSTEM_ZSTD) changes.

[rzikm]

Edit: seem to be fixed now

There seems to be some problems with linux x86 build that I am unable to debug on my own.

The build fails because it fails to copy libzstd.a to the coreclr build output, because it attempts to do so before the library is built. The weird part for me is that on all other builds that I checked, the "Installing: ...." steps are happening at the very end of the build, but for this particular pipeline they are happening sometime in the middle, see e.g. https://dev.azure.com/dnceng-public/public/_build/results?buildId=1214394&view=logs&jobId=d82b64b5-3dbf-5e48-910e-ffa3eb07d628&j=d82b64b5-3dbf-5e48-910e-ffa3eb07d628&t=3aaa1282-8fdd-59be-7f23-95b71c611ad2

  -- Install configuration: "CHECKED"
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/./watchdog
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/./watchdog.dbg
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/./corerun
  [ 35%] Built target hosts
  [ 35%] Building C object _deps/zstd-build/lib/CMakeFiles/libzstd_static.dir/__w/1/s/src/native/external/zstd/lib/legacy/zstd_v05.c.o
  [ 35%] Built target eventing_headers
  [ 35%] Building C object _deps/zstd-build/lib/CMakeFiles/libzstd_static.dir/__w/1/s/src/native/external/zstd/lib/legacy/zstd_v06.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/cluster.c.o
  -- Install configuration: "CHECKED"
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/lib/cmake/zstd/zstdTargets.cmake
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/lib/cmake/zstd/zstdTargets-checked.cmake
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/lib/cmake/zstd/zstdConfig.cmake
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/lib/cmake/zstd/zstdConfigVersion.cmake
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/lib/pkgconfig/libzstd.pc
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/include/zdict.h
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/include/zstd.h
  -- Installing: /__w/1/s/artifacts/bin/coreclr/linux.x86.Checked/include/zstd_errors.h
  CMake Error at _deps/zstd-build/lib/cmake_install.cmake:58 (file):
    file INSTALL cannot find
    "/__w/1/s/artifacts/obj/coreclr/linux.x86.Checked/_deps/zstd-build/lib/libzstd.a":
    No such file or directory.
  Call Stack (most recent call first):
    _deps/zstd-build/cmake_install.cmake:77 (include)
    libs-native/System.IO.Compression.Native/cmake_install.cmake:57 (include)
    libs-native/cmake_install.cmake:47 (include)
    cmake_install.cmake:52 (include)
  
  
  make[3]: *** [CMakeFiles/coreclr_misc.dir/build.make:70: CMakeFiles/coreclr_misc] Error 1
  make[2]: *** [CMakeFiles/Makefile2:1985: CMakeFiles/coreclr_misc.dir/all] Error 2
  make[2]: *** Waiting for unfinished jobs....
  [ 35%] Building C object _deps/zstd-build/lib/CMakeFiles/libzstd_static.dir/__w/1/s/src/native/external/zstd/lib/legacy/zstd_v07.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/command.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/compound_dictionary.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/compress_fragment.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/compress_fragment_two_pass.c.o
  [ 35%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/dictionary_hash.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/encode.c.o
  [ 37%] Linking C static library libzstd.a
  [ 37%] Built target libzstd_static
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/encoder_dict.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/entropy_encode.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/fast_log.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/histogram.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/literal_cost.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/memory.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/metablock.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/static_dict.c.o
  [ 37%] Building C object _deps/brotli-build/CMakeFiles/brotlienc.dir/c/enc/utf8_util.c.o
  [ 37%] Linking C static library libbrotlienc.a
  [ 37%] Built target brotlienc
  make[1]: *** [CMakeFiles/Makefile2:1798: CMakeFiles/runtime.dir/rule] Error 2
  make: *** [Makefile:241: runtime] Error 2
  /__w/1/s/src/coreclr

I think I have exhausted the more reasonable debugging Ideas, Can somebody assist here? Maybe @jkoritzinsky? Your name pops up a lot in CMakeLists git blame.