Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support of blosc2 plugins to the blosc2 filter; Updated c-blosc2 v2.13.2 #295

Merged
merged 14 commits into from
Feb 8, 2024

Conversation

t20100
Copy link
Member

@t20100 t20100 commented Feb 6, 2024

This PR updates the build of the blosc2 filter to add the embedded plugins and support for the blosc2 plugin system.
Still need to add some tests.

closes #254

@t20100 t20100 added this to the Next release milestone Feb 6, 2024
.github/workflows/ci.yml Show resolved Hide resolved
.github/workflows/release.yml Outdated Show resolved Hide resolved
@t20100 t20100 marked this pull request as ready for review February 6, 2024 16:02
@t20100
Copy link
Member Author

t20100 commented Feb 6, 2024

There's an issue with the release CI when building bytedelta.c:

Full log..
INFO:root:gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_PLUGINS=1 -DSHUFFLE_AVX512_ENABLED=1 -DSHUFFLE_NEON_ENABLED=1 -DHAVE_ZLIB=1 -DHAVE_ZSTD=1 -DH5_USE_18_API -DSHUFFLE_SSE2_ENABLED=1 -Isrc/hdf5/include -Isrc/hdf5/include/linux -Isrc/c-blosc2 -Isrc/c-blosc2/blosc -Isrc/c-blosc2/include -Isrc/c-blosc2/plugins/codecs/zfp/include -Isrc/c-blosc2/internal-complibs/lz4-1.9.4 -Isrc/c-blosc/internal-complibs/zlib-1.2.13 -Isrc/c-blosc2/internal-complibs/zstd-1.5.5 -Isrc/c-blosc2/internal-complibs/zstd-1.5.5/common -Isrc/PyTables/hdf5-blosc2/src -I/opt/_internal/cpython-3.11.6/include/python3.11 -c src/c-blosc2/plugins/filters/bytedelta/bytedelta.c -o build/temp.linux-x86_64-cpython-311/src/c-blosc2/plugins/filters/bytedelta/bytedelta.o -O3 -ffast-math -std=gnu99 -pthread -msse2
        In file included from src/c-blosc2/plugins/filters/bytedelta/bytedelta.c:25:
        src/c-blosc2/plugins/filters/bytedelta/bytedelta.c: In function ‘simd_concat’:
        /opt/rh/devtoolset-10/root/usr/lib/gcc/x86_64-redhat-linux/10/include/tmmintrin.h:185:1: error: inlining failed in call to ‘always_inline’ ‘_mm_alignr_epi8’: target specific option mismatch
          185 | _mm_alignr_epi8(__m128i __X, __m128i __Y, const int __N)
              | ^~~~~~~~~~~~~~~
        src/c-blosc2/plugins/filters/bytedelta/bytedelta.c:32:54: note: called from here
           32 | bytes16 simd_concat(bytes16 hi, bytes16 lo) { return _mm_alignr_epi8(hi, lo, 15); }
              |                                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~
        error: command '/opt/rh/devtoolset-10/root/usr/bin/gcc' failed with exit code 1

Due to a missing SIMD flag since HDF5PLUGIN_AVX2=False for this build and this relies on SSSE3 instructions..

As opposed to SSE2 and AVX*, there is no compilation option to toggle the use of SSSE3.
BTW on Windows there is no /arch:SSSE3

@t20100
Copy link
Member Author

t20100 commented Feb 7, 2024

A fix for the SSSE3 issue is proposed here: Blosc/c-blosc2#586

@t20100 t20100 marked this pull request as draft February 7, 2024 11:00
@t20100
Copy link
Member Author

t20100 commented Feb 7, 2024

PR Blosc/c-blosc2#586 has been merged, and this fix is included in this PR to make it work until the next release of c-blsoc2.

@t20100 t20100 marked this pull request as ready for review February 7, 2024 12:22
@t20100 t20100 requested a review from vasole February 7, 2024 12:22
@FrancescAlted
Copy link

A new release of C-Blosc2 (2.13.2), with the PR Blosc/c-blosc2#586 in, has been made. Feel free to check with it.

@t20100
Copy link
Member Author

t20100 commented Feb 7, 2024

Thanks @FrancescAlted for the quick release! I will update hdf5plugin to use it.

ed9002c99 Getting ready for release 2.13.2
e0918a19b Document global registered filters and codecs
8c519b428 rely on __SSSE3__ macro rather that on architecture macros
f0183767d Quodana does not seem designed for C (yet)
103e17cb5 Initialization file for quodana
472dc12f2 Add code quality assessment with Qodana
ebfb27d36 Fixed some warnings found by clang-tidy
9f6eaf6f2 Fixed a couple of issues introduced in last PR
a4d52c6db Adding decompress_file example, in order to have a counterpart of compress_file example
98ba23b25 Fix a compiler warning
a52b8debf Add private functions section
bd3d21365 Post 2.13.1 release actions done

git-subtree-dir: src/c-blosc2
git-subtree-split: 85a8f739158cff6ce92e8d9d750366e18ce3172d
@t20100 t20100 changed the title Added blosc2 plugins to the blosc2 filter Added support of blosc2 plugins to the blosc2 filter; Updated c-blosc2 v2.13.2 Feb 7, 2024
@t20100
Copy link
Member Author

t20100 commented Feb 7, 2024

If CI is happy, this is ready for review.

@t20100 t20100 marked this pull request as draft February 8, 2024 08:37
@t20100
Copy link
Member Author

t20100 commented Feb 8, 2024

Looks to have an issue in the release CI with a missing symbol:

Failed to load filter blosc2: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/_utils.py", line 142, in register_filter
    lib = ctypes.CDLL(filename)
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so: undefined symbol: __pow_finite

@FrancescAlted
Copy link

Looks to have an issue in the release CI with a missing symbol:

Failed to load filter blosc2: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/_utils.py", line 142, in register_filter
    lib = ctypes.CDLL(filename)
  File "/opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/hostedtoolcache/Python/3.7.17/x64/lib/python3.7/site-packages/hdf5plugin/plugins/libh5blosc2.so: undefined symbol: __pow_finite

Interesting. I have never seen this myself. Looks like an issue in clang 10? https://stackoverflow.com/questions/62334452/fast-math-cause-undefined-reference-to-pow-finite

@t20100
Copy link
Member Author

t20100 commented Feb 8, 2024

I also saw an issue with glibc >=2.31 (used in Ubuntu20.04) having removed some symbols used when compiling with -ffast-math compared to previous versions (and here it is compiled with manylinux2014).
I'm now trying with removing -ffast-math since it is not used in blosc2 cmake files anyway.

@t20100 t20100 marked this pull request as ready for review February 8, 2024 11:43
@t20100
Copy link
Member Author

t20100 commented Feb 8, 2024

Missing symbol issue fixed by removing -ffast-math and manually started release CI passes (see https://github.com/t20100/hdf5plugin/actions/runs/7828434787).

@t20100
Copy link
Member Author

t20100 commented Feb 8, 2024

Ready for me

doc/usage.rst Outdated Show resolved Hide resolved
@vasole vasole merged commit a0f37f3 into silx-kit:main Feb 8, 2024
7 checks passed
@t20100 t20100 deleted the add-blosc2-plugins branch February 8, 2024 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add compilation of blosc2 "plugins"
3 participants