Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
171fe20
Show warning just when BTUNE_TRACE mode is on
FrancescAlted Jul 31, 2025
4096030
Modernize CI toolchain
FrancescAlted Jul 31, 2025
11405c2
Upgrade to upload-artifact v4
FrancescAlted Jul 31, 2025
b665fe8
Remove x86 binaries for macosx wheels
FrancescAlted Jul 31, 2025
10da133
Re-add Linux platform
FrancescAlted Jul 31, 2025
0a03bf4
Make sure we run prebuild.sh
FrancescAlted Jul 31, 2025
5c42330
Do not skip tests on macosx arm64 anymore
FrancescAlted Jul 31, 2025
98ee61c
Make artifacts names unique
FrancescAlted Jul 31, 2025
265447e
Avoid generation of SME instruction set
FrancescAlted Jul 31, 2025
96b86d7
Fix YAML issue
FrancescAlted Jul 31, 2025
ad88ba3
Add new graphic
FrancescAlted Jul 31, 2025
c1be917
Use an older version of macos when building
FrancescAlted Jul 31, 2025
35744c3
Exclude aarch in macos-13 too
FrancescAlted Jul 31, 2025
5596de9
macos-13 does not work either. go back to python 3.10
FrancescAlted Jul 31, 2025
ebec59e
Try to use x86_64 with macos
FrancescAlted Jul 31, 2025
d6a2d7d
x86_64 on mac failed. try using arm64 with python 3.11
FrancescAlted Jul 31, 2025
d755572
Attempt on tf v2.14.0
FrancescAlted Jul 31, 2025
7d36d75
Attempt on tf v2.18.0
FrancescAlted Jul 31, 2025
f121369
Attempt on tf v2.18.0 and python 3.12
FrancescAlted Jul 31, 2025
1311c0a
Revert to tf v2.19.0 and python 3.12
FrancescAlted Jul 31, 2025
8d5884a
Check with main branch of tensorflow
FrancescAlted Jul 31, 2025
6b85ff2
Revert to v2.19.0 of tensorflow
FrancescAlted Jul 31, 2025
a00dc9d
Be explicit on what we are building
FrancescAlted Jul 31, 2025
bb6ffbd
Remove the generation of MacOS wheels for now (use <=1.2.0 if needed)
FrancescAlted Aug 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 40 additions & 21 deletions .github/workflows/cibuildwheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,19 @@ on:
env:
CIBW_SKIP: '*-win32 *-manylinux*_i686 *-musllinux_*'
CIBW_TEST_REQUIRES: blosc2 pillow
CIBW_TEST_SKIP: "*macosx*arm64*"
# CIBW_TEST_SKIP: "*macosx*arm64*"
CIBW_TEST_COMMAND: 'BTUNE_TRACE=1 python {project}/examples/btune_config.py;
BTUNE_TRADEOFF="(0.3, 0.1, 0.6)" BTUNE_PERF_MODE=DECOMP BTUNE_TRACE=1 python {project}/examples/lossy.py {project}/examples/lossy_example.tif'
CIBW_BUILD_VERBOSITY: 1
CIBW_ENVIRONMENT: "CMAKE_ARGS='-DCMAKE_POLICY_VERSION_MINIMUM=3.5'"
# The next are necessary to avoid the generation of SME instruction set
CIBW_ENVIRONMENT_MACOS: >
CMAKE_ARGS='-DCMAKE_POLICY_VERSION_MINIMUM=3.5
-DTFLITE_ENABLE_XNNPACK=OFF
-DTFLITE_ENABLE_GPU=OFF
-DXNNPACK_ENABLE_ARM_BF16=OFF
-DXNNPACK_ENABLE_ARM_FP16_VECTOR=OFF
-DXNNPACK_ENABLE_ARM_FP16_SCALAR=OFF'

jobs:

Expand All @@ -27,21 +36,28 @@ jobs:
env:
CIBW_BUILD: ${{ matrix.cibw_build }}
CIBW_ARCHS_LINUX: ${{ matrix.arch }}
CIBW_ARCHS_MACOS: "x86_64 arm64"
CIBW_ARCHS_MACOS: "arm64"
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
arch: [x86_64, aarch64]
#cibw_build: ["cp3{9,10,11,12}-*"]
#p_ver: ["3.9-3.12"]
cibw_build: ["cp310-*"]
p_ver: ["3.10"]
exclude:
- os: windows-latest
arch: aarch64
# cibuild is already in charge to build aarch64 (see CIBW_ARCHS_MACOS)
- os: macos-latest
include:
- os: ubuntu-latest
arch: x86_64
cibw_build: "cp312-*"
p_ver: "3.12"
- os: ubuntu-latest
arch: aarch64
cibw_build: "cp312-*"
p_ver: "3.12"
- os: windows-latest
arch: x86_64
cibw_build: "cp312-*"
p_ver: "3.12"
# We are blocked on MacOS because tensorflow issues:
# https://github.com/tensorflow/tensorflow/issues/98002
# - os: macos-latest
# arch: arm64
# cibw_build: "cp312-*"
# p_ver: "3.12"

steps:
- name: Checkout repo
Expand All @@ -52,7 +68,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
python-version: '3.12'

- name: Set up QEMU
if: ${{ matrix.arch == 'aarch64' }}
Expand All @@ -79,15 +95,17 @@ jobs:
cmd /V /C "set BTUNE_TRACE=1; set BTUNE_TRADEOFF="(0.3, 0.1, 0.6)"; set BTUNE_PERF_MODE=DECOMP;
python {project}/examples/lossy.py {project}/examples/lossy_example.tif"

- name: Build wheels (Linux / Mac OSX)
if: runner.os != 'Windows'
- name: Build wheels (Linux)
if: runner.os == 'Linux'
run: |
python -m pip install --upgrade pip
python -m pip install cibuildwheel
python -m cibuildwheel --output-dir wheelhouse
env:
CIBW_BEFORE_BUILD: CMAKE_OSX_ARCHITECTURES=x86_64 bash prebuild.sh
CIBW_ARCHS_MACOS: "x86_64"
CIBW_BEFORE_BUILD: bash prebuild.sh
CIBW_ENVIRONMENT: >
CMAKE_ARGS='-DCMAKE_POLICY_VERSION_MINIMUM=3.5
${{ matrix.arch == 'aarch64' && '-DTFLITE_ENABLE_XNNPACK=OFF -DTFLITE_ENABLE_GPU=OFF' || '' }}'

- name: Build wheels (Mac OSX arm64)
if: runner.os != 'Windows' && runner.os != 'Linux'
Expand All @@ -100,8 +118,9 @@ jobs:
CIBW_ARCHS_MACOS: "arm64"

- name: Upload wheels
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: wheels-${{ matrix.os }}-${{ runner.os == 'macOS' && 'arm64' || matrix.arch }}
path: ./wheelhouse/*.whl


Expand All @@ -111,10 +130,10 @@ jobs:
# Only upload wheels when tagging (typically a release)
if: startsWith(github.event.ref, 'refs/tags')
steps:
- uses: actions/download-artifact@v3
- uses: actions/download-artifact@v4
with:
name: artifact
path: dist
merge-multiple: true

- uses: pypa/gh-action-pypi-publish@release/v1
with:
Expand Down
6 changes: 3 additions & 3 deletions README-DEVELOPERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Also, tensorflow comes with its own version, so this is not needed.
For Linux:

```shell
CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp311-manylinux_x86_64'
CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp312-manylinux_x86_64'
```

Please note that the prebuild.sh will be executed from inside the docker
Expand All @@ -20,13 +20,13 @@ Please note that the prebuild.sh will be executed from inside the docker
For Mac x86_64:

```shell
CMAKE_OSX_ARCHITECTURES=x86_64 CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp311-macosx_x86_64'
CMAKE_OSX_ARCHITECTURES=x86_64 CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp312-macosx_x86_64'
```

For Mac arm64:

```shell
CMAKE_OSX_ARCHITECTURES=arm64 CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp311-macosx_arm64'
CMAKE_OSX_ARCHITECTURES=arm64 CIBW_BEFORE_BUILD="bash prebuild.sh" python -m cibuildwheel --only 'cp312-macosx_arm64'
```

## Install the wheel
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Btune for Blosc2

<img src="img/btune-preview-running.png">
<img src="img/blosc2-btune-diagram-2.png">

Btune is a dynamic plugin for Blosc2 that assists in finding the optimal combination of compression parameters. It works by training a neural network on your most representative datasets.

Expand Down
Binary file added img/blosc2-btune-diagram-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified img/btune-preview-running.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 3 additions & 1 deletion prebuild.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@
# v2.12.0 does not seem to work on neither Linux nor Mac (and static compiling)
# v2.13.0-rc0 does seems to work again on both platforms
# v2.15.0 does not seem to work on Win (error C1083: Cannot open include file: 'sys/mman.h': No such file or directory)
TENSORFLOW_VERSION="v2.14.0"
TENSORFLOW_VERSION="v2.19.0"
if [ ! -d "tensorflow_src" ]
then
git clone --depth=1 -b $TENSORFLOW_VERSION https://github.com/tensorflow/tensorflow.git tensorflow_src
# main branch:
# git clone --depth=1 https://github.com/tensorflow/tensorflow.git tensorflow_src
else
echo "TensorFlow ($TENSORFLOW_VERSION) already cloned"
fi
Expand Down
2 changes: 1 addition & 1 deletion src/btune_model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@ int most_predicted(btune_struct *btune_params, int *compcode,
// Get most predicted category
metadata_t *meta = (metadata_t *) btune_params->metadata;
if (meta == NULL) {
printf("WARNING: Empty metadata, no inference performed\n");
BTUNE_TRACE("WARNING: Empty metadata, no inference performed\n");
return -1;
}
int best_idx = 0;
Expand Down