Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable build of linux-aarch64 wheels #34

Merged
merged 7 commits into from
Mar 9, 2024
Merged

Enable build of linux-aarch64 wheels #34

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4a3e113
Enable Linux ARM builds; skip tests for Linux ARM; Use external image…
Mar 1, 2024
08dd6c4
Test if directly using cibuildwheel recipe fixes QEMU problems
Mar 1, 2024
5be6d53
Move to new chatnoir-eu images
Mar 8, 2024
f062886
Move aarch64 build to separate job since no progress on bug pypa/cibu…
Mar 8, 2024
ff1fc9f
Update buil-asan job to new image
Mar 9, 2024
d006d35
Remove old comment
Mar 9, 2024
4c10b5c
Fix whitespace
Mar 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 63 additions & 10 deletions .github/workflows/build-wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ jobs:
os: [ ubuntu-latest, windows-latest, macOS-latest ]
env:
CIBW_SKIP: "*-musllinux*"
CIBW_MANYLINUX_X86_64_IMAGE: ghcr.io/chatnoir-eu/resiliparse-manylinux2014_x86_64
CIBW_MANYLINUX_X86_64_IMAGE: ghcr.io/chatnoir-eu/resiliparse-manylinux_2_28_x86_64:latest
CIBW_ARCHS_MACOS: "x86_64 arm64"
CIBW_ARCHS_LINUX: "x86_64"
CIBW_TEST_SKIP: "*-macosx_arm64" # Apple Silicon wheels cannot be tested
CIBW_REPAIR_WHEEL_COMMAND_MACOS: >-
DYLD_LIBRARY_PATH=$LIBRARY_PATH delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}
Expand Down Expand Up @@ -75,16 +76,68 @@ jobs:
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install cibuildwheel
run: python -m pip install cibuildwheel
- name: Build FastWARC
uses: pypa/cibuildwheel@v2.16.5
with:
package-dir: fastwarc
output-dir: wheelhouse
env:
CIBW_TEST_COMMAND: python -m pytest --capture=sys --verbose {project}/tests/fastwarc

- name: Build Resiliparse
uses: pypa/cibuildwheel@v2.16.5
with:
package-dir: resiliparse
output-dir: wheelhouse
env:
CIBW_BEFORE_TEST: >-
python -c "import glob, platform; open('fastwarc.txt', 'w').write(glob.glob('wheelhouse/FastWARC-*cp' + ''.join(map(str, platform.python_version_tuple()[:2])) + '-*_' + platform.machine().lower() + '.whl')[0])" &&
python -m pip install -r fastwarc.txt
CIBW_TEST_COMMAND: python -m pytest --capture=sys --verbose {project}/tests/resiliparse

- name: Upload Wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: ./wheelhouse/*.whl

build-wheels-qemu:
runs-on: ubuntu-latest
env:
CIBW_SKIP: "*-musllinux*"
CIBW_MANYLINUX_AARCH64_IMAGE: ghcr.io/chatnoir-eu/resiliparse-manylinux_2_28_aarch64:latest
CIBW_ARCHS_LINUX: "aarch64"
CIBW_TEST_SKIP: "*-manylinux_aarch64"
# https://github.com/pypa/cibuildwheel/issues/1771#issuecomment-1973003145
CIBW_CONTAINER_ENGINE: "docker; create_args: --platform linux/arm64/v8"

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: all

- name: Build FastWARC
run: python -m cibuildwheel --output-dir wheelhouse fastwarc
uses: pypa/cibuildwheel@v2.16.5
with:
package-dir: fastwarc
output-dir: wheelhouse
env:
CIBW_TEST_COMMAND: python -m pytest --capture=sys --verbose {project}/tests/fastwarc

- name: Build Resiliparse
run: python -m cibuildwheel --output-dir wheelhouse resiliparse
uses: pypa/cibuildwheel@v2.16.5
with:
package-dir: resiliparse
output-dir: wheelhouse
env:
CIBW_BEFORE_TEST: >-
python -c "import glob, platform; open('fastwarc.txt', 'w').write(glob.glob('wheelhouse/FastWARC-*cp' + ''.join(map(str, platform.python_version_tuple()[:2])) + '-*_' + platform.machine().lower() + '.whl')[0])" &&
Expand Down Expand Up @@ -126,7 +179,7 @@ jobs:
build-asan:
runs-on: ubuntu-latest
container:
image: ghcr.io/chatnoir-eu/resiliparse-manylinux2014_x86_64
image: ghcr.io/chatnoir-eu/resiliparse-manylinux_2_28_x86_64:latest
env:
DEBUG: "1"
ASAN: "1"
Expand All @@ -152,7 +205,7 @@ jobs:
build-coverage:
runs-on: ubuntu-latest
container:
image: ghcr.io/chatnoir-eu/resiliparse-manylinux2014_x86_64
image: ghcr.io/chatnoir-eu/resiliparse-manylinux_2_28_x86_64:latest
needs: [ build-wheels, build-asan, build-sdist ]
env:
TRACE: "1"
Expand Down Expand Up @@ -211,8 +264,8 @@ jobs:
grep -vE "fastwarc|resiliparse" docs/requirements.txt | xargs python -m pip install

PYTHON_ABI="cp${PYTHON_VERSION/./}"
find wheelhouse -name "FastWARC-*-${PYTHON_ABI}-*-manylinux*.whl" | xargs -I% python -m pip install "%[all]"
find wheelhouse -name "Resiliparse-*-${PYTHON_ABI}-*-manylinux*.whl" | xargs -I% python -m pip install "%[all]"
find wheelhouse -name "FastWARC-*-${PYTHON_ABI}-*-manylinux_x86_64.whl" | xargs -I% python -m pip install "%[all]"
find wheelhouse -name "Resiliparse-*-${PYTHON_ABI}-*-manylinux_x86_64.whl" | xargs -I% python -m pip install "%[all]"

cd docs
make html
Expand All @@ -228,7 +281,7 @@ jobs:
publish-wheels:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
needs: [ build-wheels, build-sdist, build-documentation ]
needs: [ build-wheels, build-wheels-qemu, build-sdist, build-documentation ]
steps:
- name: Download Wheels
uses: actions/download-artifact@v3
Expand Down
Loading