Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update deploy things for py312 + OpenID Connect publishing #4325

Merged
merged 5 commits into from
Oct 27, 2023
Merged

Update deploy things for py312 + OpenID Connect publishing #4325

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ebce8b8
update things for py312
IAlibay Oct 27, 2023
bf11c27
add a pr shim for testing
IAlibay Oct 27, 2023
cff1d90
fix test shim
IAlibay Oct 27, 2023
069e9bb
MDA now has a trusted publisher
IAlibay Oct 27, 2023
cd9a997
remove test trigger
IAlibay Oct 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/darkerbot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- uses: actions/setup-python@v4
with:
Expand Down
49 changes: 21 additions & 28 deletions .github/workflows/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
pull_request:
branches:
- "package-*"
- develop
IAlibay marked this conversation as resolved.
Show resolved Hide resolved
push:
branches:
- "package-*"
Expand Down Expand Up @@ -37,31 +38,32 @@ jobs:
fail-fast: false
matrix:
buildplat:
- [ubuntu-20.04, manylinux_x86_64]
- [macos-11, macosx_*]
- [windows-2019, win_amd64]
python: ["cp39", "cp310", "cp311"]
- [ubuntu-22.04, manylinux_x86_64, x86_64]
- [macos-11, macosx_*, x86_64]
- [windows-2019, win_amd64, AMD64]
python: ["cp39", "cp310", "cp311", "cp312"]
defaults:
run:
working-directory: ./package
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Build wheels
uses: pypa/cibuildwheel@v2.11.2
uses: pypa/cibuildwheel@v2.16.2
with:
package-dir: package
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
CIBW_ARCHS: ${{ matrix.buildplat[2] }}
CIBW_BUILD_VERBOSITY: 1

- name: upload artifacts
if: |
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/package')) ||
(github.event_name == 'release' && github.event.action == 'published')
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
path: wheelhouse/*.whl
retention-days: 7
Expand All @@ -75,7 +77,7 @@ jobs:
run:
working-directory: ./package
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0

Expand All @@ -86,7 +88,7 @@ jobs:
if: |
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/package')) ||
(github.event_name == 'release' && github.event.action == 'published')
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
path: package/dist/*.tar.gz
retention-days: 7
Expand All @@ -100,7 +102,7 @@ jobs:
run:
working-directory: ./testsuite
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0

Expand All @@ -111,7 +113,7 @@ jobs:
if: |
(github.event_name == 'push' && startsWith(github.ref, 'refs/tags/package')) ||
(github.event_name == 'release' && github.event.action == 'published')
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3
with:
path: testsuite/dist/*.tar.gz
retention-days: 7
Expand All @@ -125,7 +127,7 @@ jobs:
runs-on: ubuntu-latest
needs: [build_wheels, build_sdist, build_sdist_tests]
steps:
- uses: actions/download-artifact@v2
- uses: actions/download-artifact@v3
with:
name: artifact
path: dist
Expand All @@ -136,18 +138,14 @@ jobs:
mv dist/MDAnalysisTests-* testsuite/dist

- name: upload_source_and_wheels
uses: pypa/gh-action-pypi-publish@v1.5.0
uses: pypa/gh-action-pypi-publish@v1.8.10
with:
user: __token__
password: ${{ secrets.TESTPYPI_API_TOKEN_SRC }}
skip_existing: true
repository_url: https://test.pypi.org/legacy/

- name: upload_tests
uses: pypa/gh-action-pypi-publish@v1.5.0
uses: pypa/gh-action-pypi-publish@v1.8.10
with:
user: __token__
password: ${{ secrets.TESTPYPI_API_TOKEN_TEST }}
packages_dir: testsuite/dist
skip_existing: true
repository_url: https://test.pypi.org/legacy/
Expand All @@ -161,7 +159,7 @@ jobs:
runs-on: ubuntu-latest
needs: [build_wheels, build_sdist, build_sdist_tests]
steps:
- uses: actions/download-artifact@v2
- uses: actions/download-artifact@v3
with:
name: artifact
path: dist
Expand All @@ -172,16 +170,11 @@ jobs:
mv dist/MDAnalysisTests-* testsuite/dist

- name: upload_source_and_wheels
uses: pypa/gh-action-pypi-publish@v1.5.0
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN_SRC }}
uses: pypa/gh-action-pypi-publish@v1.18.10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the MDA deployment so complicated that we can't use our action https://github.com/MDAnalysis/pypi-deployment ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cibuildwheels deployments don't work via pypi-deployment right now - that's a bit more complicated so I've not added that branch yet (it's a bit more messy because you usually want to parallelise at the level of runners for each flavour you build wheels for - you don't fully have to though, you can just trust the docker container to get it right).

One day I'll get that done, it's just not been a burning priority (although the amount of cython packages we're maintinaing is starting to grow).

What I can't stuff in a reusable action though is the arm builds (using Cirrus), those will need to exist everywhere we need those builds (currently just here).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanations!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra info for anyone reading this and going "but why cibuildwheels here and not everywhere else" (it's a recurring question we should document) - we use cibuildwheels to build anything that can't do a pure Python wheel (i.e. uses Cython).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a "maintainer FAQ" somewhere — wiki or NF drive?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately not, we have barbones deployment docs on the userguide but yeah there's just not been enough of my time to add docs for everything - that's what we applied for that SDG for though so fingers crossed!


- name: upload_tests
uses: pypa/gh-action-pypi-publish@v1.5.0
uses: pypa/gh-action-pypi-publish@v1.18.10
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN_TEST }}
packages_dir: testsuite/dist

check_testpypi:
Expand All @@ -196,7 +189,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.9", "3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11", "3.12"]
type: ["FULL", "MIN"]
exclude:
# Multiple deps don't like windows
Expand All @@ -206,7 +199,7 @@ jobs:
MPLBACKEND: agg

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_micromamba
uses: mamba-org/setup-micromamba@v1
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/gh-ci-cron.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
timeout-minutes: 60

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -85,7 +85,7 @@ jobs:
timeout-minutes: 60

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -138,7 +138,7 @@ jobs:
os: [ubuntu-20.04, macos-11]

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -186,7 +186,7 @@ jobs:
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -225,7 +225,7 @@ jobs:
os: [ubuntu, macos]
python-version: ["3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -273,7 +273,7 @@ jobs:
wheels: ['true', 'false']
steps:
# Checkout to have access to local actions (i.e. setup-os)
- uses: actions/checkout@v3
- uses: actions/checkout@v4

# We need this here in order to make sure we assign the correct compiler (i.e. clang for macos)
- name: setup_os
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/gh-ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
MPLBACKEND: agg

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_os
uses: ./.github/actions/setup-os
Expand Down Expand Up @@ -139,7 +139,7 @@ jobs:
MPLBACKEND: agg

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_micromamba
uses: mamba-org/setup-micromamba@v1
Expand Down Expand Up @@ -237,7 +237,7 @@ jobs:
timeout-minutes: 30

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- uses: actions/setup-python@v4
with:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/linters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
shell: bash

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0

Expand Down Expand Up @@ -102,7 +102,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- uses: actions/setup-python@v4
with:
Expand All @@ -124,7 +124,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: setup_micromamba
uses: mamba-org/setup-micromamba@v1
Expand Down
6 changes: 3 additions & 3 deletions maintainer/ci/cirrus-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ macos_arm64_task:

ci_script: |
brew update
brew install python@3.11
/opt/homebrew/bin/python3 -m venv ~/py_311
source ~/py_311/bin/activate
brew install python@3.12
/opt/homebrew/bin/python3 -m venv ~/py_312
source ~/py_312/bin/activate
cd package
python -m pip install .
cd ../testsuite
Expand Down
8 changes: 5 additions & 3 deletions maintainer/ci/cirrus-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
build_and_store_wheels: &BUILD_AND_STORE_WHEELS
install_cibuildwheel_script:
- python -m pip install cibuildwheel==2.12.3
- python -m pip install cibuildwheel==2.16.2
run_cibuildwheel_script:
- cibuildwheel ./package
wheels_artifacts:
Expand All @@ -16,7 +16,8 @@ linux_aarch64_wheel_task:
cpu: 4
memory: 4G
env:
CIBW_BUILD: cp39-manylinux* cp310-manylinux* cp311-manylinux*
CIBW_BUILD: cp39-manylinux_* cp310-manylinux_* cp311-manylinux_* cp312-manylinux_*
CIBW_ARCHS: ARM64

build_script: |
apt install -y python3-venv python-is-python3
Expand All @@ -29,7 +30,8 @@ macos_arm64_wheel_task:
macos_instance:
image: ghcr.io/cirruslabs/macos-monterey-xcode:14
env:
CIBW_BUILD: cp39-* cp310-* cp311-*
CIBW_BUILD: cp39-* cp310-* cp311-* cp312-*
CIBW_ARCHS: ARM64
PATH: /opt/homebrew/opt/python@3.10/bin:$PATH

build_script: |
Expand Down
Loading