Add infra for publishing PyPI wheel (#227) * initial setup for PyPi publishing * update tests to run w/o initial global config * update pytest run * skip problematic test cases for pypi build step * try dependency resolution * fix ppe8 error and update dep list - jfennick/workflow-inference-compiler - master #459
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Lint And Test | |
| on: | |
| workflow_dispatch: | |
| # Have to declare parameters here for those that will be sent through 'workflow-dispatch' | |
| # event in branch_dispatch.yml. Otherwise, there'll be github API errors: | |
| # '"message": "Unexpected inputs provided: ...",' | |
| inputs: | |
| event_type: | |
| description: An arbitrary string used to dispatch steps | |
| required: true | |
| type: string | |
| commit_message: | |
| description: The commit message | |
| required: true | |
| type: string | |
| sender_repo: | |
| description: The repository which initiated the workflow dispatch | |
| required: true | |
| type: string | |
| sender_repo_owner: | |
| description: The account name of the repository initiated the workflow dispatch | |
| required: true | |
| type: string | |
| wic_owner: | |
| description: The account name of the wic repository | |
| required: true | |
| type: string | |
| wic_ref: | |
| description: The branch name within the wic repository | |
| required: true | |
| type: string | |
| mm-workflows_owner: | |
| description: The account name of the mm-workflows repository | |
| required: true | |
| type: string | |
| mm-workflows_ref: | |
| description: The branch name within the mm-workflows repository | |
| required: true | |
| type: string | |
| image-workflows_owner: | |
| description: The account name of the image-workflows repository | |
| required: true | |
| type: string | |
| image-workflows_ref: | |
| description: The branch name within the image-workflows repository | |
| required: true | |
| type: string | |
| # See https://github.com/orgs/community/discussions/15690 | |
| # For workflow_dispatch, the commit message, repo, and branch name are not displayed by default. | |
| run-name: "${{ inputs.commit_message}} - ${{ inputs.sender_repo }} - ${{ inputs.wic_ref }}" | |
| defaults: | |
| run: | |
| shell: bash -l {0} # Invoke bash in login mode, NOT interactive mode. | |
| # This will cause bash to look for the startup file ~/.bash_profile, NOT ~/.bashrc | |
| # This is important since conda init writes to ~/.bashrc | |
| # https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#jobs | |
| # Rather than use a single job with a linear pipeline of steps, you may be | |
| # tempted to make each step into a separate job and specify the dependencies | |
| # using the `needs` syntax for more parallelism. | |
| # However, data cannot be shared between jobs because each job will be run on a | |
| # different runner. Even on a self-hosted runner, the `needs` syntax does not | |
| # guarantee that the data can be shared! | |
| # Using if: always() allows all steps to run, while still properly reporting failure. | |
| # See https://stackoverflow.com/questions/62045967/github-actions-is-there-a-way-to-continue-on-error-while-still-getting-correct | |
| permissions: | |
| actions: read | |
| contents: read | |
| pull-requests: read | |
| jobs: | |
| lint_and_test: | |
| # See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency | |
| # This will prevent DOS attacks from people blasting the CI with rapid fire commits. | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ matrix.os }}-${{ github.ref }}-${{ inputs.sender_repo }}-${{ inputs.mm-workflows_ref}} | |
| cancel-in-progress: true | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| # NOTE: Due to 10X minute multiplier, do NOT run on macos-latest on every push! | |
| # https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#minute-multipliers | |
| # Run on macos on a separate weekly schedule. | |
| os: [ubuntu-latest, windows-latest] | |
| runs-on: ${{ matrix.os }} | |
| steps: | |
| - name: Checkout workflow-inference-compiler | |
| if: always() | |
| uses: actions/checkout@v3 | |
| with: | |
| repository: ${{ inputs.wic_owner }}/workflow-inference-compiler | |
| ref: ${{ inputs.wic_ref }} | |
| path: workflow-inference-compiler | |
| - name: Checkout biobb_adapters | |
| if: always() | |
| uses: actions/checkout@v3 | |
| with: | |
| # NOTE: temporarily hardcode jfennick & master because we can only | |
| # have up to 10 input parameters for workflow_dispatch... | |
| repository: jfennick/biobb_adapters | |
| ref: master | |
| path: biobb_adapters | |
| - name: Checkout mm-workflows | |
| if: always() | |
| uses: actions/checkout@v3 | |
| with: | |
| repository: ${{ inputs.mm-workflows_owner }}/mm-workflows | |
| ref: ${{ inputs.mm-workflows_ref }} | |
| path: mm-workflows | |
| - name: Checkout image-workflows | |
| if: always() | |
| uses: actions/checkout@v3 | |
| with: | |
| repository: ${{ inputs.image-workflows_owner }}/image-workflows | |
| ref: ${{ inputs.image-workflows_ref }} | |
| path: image-workflows | |
| # NOTE: pypy actually decreases performance for lint_and_test.yml (by a factor of ~2) | |
| # - name: Append pypy to conda environment files | |
| # if: runner.os != 'Windows' | |
| # run: cd workflow-inference-compiler/install/ && echo " - pypy" >> system_deps.yml | |
| # - name: Append pypy to conda environment files | |
| # if: runner.os == 'Windows' | |
| # run: cd workflow-inference-compiler/install/ && echo " - pypy" >> system_deps_windows.yml | |
| - name: Setup mamba (linux, macos) | |
| if: runner.os != 'Windows' | |
| uses: conda-incubator/setup-miniconda@v3.0.1 | |
| with: | |
| miniforge-variant: Mambaforge-pypy3 | |
| miniforge-version: latest | |
| environment-file: workflow-inference-compiler/install/system_deps.yml | |
| activate-environment: wic | |
| channels: conda-forge | |
| python-version: "3.9.*" # pypy is not yet compatible with 3.10 and 3.11 | |
| - name: Setup mamba (windows) | |
| if: runner.os == 'Windows' | |
| uses: conda-incubator/setup-miniconda@v3.0.1 | |
| with: | |
| miniforge-variant: Mambaforge-pypy3 | |
| miniforge-version: latest | |
| environment-file: workflow-inference-compiler/install/system_deps_windows.yml | |
| activate-environment: wic | |
| channels: conda-forge | |
| python-version: "3.9.*" # pypy is not yet compatible with 3.10 and 3.11 | |
| - name: ShellCheck Script Quality | |
| if: always() | |
| # "SC1017 (error): Literal carriage return. Run script through tr -d '\r' ." | |
| run: shellcheck -e SC1017 $(find workflow-inference-compiler/ -name "*.sh" -and -not -path "./3/*") | |
| - name: Install Workflow Inference Compiler | |
| if: always() | |
| run: cd workflow-inference-compiler/ && pip install ".[all_except_runner_src]" | |
| - name: Install Molecular Modeling Workflows | |
| if: always() | |
| # Also run mm-workflows command to generate | |
| # mm-workflows/autogenerated/schemas/config_schemas.json | |
| # NOTE: Use ".[test]" instead of ".[all_except_runner_src]" | |
| # We do not want or need to install the workflow_deps extra. | |
| # (Many of the packages conflict with pypy.) | |
| run: cd mm-workflows/ && pip install ".[test]" && mm-workflows --generate_schemas | |
| - name: Generate WIC Python API Workflows (*.py -> *.wic) | |
| if: always() | |
| run: cd workflow-inference-compiler/ && python -c 'import wic; import wic.plugins; wic.plugins.blindly_execute_python_workflows()' | |
| - name: Generate WIC Validation Jsonschema | |
| if: always() | |
| run: cd workflow-inference-compiler/ && wic --generate_schemas | |
| # Please read docs/validation.md#Property-Based-Testing | |
| # This is essentially an integration test for all of the | |
| # WIC Python API workflows as well as the WIC Python API itself. | |
| - name: Validate WIC Python API Workflows (*.py -> *.wic) | |
| if: always() | |
| run: cd workflow-inference-compiler/ && python -c 'import wic; import wic.plugins; wic.plugins.blindly_execute_python_workflows()' | |
| - name: Build Documentation | |
| if: always() | |
| run: cd workflow-inference-compiler/docs && make html | |
| - name: MyPy Check Type Annotations | |
| if: always() | |
| run: cd workflow-inference-compiler/ && mypy src/ examples/ tests/ | |
| # NOTE: Do not use `mypy .` because then mypy will check both src/ and build/ causing: | |
| # src/workflow-inference-compiler/__init__.py: error: Duplicate module named "wic" | |
| # (also at "./build/lib/workflow-inference-compiler/__init__.py") | |
| - name: PyLint Check Code Quality | |
| if: always() | |
| run: cd workflow-inference-compiler/ && pylint src/ examples/**/*.py tests/ | |
| # NOTE: See fail-under threshold in .pylintrc | |
| - name: PEP8 Code Formatting | |
| if: always() | |
| id: autopep8 | |
| run: cd workflow-inference-compiler/ && autopep8 --exit-code --recursive --diff --max-line-length 120 examples/ src/ tests/ | |
| - name: Fail if autopep8 made changes | |
| if: steps.autopep8.outputs.exit-code == 2 | |
| run: exit 1 | |
| # NOTE: Do NOT add coverage to PYPY CI runs https://github.com/tox-dev/tox/issues/2252 | |
| - name: PyTest CWL Embedding Independence | |
| if: always() | |
| run: cd workflow-inference-compiler/ && pytest -k test_cwl_embedding_independence # --cov --cov-config=.coveragerc_serial | |
| # NOTE: This test MUST be run in serial! See is_isomorphic_with_timeout() | |
| timeout-minutes: 5 # backup timeout for windows | |
| - name: PyTest Inline Subworkflows | |
| if: always() | |
| run: cd workflow-inference-compiler/ && pytest -k test_inline_subworkflows # --cov --cov-config=.coveragerc_serial | |
| # NOTE: This test MUST be run in serial! See is_isomorphic_with_timeout() | |
| timeout-minutes: 5 # backup timeout for windows | |
| - name: PyTest Scattering Scaling | |
| if: runner.os == 'Linux' | |
| # Avoid Windows and macOS due to 2X and 10X minute multipliers. | |
| # https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#minute-multipliers | |
| # Also, cannot run on Windows (natively) yet | |
| run: cd workflow-inference-compiler/ && pytest -k test_scattering_scaling | |
| # NOTE: The steps below are for repository_dispatch only. For all other steps, please insert above | |
| # this comment. | |
| # Need to store success value in environment variable, rather than using 'success()' in-line inside a run tag. | |
| # Otherwise: "The workflow is not valid. ... Unrecognized function: 'success'." | |
| # https://github.com/actions/runner/issues/834#issuecomment-889484016 | |
| - name: The workflow has succeeded | |
| if: success() | |
| run: | | |
| echo 'workflow_success=true' >> "$GITHUB_ENV" | |
| # It is not clear from the documentation, but the 'success()' and 'failure()' functions | |
| # do not consider skipped steps. Specifically, the above 'success()' function will not | |
| # affect the 'failure()' function here. | |
| # https://docs.github.com/en/actions/learn-github-actions/expressions#status-check-function | |
| - name: The workflow has failed | |
| if: failure() | |
| run: | | |
| echo 'workflow_success=false' >> "$GITHUB_ENV" | |
| # See token.md | |
| - name: Generate a token | |
| if: always() | |
| id: generate_token | |
| uses: tibdex/github-app-token@b62528385c34dbc9f38e5f4225ac829252d1ea92 | |
| with: | |
| app_id: ${{ secrets.APP_ID }} | |
| private_key: ${{ secrets.APP_PRIVATE_KEY }} | |
| - name: Reply CI results to sender | |
| # In case of failure, we still need to return the failure status to the original repository. | |
| # Use 'always()' so this step runs even if there's a failure and use an internal if-statement | |
| # to only run this step only if the repository_dispatch sends the signal. | |
| # https://github.com/actions/runner/issues/834#issuecomment-907096707 | |
| # Use inputs.sender_repo to reply the original sender. | |
| if: always() | |
| uses: ./workflow-inference-compiler/.github/my_actions/reply_sender/ # Must start with ./ | |
| with: | |
| github_repository: ${{ github.repository }} | |
| event_type: ${{ inputs.event_type }} | |
| sender_repo: ${{ inputs.sender_repo }} | |
| operating_system: ${{ matrix.os }} | |
| commit_message: ${{ inputs.commit_message }} | |
| mm_workflows_ref: ${{ inputs.mm-workflows_ref }} | |
| workflow_success: ${{ env.workflow_success }} | |
| access_token: ${{ steps.generate_token.outputs.token }} |