Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better handling and documentation of dependencies #1589

Merged
merged 28 commits into from
Dec 1, 2021
Merged

Better handling and documentation of dependencies #1589

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
709b1e6
Rename and add environments
tjtg Oct 5, 2021
0f3eae9
Thin out requirements in setup.cfg, add note
tjtg Oct 18, 2021
c2cc73b
Remove improver_tests tests from built package
tjtg Oct 18, 2021
5ef5236
Skip checksum sorted test if file not available
tjtg Oct 5, 2021
0cb26a1
Pytest skips for stratify
tjtg Oct 5, 2021
0e71185
Move stratify import inside function
tjtg Oct 18, 2021
07a0e97
Remove duplicated statsmodels
tjtg Oct 18, 2021
ea9e48a
Better explanation for latest environment
tjtg Oct 18, 2021
a76687a
Add sphinx typehints to conda-forge tests section
tjtg Oct 18, 2021
48ee114
Run security checks on all environments
tjtg Oct 18, 2021
1d913fe
Separate coverage and no-coverage environments
tjtg Oct 18, 2021
c13391d
Add detailed pinning to A/B environments
tjtg Oct 18, 2021
3eb236a
Pin numpy/cartopy in latest environment
tjtg Oct 18, 2021
b83547c
Start documentation
tjtg Oct 19, 2021
27abdcd
Remove dateutil dependency
tjtg Oct 19, 2021
0a99d6a
More documentation
tjtg Oct 19, 2021
1d4c031
Remove timezone database dependence in unit test
tjtg Oct 19, 2021
df67dcd
Fix duplicated name in conda-forge environment
tjtg Oct 20, 2021
be5ebe9
Add note that not all CLIs are available with environment_b
tjtg Oct 20, 2021
9c8b238
Remove leftover expected data from test_process
tjtg Nov 8, 2021
f80c479
Fix flake8
tjtg Nov 8, 2021
3fd5cc5
Remove not-really-working install_requires section
tjtg Nov 22, 2021
1774aea
Clarify conda-forge environment comment
tjtg Nov 22, 2021
3c9a875
Move latest environment tests to schedule
tjtg Nov 22, 2021
5f1b2e2
Fix actions YAML
tjtg Nov 22, 2021
daea7aa
Reschedule
tjtg Nov 22, 2021
196c83e
Fix scheduling, add manual running, metoppv-only
tjtg Nov 23, 2021
c2c7591
Add timezone mask test on europe lat/lon domain
tjtg Nov 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 38 additions & 62 deletions .github/workflows/tests.yml → .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Tests
name: CI Tests

on: [pull_request, push]
jobs:
Expand All @@ -7,7 +7,7 @@ jobs:
strategy:
fail-fast: false
matrix:
env: [py37_iris30, py38_iris30]
env: [environment_a, environment_b, conda-forge]
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
Expand All @@ -16,45 +16,54 @@ jobs:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/improver_${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/environment_{0}.yml', matrix.env))) }}
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/environment_${{ matrix.env }}.yml --name improver_${{ matrix.env }}
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
conda info
conda list
- name: sphinx-build
- name: sphinx documentation
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
make -C doc html SPHINXOPTS="-W --keep-going"
- name: pytest unit-tests & cov-report
- name: pytest without coverage
if: matrix.env != 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
pytest
- name: pytest with coverage
if: matrix.env == 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
pytest --cov=improver --cov-report xml:coverage.xml
- name: codacy-coverage
if: env.CODACY_PROJECT_TOKEN
- name: codacy upload
if: env.CODACY_PROJECT_TOKEN && matrix.env == 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
python-codacy-coverage -v -r coverage.xml
env:
CODACY_PROJECT_TOKEN: ${{ secrets.CODACY_PROJECT_TOKEN }}
- uses: codecov/codecov-action@v1
- name: codecov upload
uses: codecov/codecov-action@v1
if: matrix.env == 'environment_a'
Codestyle-and-flake8:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [py37_iris30]
env: [environment_a]
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
Expand All @@ -63,41 +72,41 @@ jobs:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/improver_${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/environment_{0}.yml', matrix.env))) }}
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/environment_${{ matrix.env }}.yml --name improver_${{ matrix.env }}
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
conda info
conda list
- name: isort
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
isort --check-only .
- name: black
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
black --check .
- name: flake8
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
flake8 improver improver_tests
Safety-Bandit:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [py37_iris30, py38_iris30]
env: [environment_a, environment_b, conda-forge]
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
Expand All @@ -106,60 +115,27 @@ jobs:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/improver_${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/environment_{0}.yml', matrix.env))) }}
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/environment_${{ matrix.env }}.yml --name improver_${{ matrix.env }}
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
conda info
conda list
- name: safety
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
safety check || true
- name: bandit
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda activate im${{ matrix.env }}
bandit -r improver
Type-checking:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [py37_iris30]
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
id: cache
env:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/improver_${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/environment_{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/environment_${{ matrix.env }}.yml --name improver_${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
conda info
conda list
- name: mypy
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate improver_${{ matrix.env }}
mypy improver || true
136 changes: 136 additions & 0 deletions .github/workflows/scheduled.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
name: Nightly tests

on:
schedule:
- cron: '10 * * * *'
MoseleyS marked this conversation as resolved.
Show resolved Hide resolved
jobs:
Sphinx-Pytest-Coverage:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [latest]
if: secrets.SCHEDULED_JOBS == 'yes'
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
id: cache
env:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
conda info
conda list
- name: sphinx documentation
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
make -C doc html SPHINXOPTS="-W --keep-going"
- name: pytest without coverage
if: matrix.env != 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
pytest
- name: pytest with coverage
if: matrix.env == 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
pytest --cov=improver --cov-report xml:coverage.xml
- name: codacy upload
if: env.CODACY_PROJECT_TOKEN && matrix.env == 'environment_a'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
python-codacy-coverage -v -r coverage.xml
env:
CODACY_PROJECT_TOKEN: ${{ secrets.CODACY_PROJECT_TOKEN }}
- name: codecov upload
uses: codecov/codecov-action@v1
if: matrix.env == 'environment_a'
Safety-Bandit:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [latest]
if: secrets.SCHEDULED_JOBS == 'yes'
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
id: cache
env:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
conda info
conda list
- name: safety
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
safety check || true
- name: bandit
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
bandit -r improver
Type-checking:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
env: [latest]
if: secrets.SCHEDULED_JOBS == 'yes'
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
id: cache
env:
# Increase this value to reset cache
CACHE_NUMBER: 2
with:
path: /usr/share/miniconda/envs/im${{ matrix.env }}
key: ${{ format('{0}-conda-improver-{1}-{2}-{3}', runner.os, env.CACHE_NUMBER, matrix.env, hashFiles(format('envs/{0}.yml', matrix.env))) }}
- name: conda env update
if: steps.cache.outputs.cache-hit != 'true'
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda install -c conda-forge mamba
mamba env update -q --file envs/${{ matrix.env }}.yml --name im${{ matrix.env }}
- name: conda info
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
conda info
conda list
- name: mypy
run: |
source '/usr/share/miniconda/etc/profile.d/conda.sh'
conda activate im${{ matrix.env }}
mypy improver || true
Loading