Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pants ci: add integration tests jobs #6273

Merged
merged 8 commits into from
Dec 5, 2024
Merged

pants ci: add integration tests jobs #6273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c1bf132
pants ci: Consolidate env var definitions (DRY)
cognifloyd Nov 11, 2024
4e0fd65
pants ci: add ST2_CI var to enable ci-only tests
cognifloyd Nov 7, 2024
a7cd72d
pants ci: add integration tests job
cognifloyd Nov 2, 2024
8fb0deb
pants ci: orquesta (uses=st2cluster) itests
cognifloyd Nov 6, 2024
85da1cb
pants ci: copy Reconfigure RabbitMQ task into pants workflow
cognifloyd Nov 7, 2024
4007f0d
Put bitnami/rabbitmq logs on stdout
cognifloyd Nov 12, 2024
ec6fcb6
pants ci: fix rabbitmq disk space alert
cognifloyd Nov 12, 2024
432a6ca
update changelog entry
cognifloyd Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
278 changes: 252 additions & 26 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,19 @@ on:
# # run every night at midnight
# - cron: '0 0 * * *'

env:
COLUMNS: '120'

# Tell StackStorm that we are indeed in CI mode, using our CI-provider agnostic var.
ST2_CI: 'true'

# GitHub Actions uses the 'runner' user, so use that instead of stanley.
ST2TESTS_SYSTEM_USER: 'runner'

# This is the host:port provided by services.redis
ST2TESTS_REDIS_HOST: '127.0.0.1'
ST2TESTS_REDIS_PORT: '6379'

jobs:
pants-plugins-tests:
name: 'Pants Plugins Tests (pants runs: pytest) - Python ${{ matrix.python.version-short }}'
Expand Down Expand Up @@ -63,9 +76,6 @@ jobs:
ports:
- 6379:6379/tcp

env:
COLUMNS: '120'

steps:
- name: Checkout repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -99,11 +109,6 @@ jobs:
gha-cache-key: cache0-py${{ matrix.python.version }}

- name: Test pants-plugins
env:
# Github Actions uses the 'runner' user, so use that instead of stanley.
ST2TESTS_SYSTEM_USER: 'runner'
ST2TESTS_REDIS_HOST: '127.0.0.1'
ST2TESTS_REDIS_PORT: '6379'
run: |
pants test pants-plugins/::

Expand Down Expand Up @@ -161,9 +166,6 @@ jobs:
ports:
- 6379:6379/tcp

env:
COLUMNS: '120'

steps:
- name: Checkout repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -197,11 +199,6 @@ jobs:
gha-cache-key: cache0-py${{ matrix.python.version }}

- name: Unit Tests
env:
# Github Actions uses the 'runner' user, so use that instead of stanley.
ST2TESTS_SYSTEM_USER: 'runner'
ST2TESTS_REDIS_HOST: '127.0.0.1'
ST2TESTS_REDIS_PORT: '6379'
run: >
pants
--python-bootstrap-search-path=[]
Expand Down Expand Up @@ -257,9 +254,6 @@ jobs:
ports:
- 6379:6379/tcp

env:
COLUMNS: '120'

steps:
- name: Checkout repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -293,11 +287,6 @@ jobs:
gha-cache-key: cache0-py${{ matrix.python.version }}

- name: Pack Tests
env:
# Github Actions uses the 'runner' user, so use that instead of stanley.
ST2TESTS_SYSTEM_USER: 'runner'
ST2TESTS_REDIS_HOST: '127.0.0.1'
ST2TESTS_REDIS_PORT: '6379'
run: >
pants
--python-bootstrap-search-path=[]
Expand All @@ -312,7 +301,243 @@ jobs:
path: .pants.d/pants.log
if: always() # We want the log even on failures.

#integration-tests: TODO: run integration tests
integration-tests:
name: 'Integration Tests (pants runs: pytest) - Python ${{ matrix.python.version-short }}'
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
# NOTE: We need to use full Python version as part of Python deps cache key otherwise
# setup virtualenv step will fail.
python:
- {version-short: '3.8', version: '3.8.10'}
- {version-short: '3.9', version: '3.9.14'}

services:
mongo:
image: mongo:7.0
ports:
- 27017:27017

# In GHA, these services are started first before the code is checked out.
# We use bitnami images to facilitate reconfiguring RabbitMQ during integration tests.
# We rely on custom config and SSL certs that are in the repo.
# Many images require config in env vars (which we can't change during the test job)
# or they require config in entrypoint args (which we can't override for GHA services)
# bitnami builds ways to get config files from mounted volumes.
rabbitmq:
image: bitnami/rabbitmq:3.8
volumes:
- /home/runner/rabbitmq_conf:/bitnami/conf # RABBITMQ_MOUNTED_CONF_DIR
env:
# tell bitnami/rabbitmq to enable this by default
RABBITMQ_PLUGINS: rabbitmq_management
RABBITMQ_USERNAME: guest
RABBITMQ_PASSWORD: guest
RABBITMQ_LOGS: '-'
# bitnami's default relative limit was too high
RABBITMQ_DISK_FREE_ABSOLUTE_LIMIT: 50MB

# These are strictly docker options, not entrypoint args (GHA restriction)
options: >-
--name rabbitmq
ports:
# These 6 ports are exposed by bitnami/rabbitmq (see https://www.rabbitmq.com/networking.html#ports)
# host_port:container_port/protocol
- 5671:5671/tcp # AMQP SSL port
- 5672:5672/tcp # AMQP standard port
- 15672:15672/tcp # Management: HTTP, CLI
#- 15671:15671/tcp # Management: SSL port
#- 25672:25672/tcp # inter-node or CLI
#- 4369:4369/tcp # epmd

redis:
# Docker Hub image
image: redis
# Set health checks to wait until redis has started
options: >-
--name "redis"
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379/tcp

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
# a test uses a submodule, and pants needs access to it to calculate deps.
submodules: 'recursive'
# sadly, the submodule will only have fetch-depth=1, which is what we want
# for st2.git, but not for the submodules. We still want actions/checkout
# to do the initial checkout, however, so that it adds auth for fetching
# in the submodule.

- name: Fetch repository submodules
run: |
git submodule status
git submodule foreach 'git fetch --all --tags'
git submodule foreach 'git tag'

- name: 'Set up Python (${{ matrix.python.version }})'
id: python
uses: actions/setup-python@v5
with:
python-version: '${{ matrix.python.version }}'

- name: Cache and Install APT Dependencies
uses: ./.github/actions/apt-packages

- name: Initialize Pants and its GHA caches
uses: ./.github/actions/init-pants
with:
# To ignore a bad cache, bump the cache* integer.
gha-cache-key: cache0-py${{ matrix.python.version }}

# This is only required for st2common/tests/integration/test_rabbitmq_ssl_listener.py
- name: Reconfigure RabbitMQ
# bitnami image allows (see bitnami/rabbitmq readme):
# Here we're copying a rabbitmq.config file which won't do anything.
# We need to switch to custom.conf or advanced.config.
timeout-minutes: 2 # may die if rabbitmq fails to start
run: |
./scripts/github/configure-rabbitmq.sh

- name: Integration Tests
run: >
pants
--python-bootstrap-search-path=[]
--python-bootstrap-search-path=${{ steps.python.outputs.python-path }}
--tag=integration
--tag=-st2cluster
test '::'

- name: Upload pants log
uses: actions/upload-artifact@v4
with:
name: pants-log-py${{ matrix.python.version }}-integration-tests
path: .pants.d/pants.log
if: always() # We want the log even on failures.

integration-st2cluster-tests:
name: 'Integration Tests (Full Cluster)- Python ${{ matrix.python.version-short }}'
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
# NOTE: We need to use full Python version as part of Python deps cache key otherwise
# setup virtualenv step will fail.
python:
- {version-short: '3.8', version: '3.8.10'}
- {version-short: '3.9', version: '3.9.14'}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isn't python 3.10 being tested here also?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't merged the changes that add 3.10 to any of the other CI jobs yet. I plan to work on that once I've got the self check running with pants.


services:
mongo:
image: mongo:7.0
ports:
- 27017:27017

rabbitmq:
image: rabbitmq:3.8-management
options: >-
--name rabbitmq
ports:
- 5671:5671/tcp # AMQP SSL port
- 5672:5672/tcp # AMQP standard port
- 15672:15672/tcp # Management: HTTP, CLI

redis:
# Docker Hub image
image: redis
# Set health checks to wait until redis has started
options: >-
--name "redis"
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379/tcp

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
# a test uses a submodule, and pants needs access to it to calculate deps.
submodules: 'recursive'
# sadly, the submodule will only have fetch-depth=1, which is what we want
# for st2.git, but not for the submodules. We still want actions/checkout
# to do the initial checkout, however, so that it adds auth for fetching
# in the submodule.

- name: Fetch repository submodules
run: |
git submodule status
git submodule foreach 'git fetch --all --tags'
git submodule foreach 'git tag'

- name: 'Set up Python (${{ matrix.python.version }})'
id: python
uses: actions/setup-python@v5
with:
python-version: '${{ matrix.python.version }}'

- name: Cache and Install APT Dependencies
uses: ./.github/actions/apt-packages

- name: Initialize Pants and its GHA caches
uses: ./.github/actions/init-pants
with:
# To ignore a bad cache, bump the cache* integer.
gha-cache-key: cache0-py${{ matrix.python.version }}

- name: Export virtualenv to run Dev ST2 Cluster
run: >
pants
--python-bootstrap-search-path=[]
--python-bootstrap-search-path=${{ steps.python.outputs.python-path }}
export
--resolve=st2

- name: Launch Dev ST2 Cluster
env:
VIRTUALENV_DIR: ./dist/export/python/virtualenvs/st2/${{ steps.python.outputs.python-version }}
run: |
sudo -E ./scripts/github/prepare-integration.sh

- name: Integration Tests
env:
ST2_CI_RUN_ORQUESTA_PAUSE_RESUME_TESTS: 'true'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With all of the recent changes to use redis and to make running tests in parallel safer, and with pants+pytest running the tests, I attempted to run tests that we've been skipping even in CI for a while. Happily, they seem much more reliable now, so this line re-enables the tests in pants+pytest CI.

run: >
pants
--python-bootstrap-search-path=[]
--python-bootstrap-search-path=${{ steps.python.outputs.python-path }}
--tag=integration
--tag=st2cluster
test '::'

- name: Compress Service Logs Before upload
if: ${{ failure() }}
run: |
./tools/launchdev.sh stop # stop st2 before collecting logs
tar cvzpf logs.tar.gz logs/*

- name: Upload StackStorm services Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: logs-py${{ matrix.python-version }}-st2cluster-integration-tests
path: logs.tar.gz
retention-days: 7

- name: Upload pants log
uses: actions/upload-artifact@v4
with:
name: pants-log-py${{ matrix.python.version }}-st2cluster-integration-tests
path: .pants.d/pants.log
if: always() # We want the log even on failures.

set_merge_ok:
name: Set Merge OK (Tests)
Expand All @@ -321,7 +546,8 @@ jobs:
- pants-plugins-tests
- unit-tests
- pack-tests
#- integration-tests
- integration-tests
- integration-st2cluster-tests
outputs:
merge_ok: ${{ steps.set_merge_ok.outputs.merge_ok }}
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Added
working on StackStorm, improve our security posture, and improve CI reliability thanks in part
to pants' use of PEX lockfiles. This is not a user-facing addition.
#6118 #6141 #6133 #6120 #6181 #6183 #6200 #6237 #6229 #6240 #6241 #6244 #6251 #6253
#6254 #6258 #6259 #6260 #6269 #6275 #6279 #6278 #6282 #6283
#6254 #6258 #6259 #6260 #6269 #6275 #6279 #6278 #6282 #6283 #6273
Contributed by @cognifloyd
* Build of ST2 EL9 packages #6153
Contributed by @amanda11
Expand Down
3 changes: 3 additions & 0 deletions pants.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,9 @@ extra_env_vars = [
# "ST2_COORDINATION__URL", # Tests will override this with one of:
# "redis://{ST2TESTS_REDIS_HOST}:{ST2TESTS_REDIS_PORT}?namespace=_st2_test{ST2TESTS_PARALLEL_SLOT}
# "zake://"
# CI-specific vars
"ST2_CI",
"ST2_CI_RUN_ORQUESTA_PAUSE_RESUME_TESTS",
Comment on lines +261 to +262
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These vars are used to skip some integration tests unless running in CI.

]
# 10 min should be more than enough even for integration tests.
timeout_default = 600 # seconds
Expand Down
Loading