Skip to content

Commit

Permalink
Try splitting up integration tests (#25312)
Browse files Browse the repository at this point in the history
Follow up to #25271 and #21774

Integration test failures will happen much faster of they occur, but now
the bottleneck is the `fleetctl` test suite.

It's trivial to continue splitting tests up now. We should look into
creating an action that checks that mock generation is up-to-date, run
it before all the tests, and then remove the mock generation step from
each test step. That would save about a minute and a half of runtime
from each test and help offset the cost of splitting the tests up.

![ci runtime
breakdown](https://github.com/user-attachments/assets/057b8ee1-782c-4e1f-9486-42c7d1169c81)
![ci runtime
max](https://github.com/user-attachments/assets/3a26995f-d9cb-490b-84d9-1a7fbb3cd6b3)

![image](https://github.com/user-attachments/assets/b4c888c8-867f-4bdd-9b69-0dc20d0d202a)
  • Loading branch information
dantecatalfamo authored Jan 10, 2025
1 parent d27d679 commit 8c338a1
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 4 deletions.
14 changes: 10 additions & 4 deletions .github/workflows/test-go.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
test-go:
strategy:
matrix:
suite: ["integration", "core", "mysql", "fleetctl", "vuln"]
suite: ["integration-core", "integration-enterprise", "integration-mdm", "core", "mysql", "fleetctl", "vuln"]
os: [ubuntu-latest]
mysql: ["mysql:8.0.36", "mysql:8.4.3", "mysql:9.1.0"] # make sure to update supported versions docs when this changes
isCron:
Expand Down Expand Up @@ -120,9 +120,15 @@ jobs:
if [[ "${{ matrix.suite }}" == "core" ]]; then
CI_TEST_PKG=main
RUN_TESTS_ARG='-skip=^TestIntegrations'
elif [[ "${{ matrix.suite }}" == "integration" ]]; then
CI_TEST_PKG=main
RUN_TESTS_ARG='-run=^TestIntegrations'
elif [[ "${{ matrix.suite }}" == "integration-core" ]]; then
CI_TEST_PKG=integration
RUN_TESTS_ARG='-run=^TestIntegrations -skip "^(TestIntegrationsMDM|TestIntegrationsEnterprise)"'
elif [[ "${{ matrix.suite }}" == "integration-mdm" ]]; then
CI_TEST_PKG=integration
RUN_TESTS_ARG='-run=^TestIntegrationsMDM'
elif [[ "${{ matrix.suite }}" == "integration-enterprise" ]]; then
CI_TEST_PKG=integration
RUN_TESTS_ARG='-run=^TestIntegrationsEnterprise'
else
CI_TEST_PKG="${{ matrix.suite }}"
RUN_TESTS_ARG=''
Expand Down
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ dlv_test_pkg_to_test := $(addprefix github.com/fleetdm/fleet/v4/,$(PKG_TO_TEST))
DEFAULT_PKG_TO_TEST := ./cmd/... ./ee/... ./orbit/pkg/... ./orbit/cmd/orbit ./pkg/... ./server/... ./tools/...
ifeq ($(CI_TEST_PKG), main)
CI_PKG_TO_TEST=$(shell go list ${DEFAULT_PKG_TO_TEST} | grep -v "server/datastore/mysql" | grep -v "cmd/fleetctl" | grep -v "server/vulnerabilities" | sed -e 's|github.com/fleetdm/fleet/v4/||g')
else ifeq ($(CI_TEST_PKG), integration)
CI_PKG_TO_TEST="server/service"
else ifeq ($(CI_TEST_PKG), mysql)
CI_PKG_TO_TEST="server/datastore/mysql/..."
else ifeq ($(CI_TEST_PKG), fleetctl)
Expand Down

0 comments on commit 8c338a1

Please sign in to comment.