Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Core: Generate xUnit Reports for e2e tests #962

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
+36 −8
Open

Core: Generate xUnit Reports for e2e tests #962

Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
bba8838
added a way to generate reports
May 3, 2024
754a54b
install go-unit-report
May 3, 2024
45853c2
Merge branch 'master' into add-reporting
May 3, 2024
b62d260
Merge branch 'codeready-toolchain:master' into add-reporting
May 13, 2024
994ee26
check and install go-junit-report
May 13, 2024
9a4f036
another try
May 13, 2024
3608890
another try 2
May 13, 2024
cfc77de
add another path
May 13, 2024
ba52cf0
changed the base image
May 13, 2024
89aa8f9
Merge branch 'master' into add-reporting
May 15, 2024
54d88be
another try with path
May 15, 2024
66d7085
try again with different base
May 15, 2024
5e78253
Update make/test.mk
May 15, 2024
dd88f5b
Merge branch 'master' into add-reporting
May 15, 2024
c6b6145
added path
May 15, 2024
38e2527
Update make/test.mk
May 16, 2024
1261cef
Update test.mk
May 16, 2024
c4c1e1f
Merge branch 'master' into add-reporting
May 16, 2024
c566e6c
remove check go-junit-report
May 16, 2024
395097d
reinstall before trigger
May 16, 2024
8f42fb8
Update make/test.mk
May 21, 2024
bb08a5c
Merge branch 'master' into add-reporting
May 21, 2024
bd89da5
add metrics report
May 21, 2024
b6edfad
tidy up
May 22, 2024
a176b82
Revert Dockerfile change
May 22, 2024
1202053
Merge branch 'master' into add-reporting
May 23, 2024
1162d6d
Merge branch 'master' into add-reporting
Jun 14, 2024
da34df6
move the binary to /tmp
Jun 19, 2024
40c634c
run only on openshift ci
Jun 19, 2024
9768327
run only on openshift ci
Jun 20, 2024
1ecbfe3
Fix Pipe everyting
Jun 20, 2024
113cd51
Merge branch 'master' into add-reporting
Jun 20, 2024
466d6a5
Merge branch 'master' into add-reporting
Jun 21, 2024
0438d14
move test output to /tmp
Jun 24, 2024
0519d8e
move test output to /tmp 2
Jun 24, 2024
8732398
generate rp if failed
Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 36 additions & 8 deletions make/test.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ E2E_PARALLELISM=1

TESTS_RUN_FILTER_REGEXP ?= ""

REPORT_PORTAL_DIR := rp_preproc/results
TEST_OUTPUT_FILE=test_output.log

.PHONY: test-e2e
## Run the e2e tests
test-e2e: INSTALL_OPERATOR=true
Expand Down Expand Up @@ -66,13 +69,13 @@ verify-migration-and-deploy-e2e: prepare-projects e2e-deploy-latest e2e-migratio
.PHONY: e2e-migration-setup
e2e-migration-setup:
@echo "Setting up the environment before testing the operator migration..."
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/migration/setup"
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/migration/setup" REPORT_NAME="xunit_e2e_migration_setup.xml"
@echo "Environment successfully setup."

.PHONY: e2e-migration-verify
e2e-migration-verify:
@echo "Updating operators and verifying resources..."
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/migration/verify"
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/migration/verify" REPORT_NAME="xunit_e2e_migration_verify.xml"
@echo "Migration tests successfully finished"

.PHONY: e2e-deploy-latest
Expand Down Expand Up @@ -141,19 +144,19 @@ test-e2e-registration-local:
.PHONY: e2e-run-parallel
e2e-run-parallel:
@echo "Running e2e tests in parallel..."
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/e2e/parallel" E2E_PARALLELISM=100
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/e2e/parallel" E2E_PARALLELISM=100 REPORT_NAME="xunit_e2e_parallel.xml"
@echo "The parallel e2e tests successfully finished"

.PHONY: e2e-run
e2e-run:
@echo "Running e2e sequential tests..."
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/e2e"
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/e2e" REPORT_NAME="xunit_e2e.xml"
@echo "The e2e sequential tests successfully finished"

.PHONY: e2e-run-metrics
e2e-run-metrics:
@echo "Running e2e metrics tests..."
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/metrics"
$(MAKE) execute-tests MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} TESTS_TO_EXECUTE="./test/metrics" REPORT_NAME="xunit_e2e_metrics.xml"
@echo "The e2e metrics tests successfully finished"

.PHONY: execute-tests
Expand All @@ -163,9 +166,17 @@ execute-tests:
@echo "Status of ToolchainStatus"
-oc get ToolchainStatus -n ${HOST_NS} -o yaml
@echo "Starting test $(shell date)"
MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast || \
($(MAKE) print-logs HOST_NS=${HOST_NS} MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} && exit 1)

ifneq ($(OPENSHIFT_BUILD_NAMESPACE),)
$(MAKE) check-go-junit-report
MatousJobanek marked this conversation as resolved.
Show resolved Hide resolved
else
@echo "Skipping Go Junit report check and install"
endif
@echo "Running tests"
MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} \
go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee /tmp/test_output.log; \
STATUS=$$?; \
$(MAKE) generate-report REPORT_NAME=${REPORT_NAME}; \
if [ $$STATUS -ne 0 ]; then $(MAKE) print-logs && exit 1; fi
Comment on lines +175 to +179
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tested it and it doesn't work as expected - if one of the test suites fails, then it keeps running with the following test suites, but it should stop instead

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my guess is that STATUS=$$? takes the exit value from the tee command that is executed "after" the test as part of the pipe

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of changing to STATUS=${PIPESTATUS[0]}; \ to capture the exit status of go test

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that could work, I didn't know about PIPESTATUS 👍

.PHONY: print-logs
print-logs:
@echo "Time: $(shell date)"
Expand All @@ -181,6 +192,23 @@ else
$(MAKE) print-local-debug-info HOST_NS=${HOST_NS} MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS}
endif

.PHONY: check-go-junit-report

check-go-junit-report:
@command -v /tmp/go-junit-report >/dev/null 2>&1 || { echo "go-junit-report is not installed. Installing..."; GOBIN=/tmp go install github.com/jstemmer/go-junit-report/v2@latest; }
@echo "go-junit-report version:" && /tmp/go-junit-report -version

.PHONY: generate-report
generate-report:
@echo "Generating report"
ifneq ($(OPENSHIFT_BUILD_NAMESPACE),)
mkdir -p ${ARTIFACT_DIR}/${REPORT_PORTAL_DIR}
/tmp/go-junit-report < /tmp/test_output.log > ${ARTIFACT_DIR}/${REPORT_PORTAL_DIR}/${REPORT_NAME}
@echo "xUnit Report ${REPORT_NAME} Generation Successful"
else
@echo "Skipping Report as it's a local run"
endif

.PHONY: deploy-e2e-local-and-print-local-debug
deploy-e2e-local-and-print-local-debug: deploy-e2e-local print-local-debug-info

Expand Down
Loading