Skip to content
This repository was archived by the owner on Aug 19, 2024. It is now read-only.

Commit 58c9b5d

Browse files
rm3lgazarenkov
rm3l
and
gazarenkov
authored
Add E2E tests using our examples against real clusters (#204)
* Add E2E tests against our examples on real clusters - Do not error out when deleting a non-existing namespace - Stream command output to the GinkgoWriter in real-time as well This allows following what happens when calling potentially long-running commands - Implement airgap test mode - Ignore error when creating a namespace that already exists - Allow to use existing mirror registry in airgap scenario - Extract constants for test modes - Add documentation - Find an easier way to determine the IMG variable, using the Makefile - Add more examples to README.md - Add note about clusters with hosted control planes - Support k3d clusters - Support Minikube clusters - Load image into local clusters using an archive instead This allows this logic to be agnostic to the container engine used to build the image. We rely on the container image to export the image to an archive ('{podman,docker} image save'). - Run E2E test nightly on main and release branch * Try running E2E tests on PRs by leveraging the already built operator image * Revert "Try running E2E tests on PRs by leveraging the already built operator image" This reverts commit fc87e04. * Check if image exists locally before trying to export an archive If not, try to pull it automatically. This would avoid having to manually pull it. * Update README.md Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com> * Ignore gosec warnings in test code Those are not used in production * Clarify in README that a connection to a cluster in the current kubeconfig is needed * Increase timeout when waiting for controller to be up On fresh clusters, 1 minute might be too short * fixup! Clarify in README that a connection to a cluster in the current kubeconfig is needed --------- Co-authored-by: Gennady Azarenkov <gazarenkov@gmail.com>
1 parent 200cf12 commit 58c9b5d

File tree

8 files changed

+1110
-2
lines changed

8 files changed

+1110
-2
lines changed

‎.github/workflows/nightly.yaml

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
name: Nightly checks
2+
3+
on:
4+
# workflow_dispatch so that it can be triggered manually if needed
5+
workflow_dispatch:
6+
schedule:
7+
- cron: "34 23 * * *"
8+
9+
jobs:
10+
e2e-tests:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
fail-fast: false
14+
matrix:
15+
branch: [ main, 1.1.x ]
16+
name: E2E Tests - ${{ matrix.branch }}
17+
concurrency:
18+
group: ${{ github.workflow }}-${{ matrix.branch }}
19+
cancel-in-progress: true
20+
env:
21+
CONTAINER_ENGINE: podman
22+
steps:
23+
- uses: actions/checkout@v4 # default branch will be checked out by default on scheduled workflows
24+
with:
25+
fetch-depth: 0
26+
27+
- if: ${{ matrix.branch != 'main' }}
28+
name: Checkout ${{ matrix.branch }} branch
29+
run: git switch ${{ matrix.branch }}
30+
31+
# check changes in this commit for regex include and exclude matches; pipe to an env var
32+
- name: Check for changes to build
33+
run: |
34+
# don't fail if nothing returned by grep
35+
set +e
36+
CHANGES="$(git diff --name-only HEAD~1 | \
37+
grep -E "workflows/.+-container-build.yaml|Makefile|bundle/|config/|go.mod|go.sum|.+\.go|docker/|\.dockerignore" | \
38+
grep -v -E ".+_test.go|/.rhdh/")";
39+
echo "Changed files for this commit:"
40+
echo "=============================="
41+
echo "$CHANGES"
42+
echo "=============================="
43+
{
44+
echo 'CHANGES<<EOF'
45+
echo $CHANGES
46+
echo EOF
47+
} >> "$GITHUB_ENV"
48+
49+
- name: Determine built operator image
50+
# run this stage only if there are changes that match the includes and not the excludes
51+
if: ${{ env.CHANGES != '' }}
52+
run: |
53+
SHORT_SHA=$(git rev-parse --short HEAD)
54+
BASE_VERSION=$(grep -E "^VERSION \?=" Makefile | sed -r -e "s/.+= //") # 0.1.0
55+
echo "OPERATOR_IMAGE=quay.io/janus-idp/operator:${BASE_VERSION}-${SHORT_SHA}" >> $GITHUB_ENV
56+
57+
- name: Wait until image exists in registry or timeout is reached
58+
# run this stage only if there are changes that match the includes and not the excludes
59+
if: ${{ env.CHANGES != '' }}
60+
timeout-minutes: 10
61+
run: |
62+
echo "Waiting until operator image is found or timeout expires: ${{ env.OPERATOR_IMAGE }}..."
63+
until ${CONTAINER_ENGINE} image pull "${{ env.OPERATOR_IMAGE }}"; do
64+
sleep 2
65+
echo ...
66+
done
67+
echo "... operator image found: ${{ env.OPERATOR_IMAGE }}."
68+
69+
- name: Start Minikube
70+
# run this stage only if there are changes that match the includes and not the excludes
71+
if: ${{ env.CHANGES != '' }}
72+
uses: medyagh/setup-minikube@606b71970c783154fe49b711486c717f5780f485 # v0.0.15
73+
with:
74+
addons: ingress
75+
76+
- name: Run E2E tests
77+
# run this stage only if there are changes that match the includes and not the excludes
78+
if: ${{ env.CHANGES != '' }}
79+
env:
80+
BACKSTAGE_OPERATOR_TESTS_PLATFORM: minikube
81+
IMG: ${{ env.OPERATOR_IMAGE }}
82+
run: make test-e2e

‎Makefile

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ VERSION ?= 0.2.0
88
# Using docker or podman to build and push images
99
CONTAINER_ENGINE ?= docker
1010

11+
PKGS := $(shell go list ./... | grep -v /tests)
12+
1113
# CHANNELS define the bundle channels used in the bundle.
1214
# Add a new line here if you would like to change its default config. (E.g CHANNELS = "candidate,fast,stable")
1315
# To re-generate a bundle for other specific channels without changing the standard setup, you can:
@@ -127,7 +129,7 @@ vet: ## Run go vet against code.
127129
.PHONY: test
128130
test: manifests generate fmt vet envtest ## Run tests. We need LOCALBIN=$(LOCALBIN) to get correct default-config path
129131
mkdir -p $(LOCALBIN)/default-config && cp config/manager/$(CONF_DIR)/* $(LOCALBIN)/default-config
130-
LOCALBIN=$(LOCALBIN) KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile cover.out
132+
LOCALBIN=$(LOCALBIN) KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $(PKGS) -coverprofile cover.out
131133

132134
##@ Build
133135

@@ -228,6 +230,9 @@ GOSEC_VERSION ?= v2.18.2
228230
GOSEC_FMT ?= sarif # for other options, see https://github.com/securego/gosec#output-formats
229231
GOSEC_OUTPUT_FILE ?= gosec.sarif
230232

233+
GINKGO ?= $(LOCALBIN)/ginkgo
234+
GINKGO_VERSION ?= v2.9.5
235+
231236
KUSTOMIZE_INSTALL_SCRIPT ?= "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
232237
.PHONY: kustomize
233238
kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
@@ -366,3 +371,33 @@ catalog-update: ## Update catalog source in the default namespace for catalogsou
366371
.PHONY: deploy-openshift
367372
deploy-openshift: release-build release-push catalog-update ## Deploy the operator on openshift cluster
368373

374+
# After this time, Ginkgo will emit progress reports, so we can get visibility into long-running tests.
375+
POLL_PROGRESS_INTERVAL := 120s
376+
TIMEOUT ?= 14400s
377+
378+
GINKGO_FLAGS_ALL = $(GINKGO_TEST_ARGS) --randomize-all --poll-progress-after=$(POLL_PROGRESS_INTERVAL) --poll-progress-interval=$(POLL_PROGRESS_INTERVAL) -timeout $(TIMEOUT) --no-color
379+
380+
# Flags for tests that may be run in parallel
381+
GINKGO_FLAGS=$(GINKGO_FLAGS_ALL) -nodes=$(TEST_EXEC_NODES)
382+
# Flags to run one test per core.
383+
GINKGO_FLAGS_AUTO = $(GINKGO_FLAGS_ALL) -p
384+
ifdef TEST_EXEC_NODES
385+
TEST_EXEC_NODES := $(TEST_EXEC_NODES)
386+
else
387+
TEST_EXEC_NODES := 1
388+
endif
389+
390+
.PHONY: ginkgo
391+
ginkgo: $(GINKGO) ## Download Ginkgo locally if necessary.
392+
$(GINKGO): $(LOCALBIN)
393+
test -s $(LOCALBIN)/ginkgo || GOBIN=$(LOCALBIN) go install github.com/onsi/ginkgo/v2/ginkgo@$(GINKGO_VERSION)
394+
395+
.PHONY: test-e2e
396+
test-e2e: ginkgo ## Run end-to-end tests. See the 'tests/e2e/README.md' file for more details.
397+
$(GINKGO) $(GINKGO_FLAGS) tests/e2e
398+
399+
show-img:
400+
@echo -n $(IMG)
401+
402+
show-container-engine:
403+
@echo -n $(CONTAINER_ENGINE)

‎examples/rhdh-cr-with-app-configs.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: rhdh.redhat.com/v1alpha1
22
kind: Backstage
33
metadata:
4-
name: my-backstage-app-with-app-config
4+
name: bs-app-config
55
spec:
66
database:
77
enableLocalDb: true

‎tests/e2e/README.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
## End-to-end tests
2+
3+
The end-to-end tests use the [Ginkgo framework](https://onsi.github.io/ginkgo/) and allow to test the operator against a real cluster in the following scenarios:
4+
- building and deploying the operator image off of the current code
5+
- using a specific image or a specific downstream build
6+
7+
Deployment of the operator itself can be done by:
8+
- deploying with or without OLM,
9+
- or deploying the downstream bundle in both online and air-gapped scenarios
10+
11+
To run the end-to-end tests, make sure you have an active connection to a cluster in your current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) and run:
12+
```shell
13+
# Check your current context
14+
$ kubectl config current-context
15+
$ make test-e2e
16+
```
17+
18+
### Configuration
19+
20+
The behavior is configurable using the following environment variables:
21+
22+
| Name | Type | Description | Default value | Example |
23+
|------------------------------------------------------------------------------------------------|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------|
24+
| `BACKSTAGE_OPERATOR_TEST_MODE` | string | The test mode:<br>- if not set, it will call `make deploy`<br>- `olm`: it will call `make deploy-olm`<br>- `rhdh-latest` or `rhdh-next`: it will install the operator using the [`install-rhdh-catalog-source.sh`](../../.rhdh/scripts/install-rhdh-catalog-source.sh) script<br>- `rhdh-airgap`: it will install the operator using the [`prepare-restricted-environment.sh`](../../.rhdh/scripts/prepare-restricted-environment.sh) script. | | `rhdh-latest` |
25+
| `IMG` (or any variables from the Makefile that are used by `make deploy` or `make deploy-olm`) | string | The image to use. Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm` | `VERSION` defined in [`Makefile`](../../Makefile) | `quay.io/janus-idp/operator:0.0.1-latest` |
26+
| `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` | bool | If set to `true`, it will build the operator image with `make image-build`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` |
27+
| `BACKSTAGE_OPERATOR_TESTS_PUSH_IMAGES` | bool | If set to `true`, it will push the operator image with `make image-push`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` |
28+
| `BACKSTAGE_OPERATOR_TESTS_PLATFORM` | string | The platform type, to directly load the operator image if supported instead of pushing it.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`.br>Supported values: [`kind`](#building-and-testing-local-changes-on-kind), [`k3d`](#building-and-testing-local-changes-on-k3d), [`minikube`](#building-and-testing-local-changes-on-minikube) | | `kind` |
29+
| `BACKSTAGE_OPERATOR_TESTS_KIND_CLUSTER` | string | Name of the local KinD cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `kind`. | `kind` | `kind-local-k8s-cluster` |
30+
| `BACKSTAGE_OPERATOR_TESTS_K3D_CLUSTER` | string | Name of the local k3d cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `k3d`. | `k3s-default` | `k3d-local-k8s-cluster` |
31+
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_INDEX_IMAGE` | string | Index image to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `quay.io/rhdh/iib:latest-v4.14-x86_64` | `registry.redhat.io/redhat/redhat-operator-index:v4.14` |
32+
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_OPERATOR_VERSION` | string | Operator version to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `v1.1.0` | `v1.1.0` |
33+
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY` | string | Existing mirror registry to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`<br>. | | `my-registry.example.com` |
34+
35+
### Examples
36+
37+
#### Testing the operator available for the VERSION (default)
38+
39+
In this scenario, you want to run the E2E test suite against the operator image corresponding to the `VERSION` declared in the project [`Makefile`](../../Makefile), which should be publicly available at `quay.io/janus-idp/operator:<VERSION>`.
40+
41+
This is the default behavior.
42+
43+
This should work on any Kubernetes or OpenShift cluster:
44+
45+
```shell
46+
$ make test-e2e
47+
```
48+
49+
#### Testing a specific image (e.g. PR image)
50+
51+
In this scenario, you want to run the E2E test suite against an existing operator image.
52+
53+
This should work on any Kubernetes or OpenShift cluster:
54+
55+
```shell
56+
# if the tag is already published and available at the default location: quay.io/janus-idp/operator
57+
$ make test-e2e VERSION=0.2.0-3d1c1e0
58+
59+
# or you can override the full image repo name
60+
$ make test-e2e IMG=my.registry.example.com/operator:0.2.0-3d1c1e0
61+
```
62+
63+
Note that `VERSION` and `IMG` override the respective variables declared in the project [`Makefile`](../../Makefile).
64+
65+
#### Building and testing local changes on supported local clusters
66+
67+
In this scenario, you are iterating locally, and want to run the E2E test suite against your local changes. You are already using a local cluster like [`kind`](https://kind.sigs.k8s.io/), [`k3d`](https://k3d.io/) or [`minikube`](https://minikube.sigs.k8s.io/docs/), which provide the ability to import images into the cluster nodes.
68+
69+
To do so, you can:
70+
1. set `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` to `true`, which will result in building the operator image from the local changes,
71+
2. and set `BACKSTAGE_OPERATOR_TESTS_PLATFORM` to a supported local cluster, which will result in loading the image built directly in that cluster (without having to push to a separate registry).
72+
73+
##### `kind`
74+
75+
```shell
76+
$ kind create cluster
77+
$ make test-e2e \
78+
BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \
79+
BACKSTAGE_OPERATOR_TESTS_PLATFORM=kind
80+
```
81+
82+
##### `k3d`
83+
84+
```shell
85+
$ k3d cluster create
86+
$ make test-e2e \
87+
BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \
88+
BACKSTAGE_OPERATOR_TESTS_PLATFORM=k3d
89+
```
90+
91+
##### `minikube`
92+
93+
```shell
94+
$ minikube start
95+
$ make test-e2e \
96+
BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \
97+
BACKSTAGE_OPERATOR_TESTS_PLATFORM=minikube
98+
```
99+
100+
#### Testing a specific version using OLM
101+
102+
In this scenario, you want to leverage the [Operator Lifecycle Manager (OLM)](https://olm.operatorframework.io/) to deploy the Operator.
103+
104+
This requires OLM to be installed in the cluster.
105+
106+
```shell
107+
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=olm
108+
```
109+
110+
#### Testing a downstream build of Red Hat Developer Hub (RHDH)
111+
112+
In this scenario, you want to run the E2E tests against a downstream build of RHDH.
113+
114+
This works only against OpenShift clusters. So make sure you are logged in to the OpenShift cluster using the `oc` command. See [Logging in to the OpenShift CLI](https://docs.openshift.com/container-platform/4.14/cli_reference/openshift_cli/getting-started-cli.html#cli-logging-in_cli-developer-commands) for more details.
115+
116+
You can check your current context by running `oc config current-context` or `kubectl config current-context`.
117+
118+
If testing a CI build, please follow the instructions in [Installing CI builds of Red Hat Developer Hub](../../.rhdh/docs/installing-ci-builds.adoc) to add your Quay token to the cluster.
119+
120+
```shell
121+
# latest
122+
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-latest
123+
124+
# or next
125+
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-next
126+
```
127+
128+
#### Airgap testing of Red Hat Developer Hub (RHDH)
129+
130+
In this scenario, you want to run the E2E tests against an OpenShift cluster running in a restricted network. For this, the command below will make sure to prepare it by copying all the necessary images to a mirror registry, then deploy the operator.
131+
132+
Make sure you are logged in to the OpenShift cluster using the `oc` command. See [Logging in to the OpenShift CLI](https://docs.openshift.com/container-platform/4.14/cli_reference/openshift_cli/getting-started-cli.html#cli-logging-in_cli-developer-commands) for more details.
133+
134+
You can check your current context by running `oc config current-context` or `kubectl config current-context`.
135+
136+
Also make sure to read the prerequisites in [Installing Red Hat Developer Hub (RHDH) in restricted environments](../../.rhdh/docs/airgap.adoc).
137+
138+
```shell
139+
# if you want to have a mirror registry to be created for you as part of the airgap environment setup
140+
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap
141+
142+
# or if you already have a mirror registry available and reachable from within your cluster
143+
$ make test-e2e \
144+
BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap \
145+
BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY=my-registry.example.com
146+
```

0 commit comments

Comments
 (0)