Add guidelines to write good conformance tests for k8s

contributor-docs/development/tests/conformance-tests-sonobuoy.md

@@ -0,0 +1,230 @@
+# SCS Conformance Test KaaS Sonobuoy
+
+SovereignCloudStack (SCS) makes use of [Sonobuoy][sonobuoy] as a test framework to run its tests on Kubernetes clusters, which are to be audited for compliance.
+The aim of using this framework is to make the execution of tests on a KaaS infrastructure as simple as possible.
+Hence this reduces the effort required to establish SCS conformity to a minimum.
+
+In short this is achieved by storing all tests in a container image, which can then be called and launched on the clusters managed by sonoobuoy.
+
+> [!NOTE]
+> A more detailed description of why the SCS has decided to use sonobuoys can be found on the corresponding [Decsision Record][sonbouy-decision-record].
+> In addition, sonobuoy is also used as the toolset for executing Kubernetes very own [conformance tests][k8s-conformance].
+
+This document is intended to assist conformance test authors to integrate their tests into the sonobuoy framework.
+This requires a user to write the conformance tests in Golang, as this is the language provided by the framework itself.
+
+## Step-by-step instructions for the development of sonobuoy tests using `docker` and `kind`
+
+> [!NOTE]
+> This guide refers to the brief instructions provided in the [standards repository][scs-sonobuoy-example-guide]
+
+### Prerequisite
+
+* [docker][docker-installation]
+* [kind][kind-installation]
+* [sonobuoy][sonobuoy-installation]
+* [yq][yq-installation]
+* [jq][jq-installation]
+
+```bash
+go install github.com/vmware-tanzu/sonobuoy@latest
+```
+
+### Set up development environment
+
+#### 1. Clone the standard repository containing the conformance tests and navigate to the Sonobuoy example located at `standards/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework`
+
+```bash
+git clone https://github.com/SovereignCloudStack/standards
+cd standards/Tests/kaas/kaas-sonobuoy-tests/
+```
+
+#### 2. Check the prerequisites
+
+Within this directory you will find a `Makefile`, which is also used to create the test images.
+To begin with, this can be used to check whether all the requirements for the development environment are met.
+
+```bash
+make dev-prerequests
+```
+
+#### 3. Create a kind cluster
+
+Once all the prerequisite software is installed, you can proceed by starting an kind cluster using the `Makefile` as follows:
+
+```bash
+make dev-setup
+```
+
+#### 4. Setting environment variables for the development process
+
+Set the number code of the standard you are currently working on.
+
+```bash
+export TESTFUNCTION_CODE=<number code of the standard>
+```
+
+### Create a test
+
+In general, SCS tests for KaaS are derived from standards that define certain expected behaviors and functionalities of Kubernetes.
+As an example for this step-by-step guide, let's assume a scenario in which there is a fictional standard called "scs-0299-v1-example-standard.md".
+Pretend that the fictitious standard here stipulates that at least one pod MUST run in the namespace "namespace-test-a".
+
+> [!NOTE] 
+> The functions and behaviors to be tested MUST be precisely defined in a standard.
+> If you as a developer want to test something that you think is best tested but is not yet part of any standard, you MUST update the standard accordingly.
+
+#### 1. Create example test
+
+The `scs_k8s_tests` directory contains the Golang files that define the tests.
+
+```bash
+cd standards/Tests/kaas/kaas-sonobuoy-tests/scs_k8s_tests
+```
+
+First create a test file according to your standard and adhere to the naming convention accordingly.
+
+* The prefix MUST contain the name of the standard in lower case letters.
+* As a suffix, the file must end with "_test.go".
+
+> [!NOTE]
+> The suffix requirement comes from the go test framework itself. All test files must end with `_test.go`.
+> Otherwise they will not be selected by the test environment.
+
+As an example, we will create a file for the fictitious standard "scs-0299-v1-example-standard.md" as follows:
+
+```bash
+touch scs_0299_v1_example_standard_test.go
+```
+
+In this file, we can define the behavior we want to test for.
+As an example, we test here whether there are more than zero pods in the namespace "namespace-test-a".
+The execution of this test should fail by default as there should be no pods in the namespace and the namespace itself should not exist.
+The aim is to display the results of a failed test so that we can show their interpretation in a later step.
+
+> [!NOTE]
+> Attention!!!: in order for the framework to select the functions for testing, their names must begin with "TEST_" in accordance with the naming convention of the golang test framework.
+> The framework in use is [kubernetes-sigs/e2e-framework][e2e-framework].
+> They also provide examples. Before you start implementing your own tests, you should check whether you can use one of the examples as a starting point for your own implementation.
+> Have a look at: [kubernetes-sigs/e2e-framework/examples][e2e-framework-examples]
+
+As an example of this step-by-step guide, copy the following text into the file `scs_0299_v1_example_standard_test.go`:
+
+```go
+package scs_k8s_tests
+
+import (
+  "context"
+  "testing"
+  //~ "time"
+  //~ "fmt" 
+  //~ plugin_helper "github.com/vmware-tanzu/sonobuoy-plugins/plugin-helper"
+  corev1 "k8s.io/api/core/v1"
+  "sigs.k8s.io/e2e-framework/pkg/envconf"
+  "sigs.k8s.io/e2e-framework/pkg/features"
+)
+
+
+// This function checks if there are any pods in 'namespace-test-a'.
+// The check should fail as there should be no pods present in this namespace.
+// The purpose of this function is to display the handling of failed tests.
+func Test_scs_0299_TestListPodsFailing(t *testing.T) {
+  f := features.New("pod list").Assess(
+    "pods from namespace 'namespace-test-a'",
+    func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
+      var pods corev1.PodList
+      err := cfg.Client().Resources("namespace-test-a").List(context.TODO(), &pods)
+      if err != nil {
+        t.Fatal(err)
+      }
+      t.Logf("found %d pods", len(pods.Items))
+      if len(pods.Items) == 0 {
+        t.Fatal("no pods in namespace 'namespace-test-a'")
+      }
+      return ctx
+    })
+
+  testenv.Test(t, f.Feature())
+}
+
+```
+
+#### 3. Build the test image, upload it to the kind cluster and run it
+
+To create the image, execute the following:
+
+```bash
+make dev-build
+```
+
+This creates the Sonobuoy image and automatically uploads it to the image register of the kind cluster.
+
+You can then run the tests by:
+
+```bash
+make dev-run
+```
+
+Depending on how extensive the tests are, this may take some time.
+To check the current test status manually, you can use the following command:
+
+```bash
+sonobuoy status
+```
+
+#### 4. Retrieving the results
+
+```bash
+make dev-result
+cat results/plugins/scsconformance/results/global/out.json | jq
+cat results/plugins/scsconformance/sonobuoy_results.yaml | yq
+```
+
+Once all tests have been executed successfully, you can read the results and receive feedback.
+You can call up the results as follows:
+
+> [!NOTE]
+> TODO:!!! to be described in more detail
+
+### Clean up after
+
+To clean up the resourcec used for development, you can use the following commands:
+
+```bash
+make dev-clean
+make dev-purge
+```
+
+### (optional) Run all at once:
+
+In addition, you can trigger all processes from above with one command. There is a single target in the Makefile for this:
+
+```bash
+make dev-rerun
+```
+
+### (optional) Run only your testfunctions
+
+The above process always executes all tests. This can take some time.
+Therefore, it is also possible to run only your standard test functions outside the scope of sonobuoy:
+
+```bash
+export TESTFUNCTION_CODE="0299"
+make test-function
+```
+
+> [!NOTE]
+> [e2e-framework-examples]
+
+[sonobuoy]: https://sonobuoy.io/
+[sonbouy-decision-record]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0200-v1-using-sonobuoy-for-kaas-conformance-tests.md
+[k8s-conformance]: https://github.com/cncf/k8s-conformance/blob/master/instructions.md
+[docker-installation]: https://docs.docker.com/engine/install/
+[sonobuoy-installation]: https://sonobuoy.io/docs/v0.57.1/#installation
+[kind-installation]: https://kind.sigs.k8s.io/docs/user/quick-start/#installation
+[scs-sonobuoy-example-guide]: https://github.com/SovereignCloudStack/standards/tree/main/Tests/kaas/kaas-sonobuoy-go-example-e2e-framework#sonobuoy-usage-for-development-of-tests
+
+[yq-installation]:https://github.com/mikefarah/yq/?tab=readme-ov-file#install
+[jq-installation]:https://jqlang.github.io/jq/download/
+[e2e-framework]:https://github.com/kubernetes-sigs/e2e-framework
+[e2e-framework-examples]:https://github.com/kubernetes-sigs/e2e-framework/tree/main/examples