Skip to content

Latest commit

 

History

History
301 lines (231 loc) · 14.4 KB

README.md

File metadata and controls

301 lines (231 loc) · 14.4 KB

Antrea CI: Jenkins

Reasons for Jenkins

We have tests as Github Actions but Jenkins allows tests running on a cluster of multiple nodes and offers better environment setup options.

Jenkins on cloud

At the moment these Jenkins jobs are running on VMC (VMware on AWS). As a result, all jobs' results and details are available publicly here. We are using Cluster API for vSphere (CAPV) for creating and managing workload clusters. The management cluster is a kind cluster on Jenkins node. For each job build, a completely new workload cluster will be created by this management cluster. As soon as the build finishes, the cluster should be deleted. This ensures that all tests are run on a clean testbed.

List of Jenkins jobs

Build Status Build Status Build Status

#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase e2e --registry ${DOCKER_REGISTRY}
  • ipv6-ds-conformance: conformance tests in a dual-stack setup.
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase conformance --registry ${DOCKER_REGISTRY}
  • ipv6-ds-networkpolicy: NetworkPolicy tests in a dual-stack setup.
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase networkpolicy --registry ${DOCKER_REGISTRY}
  • ipv6-only-e2e: e2e tests in an IPv6 only setup.
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase e2e --registry ${DOCKER_REGISTRY}
  • ipv6-only-conformance: conformance tests in an IPv6 only setup.
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase conformance --registry ${DOCKER_REGISTRY}
  • ipv6-only-networkpolicy: NetworkPolicy tests in an IPv6 only setup.
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase networkpolicy --registry ${DOCKER_REGISTRY}
  • Windows e2e: e2e tests in a Windows setup.
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-e2e --registry ${DOCKER_REGISTRY}
  • Windows conformance: community tests on Windows cluster with OVS daemons running inside a container, focusing on "conformance|sig-windows" and "sig-network", and skipping "LinuxOnly", "Slow", "Serial", "Disruptive", "Flaky", "Feature", "sig-cli", "sig-storage", "sig-auth", "sig-api-machinery", "sig-apps", "sig-node", "Privileged".
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-conformance --registry ${DOCKER_REGISTRY}
  • Windows network policy: community tests on Windows cluster, with OVS daemons running inside a container focusing on "Feature:NetworkPolicy".
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase windows-networkpolicy --registry ${DOCKER_REGISTRY}
  • Multicast e2e: e2e tests in a multicast cluster
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase multicast-e2e --registry ${DOCKER_REGISTRY}
  • Flexible-ipam e2e: e2e tests in a flexible-ipam cluster
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase e2e --registry ${DOCKER_REGISTRY} --testbed-type "flexible-ipam"
  • Kind conformance: conformance tests in a kind cluster
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase conformance --registry ${DOCKER_REGISTRY} --testbed-type "kind" --kind-cluster-name "${{JOB_NAME}}-${{BUILD_NUMBER}}"
  • Kind NetworkPolicy: NetworkPolicy tests in a kind cluster
#!/bin/bash
set -e
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test.sh --testcase networkpolicy --registry ${DOCKER_REGISTRY} --testbed-type "kind" --kind-cluster-name "${{JOB_NAME}}-${{BUILD_NUMBER}}"
  • "/stop-all-jobs": Trigger '/stop-all-jobs' to stop stale running or waiting jobs related to a PR, and for now this feature is enabled only for capv-related jobs.
#!/bin/bash
set -e
./ci/jenkins/stop-stale-jobs.sh --pull-request "${{ghprbPullId}}" --jenkins "${{JENKINS_URL}}"

  • whole-conformance [daily]: community tests using sonobuoy, with certified-conformance mode.

  • daily-whole-conformance: daily community tests using sonobuoy, with certified-conformance mode. If build fails, Jenkins will send an email to projectantrea-dev@googlegroups.com for notification.

  • Microsoft Windows conformance: community tests related to Microsoft Windows. It focuses on: "[sig-network].+[Conformance]|[sig-windows]". It skips: "[LinuxOnly]|[Slow]|[Serial]|[Disruptive]|[Flaky]|[Feature:.+]|[sig-cli]|[sig-storage]|[sig-auth]|[sig-api-machinery]|[sig-apps]|[sig-node]|[Privileged]|should be able to change the type from|[sig-network] Services should be able to create a functioning NodePort service [Conformance]".

  • Jenkins jobs validator [gated check-in]: this job only executes for PRs that include changes to ci/jenkins/jobs. It validates the syntax of the jenkins jobs' configuration.

  • Jenkins Windows OVS validator: this job only executes for PRs that include changes to hack/windows/Install-OVS.ps1. It validates if Windows OVS can be installed correctly.

#!/bin/bash
./ci/jenkins/test.sh --testcase windows-install-ovs
  • Rancher e2e: e2e tests in a Rancher cluster
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test-rancher.sh --registry ${DOCKER_REGISTRY} --testcase e2e --cluster-name rancher-test
  • Rancher conformance: conformance tests in a Rancher cluster
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test-rancher.sh --cluster-name rancher-test --testcase conformance --registry ${DOCKER_REGISTRY}
  • Rancher NetworkPolicy: NetworkPolicy tests in a Rancher cluster
#!/bin/bash
DOCKER_REGISTRY="$(head -n1 ci/docker-registry)"
./ci/jenkins/test-rancher.sh --cluster-name rancher-test --testcase networkpolicy --registry ${DOCKER_REGISTRY}

  • EKS conformance/network policy [bi-daily] community tests on EKS cluster using sonobuoy, focusing on "Conformance" and "Feature:NetworkPolicy", skipping the same regexes as in job conformance above, as well as "NodePort" (See #690).
    Current test environment matrix:

    K8s Version Node Type Node AMI Family Status
    1.24 EC2 t3.medium AmazonLinux2 Build Status

  • GKE conformance/network policy [bi-daily] community tests on GKE cluster using sonobuoy, focusing on "Conformance" and "Feature:NetworkPolicy", skipping the same regexes as in job conformance above.
    Current test environment matrix:

    K8s Version Node OS VPC Native Mode (on by default) Status
    1.25.5 Ubuntu On Build Status

  • AKS conformance/network policy [bi-daily] community tests on AKS cluster using sonobuoy, focusing on "Conformance" and "Feature:NetworkPolicy", skipping the same regexes as in job conformance above.
    Current test environment matrix:

    K8s Version Node Type Node OS Status
    1.24.9 Standard_DS2_v2 Ubuntu 18.04 Build Status

  • matrix-test [weekly]: runs Antrea e2e, K8s Conformance and NetworkPolicy tests, using different combinations of various operating systems and K8s releases.

    K8s Version Node OS Status
    1.17.5 CentOS 7 Build Status
    1.17.5 Photon 3.0 Build Status
    1.18.2 CentOS 7 Build Status
    1.18.2 Photon 3.0 Build Status

If you need to run the K8s community tests locally, you may use the ci/run-k8s-e2e-tests.sh script. It takes care of installing the correct version of sonobuoy and running the correct subset of community tests for Antrea:

  • To run conformance tests: ./run-k8s-e2e-tests.sh --e2e-conformance [--kubeconfig <Kubeconfig>].
  • To run whole conformance tests: ./run-k8s-e2e-tests.sh --e2e-whole-conformance [--kubeconfig <Kubeconfig>].
  • To run network policy tests: ./run-k8s-e2e-tests.sh --e2e-network-policy [--kubeconfig <Kubeconfig>].
  • To run sig-network tests: ./run-k8s-e2e-tests.sh --e2e-sig-network [--kubeconfig <Kubeconfig>].
  • To run a single test by name: ./run-k8s-e2e-tests.sh --e2e-focus <TestRegex> [--kubeconfig <Kubeconfig>].

Requirements

Yaml files under ci/jenkins/jobs can be generated via jenkins-job-builder. If you want to try out the tests on your local jenkins setup, please notice the following requirements:

  • Jenkins setup
    • Plugins: ghprb, throttle-concurrents
  • Install jenkins-job-builder
  • Define your ANTREA_GIT_CREDENTIAL which is the credential for your private repo
  • Define your ghpr_auth, antrea_admin_list, antrea_org_list and antrea_white_list as defaults variables in a separate file
  • Select only one project(projects-cloud.yaml|projects-lab.yaml) file to keep for your needs and remove others

Apply the jobs

Run the command to test if jobs can be generated correctly.

jenkins-jobs test -r ci/jenkins/jobs

Run the command to apply these jobs.

jenkins-jobs update -r ci/jenkins/jobs

Jenkins job updater

To follow GitOps best practices, there is a job-updater job in Jenkins to detect any change in ci/jenkins/jobs for every 15 min. As long as a PR to modify code under that path is merged, Jenkins jobs on cloud should be updated with new code.

Tips for Developer

  • macro.yaml: Use "{{}}" instead of "{}" to escape the "{" in "builder-list-tests", "builder-conformance" and "builder-*-win" because the macro has parameters.
  • While setting up the Rancher testbed, delete the cattle-cluster-agent deployment and use cattle-node-agent because cluster-agent adds extra watchers for all the resources. Antrea Controller counts connected Antrea Agent from watcher connections. Extra watchers lead to wrong Antrea Agent number in AntreaControllerInfo CR.