Add automation to remove old periodics and presubmits for kubevirt/ku…

…bevirt sig jobs (#1605) * Refactor job-copier and presubmit-requirer to commands Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Generalize duplicate code, further refactoring Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Reorganize folder structure Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Polish (reduce duplication, make func names non exported) Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Some more polish Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Remove options logging in flags Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Polish help texts Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Move utility methods to jobconfig Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Restructure kubevirt file layout Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Add remove cmd, implement checks Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Remove jobs implementation ready, functests missing Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Add functest Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Add new remove jobs job, update existing jobs with pr-creator Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Create Makefile for robots/cmd/kubevirt Also add workspace dirty check script. Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Fix formatting and import sorting Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Rename package jobconfig to prowjobconfigs Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Extract constant, remove duplicate check Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Change copy jobs command to use RunE Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Adjust main README to reflect changes Also add a README in the project subdir containing some more detailed information. Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Add test case and code for multiple prow test containers Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Use RunE for remove jobs command Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Remove variable name for return on named variable Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Make github return error instead of handling it itself Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Add test case for where some jobs are missing Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Remove duplication around sha256 Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Remove superfluous variable Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Use RunE for require presubmits command Signed-off-by: Daniel Hiller <dhiller@redhat.com> * Make flag parsing func return an error Signed-off-by: Daniel Hiller <dhiller@redhat.com>
kubevirt · Sep 29, 2021 · a940953 · a940953
1 parent ccbcd8e
commit a940953
Show file tree

Hide file tree

Showing 39 changed files with 8,432 additions and 692 deletions.
diff --git a/README.md b/README.md
@@ -38,26 +38,22 @@ depending on billing alerts. See [README](limiter/README.md)
   information
 
   * `robots/cmd/flakefinder`: Tool to create statistics from failed tests of PRs.
-  See [README](robots/flakefinder/README.md)
-
-  * `robots/cmd/flake-issue-creator`: Tool to create kubevirt/kubevirt issues from
-  flake test results
+  See [README](robots/cmd/flakefinder/README.md)
 
   * `robots/cmd/indexpagecreator`: Creates flakefinder index page
 
-  * `robots/cmd/kubevirt-job-copier`: Creates copies of the SIG jobs (periodic and presubmit) to enable testing kubevirt with kubevirtci
+  * `robots/cmd/kubevirt`: Provides commands to manipulate the SIG jobs (periodic and presubmit) that are testing kubevirt with kubevirtci
+
+    See [README](robots/cmd/kubevirt/README.md)
 
   * `robots/cmd/kubevirtci-bumper`: Tool to automatically bump kubevirtci providers.
-  See [README](robots/kubevirtci-bumper/README.md)
+  See [README](robots/cmd/kubevirtci-bumper/README.md)
 
   * `robots/cmd/kubevirtci-presubmit-creator`: Creates kubevirtci presubmit job
   definitions for new providers
 
   * `robots/cmd/labels-checker`: Checks whether a PR has certain labels
 
-  * `robots/cmd/kubevirt-presubmit-requirer`: Updates kubevirt presubmit jobs
-  definitions to make required jobs related to new providers
-
   * `robots/cmd/uploader`: Tool to mirror bazel dependencies on GCS. See
   [README](plugins/cmd/uploader/README.md)
 

diff --git a/github/ci/prow-deploy/files/jobs/kubevirt/project-infra/project-infra-periodics.yaml b/github/ci/prow-deploy/files/jobs/kubevirt/project-infra/project-infra-periodics.yaml
@@ -549,18 +549,11 @@ periodics:
     securityContext:
       runAsUser: 0
     containers:
-      - image: quay.io/kubevirt/builder:2105121048-a05ef0ee1
-        env:
-          - name: GIT_ASKPASS
-            value: ../project-infra/hack/git-askpass.sh
+      - image: quay.io/kubevirtci/pr-creator:v20210907-eaf738f
         command: [ "/bin/sh" , "-c" ]
         args:
           - |
-            bazel_dir=$(mktemp -d) &&
-            curl --fail -L https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64 --output ${bazel_dir}/bazelisk &&
-            chmod a+x ${bazel_dir}/bazelisk &&
-            export PATH=${PATH}:${bazel_dir} &&
-            hack/git-pr.sh -c "bazelisk run //robots/cmd/kubevirt-presubmit-requirer:kubevirt-presubmit-requirer -- -job-config-path-kubevirt-presubmit=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml -github-token-path= -dry-run=false" -r project-infra -b require-kubevirt-presubmits -T main
+            git-pr.sh -c "bazel run //robots/cmd/kubevirt require presubmits -- -job-config-path-kubevirt-presubmits=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml -github-token-path= -dry-run=false" -r project-infra -b require-kubevirt-presubmits -T main
         volumeMounts:
           - name: token
             mountPath: /etc/github
@@ -591,18 +584,48 @@ periodics:
     securityContext:
       runAsUser: 0
     containers:
-    - image: quay.io/kubevirt/builder:2105121048-a05ef0ee1
+    - image: quay.io/kubevirtci/pr-creator:v20210907-eaf738f
+      env:
+      command: [ "/bin/sh" , "-c" ]
+      args:
+      - |
+        git-pr.sh -c "bazel run //robots/cmd/kubevirt copy jobs -- -job-config-path-kubevirt-presubmits=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml -job-config-path-kubevirt-periodics=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-periodics.yaml -github-token-path= -dry-run=false" -r project-infra -b copy-kubevirt-jobs -T main
+      volumeMounts:
+      - name: token
+        mountPath: /etc/github
+      resources:
+        requests:
+          memory: "200Mi"
+    volumes:
+    - name: token
+      secret:
+        secretName: oauth-token
+
+- name: periodic-kubevirt-job-remover
+  cron: "30 0 * * *"
+  max_concurrency: 1
+  annotations:
+    testgrid-create-test-group: "false"
+  decorate: true
+  decoration_config:
+    timeout: 1h
+    grace_period: 5m
+  extra_refs:
+  - org: kubevirt
+    repo: project-infra
+    base_ref: main
+    workdir: true
+  cluster: prow-workloads
+  spec:
+    securityContext:
+      runAsUser: 0
+    containers:
+    - image: quay.io/kubevirtci/pr-creator:v20210907-eaf738f
       env:
-      - name: GIT_ASKPASS
-        value: ../project-infra/hack/git-askpass.sh
       command: [ "/bin/sh" , "-c" ]
       args:
       - |
-        bazel_dir=$(mktemp -d) &&
-        curl --fail -L https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64 --output ${bazel_dir}/bazelisk &&
-        chmod a+x ${bazel_dir}/bazelisk &&
-        export PATH=${PATH}:${bazel_dir} &&
-        hack/git-pr.sh -c "bazelisk run //robots/cmd/kubevirt-job-copier:kubevirt-job-copier -- -job-config-path-kubevirt-presubmits=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml -job-config-path-kubevirt-periodics=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-periodics.yaml -github-token-path= -dry-run=false" -r project-infra -b copy-kubevirt-jobs -T main
+        git-pr.sh -c "bazel run //robots/cmd/kubevirt remove jobs -- -job-config-path-kubevirt-presubmits=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-presubmits.yaml -job-config-path-kubevirt-periodics=$(pwd)/github/ci/prow-deploy/files/jobs/kubevirt/kubevirt/kubevirt-periodics.yaml -github-token-path= -dry-run=false" -r project-infra -b copy-kubevirt-jobs -T main
       volumeMounts:
       - name: token
         mountPath: /etc/github

diff --git a/go.mod b/go.mod
@@ -18,6 +18,7 @@ require (
 	github.com/prometheus/client_golang v1.11.0
 	github.com/shurcooL/githubv4 v0.0.0-20210725200734-83ba7b4c9228 // indirect
 	github.com/sirupsen/logrus v1.8.1
+	github.com/spf13/cobra v1.1.3 // indirect
 	golang.org/x/mod v0.4.0 // indirect
 	golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
 	google.golang.org/api v0.32.0

diff --git a/hack/check-workspace-dirty.sh b/hack/check-workspace-dirty.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+#
+# Copyright 2021 The KubeVirt Authors.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+if [ -n "$(git status --porcelain)" ]; then
+    echo "Validation failed!"
+    git status
+    exit 1
+fi