Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add malicious join test #2304

Merged
merged 35 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
09a9520
malicious node join test
msanft Aug 23, 2023
31b92d9
add e2e build tag
msanft Sep 4, 2023
716429b
add namespaces to job apply
msanft Sep 4, 2023
977df9a
fix image and workflow
msanft Sep 5, 2023
f9b7509
fix linter checks
msanft Sep 5, 2023
71f2e9b
build instructions in Dockerfile
msanft Sep 5, 2023
1ebe1cd
only print important flags
msanft Sep 5, 2023
a66caaa
use `malicious-join` namespace
msanft Sep 5, 2023
35fe485
build with bazel
msanft Sep 5, 2023
1db9b25
order imports
msanft Sep 5, 2023
589e5bf
test cases
msanft Sep 5, 2023
6c1e11e
various fixes
msanft Sep 5, 2023
d3b78cd
add missing quotes
msanft Sep 5, 2023
d863fc3
fix typo
msanft Sep 5, 2023
e0b2a84
Update e2e/malicious-join/malicious-join.go
msanft Sep 5, 2023
e4c9442
Update e2e/malicious-join/malicious-join.go
msanft Sep 5, 2023
0f4fd17
use switch case
msanft Sep 5, 2023
ac28a04
update image version
msanft Sep 5, 2023
9274985
fix linter checks
msanft Sep 6, 2023
cc2a430
wip
msanft Sep 6, 2023
758d8dd
various fixes
msanft Sep 6, 2023
369fb06
update buildfiles
msanft Sep 6, 2023
0effada
use workdir
msanft Sep 7, 2023
7b2caf9
fix linter
msanft Sep 7, 2023
b043251
add required permissions
msanft Sep 7, 2023
457f5aa
remove permissions
msanft Sep 11, 2023
c7108e3
remove packages: write permission at step
msanft Sep 11, 2023
34dcaa9
login to registry
msanft Sep 11, 2023
cf07aae
fix typo
msanft Sep 11, 2023
3efa3c0
fix log
msanft Sep 11, 2023
8ea0813
source base lib
msanft Sep 13, 2023
82ec638
fix sourcing order
msanft Sep 13, 2023
d1e1f10
export after definition
msanft Sep 13, 2023
32ad6a5
fix script header
msanft Sep 14, 2023
9088f73
dont exit after -e flag has been set
msanft Sep 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .github/actions/e2e_malicious_join/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Malicious join
description: "Verify that a malicious node cannot join a Constellation cluster."

inputs:
cloudProvider:
description: "The cloud provider the test runs on."
required: true
kubeconfig:
description: "The kubeconfig file for the cluster."
required: true
githubToken:
description: "GitHub authorization token"
required: true

runs:
using: "composite"
steps:
- name: Log in to the Container registry
id: docker-login
uses: ./.github/actions/container_registry_login
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ inputs.githubToken }}

- name: Run malicious join
shell: bash
env:
KUBECONFIG: ${{ inputs.kubeconfig }}
working-directory: e2e/malicious-join
run: |
bazel run //e2e/malicious-join:stamp_and_push
yq eval -i "(.spec.template.spec.containers[0].command) = \
[ \"/malicious-join_bin\", \
\"--js-endpoint=join-service.kube-system:9090\", \
\"--csp=${{ inputs.cloudProvider }}\", \
\"--variant=default\" ]" job.yaml
kubectl create ns malicious-join
kubectl apply -n malicious-join -f job.yaml
kubectl wait -n malicious-join --for=condition=complete --timeout=10m job/malicious-join
kubectl logs -n malicious-join job/malicious-join | tail -n 1 | jq '.'
ALL_TESTS_PASSED=$(kubectl logs -n malicious-join job/malicious-join | tail -n 1 | jq -r '.allPassed')
if [[ "$ALL_TESTS_PASSED" != "true" ]]; then
kubectl logs -n malicious-join job/malicious-join
kubectl logs -n kube-system svc/join-service
exit 1
fi
kubectl delete ns malicious-join
derpsteb marked this conversation as resolved.
Show resolved Hide resolved
14 changes: 11 additions & 3 deletions .github/actions/e2e_test/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ inputs:
description: "Azure credentials authorized to create an IAM configuration."
required: true
test:
description: "The test to run. Can currently be one of [sonobuoy full, sonobuoy quick, autoscaling, lb, perf-bench, verify, recover, nop]."
description: "The test to run. Can currently be one of [sonobuoy full, sonobuoy quick, autoscaling, lb, perf-bench, verify, recover, malicious join, nop]."
required: true
sonobuoyTestSuiteCmd:
description: "The sonobuoy test suite to run."
Expand Down Expand Up @@ -85,7 +85,7 @@ runs:
using: "composite"
steps:
- name: Check input
if: (!contains(fromJson('["sonobuoy full", "sonobuoy quick", "autoscaling", "perf-bench", "verify", "lb", "recover", "nop"]'), inputs.test))
if: (!contains(fromJson('["sonobuoy full", "sonobuoy quick", "autoscaling", "perf-bench", "verify", "lb", "recover", "malicious join", "nop"]'), inputs.test))
shell: bash
run: |
echo "::error::Invalid input for test field: ${{ inputs.test }}"
Expand Down Expand Up @@ -261,10 +261,10 @@ runs:
test: ${{ inputs.test }}
provider: ${{ inputs.cloudProvider }}
isDebugImage: ${{ inputs.isDebugImage }}

#
# Test payloads
#

- name: Nop test payload
if: inputs.test == 'nop'
shell: bash
Expand Down Expand Up @@ -326,3 +326,11 @@ runs:
controlNodesCount: ${{ inputs.controlNodesCount }}
kubeconfig: ${{ steps.constellation-create.outputs.kubeconfig }}
masterSecret: ${{ steps.constellation-create.outputs.masterSecret }}

- name: Run malicious join test
if: inputs.test == 'malicious join'
uses: ./.github/actions/e2e_malicious_join
with:
cloudProvider: ${{ inputs.cloudProvider }}
kubeconfig: ${{ steps.constellation-create.outputs.kubeconfig }}
githubToken: ${{ inputs.githubToken }}
1 change: 1 addition & 0 deletions .github/workflows/e2e-test-manual.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ on:
- "perf-bench"
- "verify"
- "recover"
- "malicious join"
- "nop"
required: true
kubernetesVersion:
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/e2e-test-weekly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,20 @@ jobs:
provider: "azure"
kubernetes-version: "v1.28"

# malicious join test on latest k8s version
- test: "malicious join"
refStream: "ref/main/stream/debug/?"
provider: "gcp"
kubernetes-version: "v1.28"
- test: "malicious join"
refStream: "ref/main/stream/debug/?"
provider: "azure"
kubernetes-version: "v1.28"
- test: "malicious join"
refStream: "ref/main/stream/debug/?"
provider: "aws"
kubernetes-version: "v1.28"

#
# Tests on release-stable refStream
#
Expand Down
88 changes: 88 additions & 0 deletions e2e/malicious-join/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
load("@com_github_ash2k_bazel_tools//multirun:def.bzl", "multirun")
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_push")
load("@rules_pkg//:pkg.bzl", "pkg_tar")
load("//bazel/sh:def.bzl", "sh_template")

go_library(
name = "malicious-join_lib",
srcs = ["malicious-join.go"],
importpath = "github.com/edgelesssys/constellation/v2/e2e/malicious-join",
visibility = ["//visibility:public"],
deps = [
"//internal/attestation/variant",
"//internal/cloud/cloudprovider",
"//internal/grpc/dialer",
"//internal/logger",
"//joinservice/joinproto",
"@org_uber_go_zap//zapcore",
],
)

go_binary(
name = "malicious-join_bin",
embed = [":malicious-join_lib"],
pure = "on",
race = "off",
visibility = ["//visibility:public"],
)

pkg_tar(
name = "layer",
srcs = [
":malicious-join_bin",
],
mode = "0755",
remap_paths = {"/malicious-join_bin": "/malicious-join_bin"},
)

oci_image(
name = "malicious-join_image",
base = "@distroless_static_linux_amd64",
entrypoint = ["/malicious-join_bin"],
tars = [
":layer",
],
visibility = ["//visibility:public"],
)

genrule(
name = "malicious-join-test_repotag",
srcs = [
"//bazel/settings:tag",
],
outs = ["repotag.txt"],
cmd = "echo -n 'ghcr.io/edgelesssys/malicious-join-test:' | cat - $(location //bazel/settings:tag) > $@",
visibility = ["//visibility:public"],
)

oci_push(
name = "malicious-join_push",
image = ":malicious-join_image",
repotags = ":repotag.txt",
)

sh_template(
name = "template_job",
data = [
"job.yaml",
":repotag.txt",
"@yq_toolchains//:resolved_toolchain",
],
substitutions = {
"@@REPO_TAG@@": "$(rootpath :repotag.txt)",
"@@TEMPLATE@@": "$(rootpath :job.yaml)",
"@@YQ_BIN@@": "$(rootpath @yq_toolchains//:resolved_toolchain)",
},
template = "job_template.sh.in",
visibility = ["//visibility:public"],
)

multirun(
name = "stamp_and_push",
commands = [
":template_job",
":malicious-join_push",
],
visibility = ["//visibility:public"],
)
12 changes: 12 additions & 0 deletions e2e/malicious-join/job.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: batch/v1
kind: Job
metadata:
name: malicious-join
spec:
template:
spec:
containers:
- name: malicious-join
image: ghcr.io/edgelesssys/malicious-join-test:latest@sha256:f36fe306d50a6731ecdae3920682606967eb339fdd1a1e978b0ce39c2ab744bd
msanft marked this conversation as resolved.
Show resolved Hide resolved
restartPolicy: Never
backoffLimit: 0 # Do not retry
26 changes: 26 additions & 0 deletions e2e/malicious-join/job_template.sh.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env bash

msanft marked this conversation as resolved.
Show resolved Hide resolved
lib=$(realpath @@BASE_LIB@@) || exit 1
stat "${lib}" >> /dev/null || exit 1
msanft marked this conversation as resolved.
Show resolved Hide resolved

# shellcheck source=../../bazel/sh/lib.bash
if ! source "${lib}"; then
echo "Error: could not find import"
exit 1
fi

yq=$(realpath @@YQ_BIN@@)
template=$(realpath @@TEMPLATE@@)
REPO_TAG=$(realpath @@REPO_TAG@@)
export REPO_TAG

cd "${BUILD_WORKING_DIRECTORY}"

if [[ $# -eq 0 ]]; then
workdir="."
else
workdir="$1"
fi

echo "Stamping job deployment with $REPO_TAG"
$yq eval '.spec.template.spec.containers[0].image |= "ghcr.io/edgelesssys/malicious-join-test:" + load_str(strenv(REPO_TAG))' "$template" > "$workdir/stamped_job.yaml"
Loading