Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(k8s): improve artifact selections for specific namespaces #8248

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
36 changes: 35 additions & 1 deletion docs/docs/target/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,41 @@ You can also specify a `kubeconfig` using the `--kubeconfig` flag:
trivy k8s --kubeconfig ~/.kube/config2
```

By default, all cluster resource images will be downloaded and scanned.
## Required roles
To successfully scan a Kubernetes cluster, `trivy kubernetes` subcommand must be executed under a role or a cluster role that has some specific permissions.

The role must have `list` verb for all resources (`"*"`) inside next API groups: core (`""`), `"apps"`, `"batch"`,`"networking.k8s.io"`, `"rbac.authorization.k8s.io"`:
```yaml
- apiGroups: [""]
resources: ["*"]
verbs: ["list"]
- apiGroups: ["apps", "batch", "networking.k8s.io", "rbac.authorization.k8s.io"]
resources: ["*"]
verbs: ["list"]
```
If `node collector` is enabled (by default), Trivy needs a cluster role with some additional permissions to run and track the jobs:
```yaml
- apiGroups: [""]
resources: ["nodes/proxy", "pods/log"]
verbs: ["get"]
- apiGroups: [""]
resources: ["events"]
verbs: ["watch"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["list", "get"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create","delete", "watch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["create"]
```

### Skip-images

By default, all cluster resource images will be downloaded and scanned.

You can control whether Trivy will scan and download the cluster resource images. To disable this feature, add the --skip-images flag.

- `--skip-images` flag will prevent the downloading and scanning of images (including vulnerabilities and secrets) in the cluster resources.
Expand Down Expand Up @@ -91,6 +122,9 @@ You can control which namespaces will be discovered using the `--include-namespa

By default, all namespaces will be included in cluster scanning.

!!! note "using `--exclude-namespaces`"
Trivy requires a complete list of namespaces to exclude specific ones. Therefore, `--exclude-namespaces` option is only available for cluster roles now.

Example:

```sh
Expand Down
18 changes: 9 additions & 9 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,13 @@ require (
github.com/aquasecurity/trivy-checks v1.4.0
github.com/aquasecurity/trivy-db v0.0.0-20241209111357-8c398f13db0e
github.com/aquasecurity/trivy-java-db v0.0.0-20240109071736-184bd7481d48
github.com/aquasecurity/trivy-kubernetes v0.6.7-0.20241101182546-89bffc3932bc
github.com/aquasecurity/trivy-kubernetes v0.7.0
github.com/aws/aws-sdk-go-v2 v1.33.0
github.com/aws/aws-sdk-go-v2/config v1.29.1
github.com/aws/aws-sdk-go-v2/credentials v1.17.54
github.com/aws/aws-sdk-go-v2/service/ec2 v1.200.0
github.com/aws/aws-sdk-go-v2/service/ecr v1.38.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.73.2
github.com/aws/aws-sdk-go-v2/service/sts v1.33.9 // indirect
github.com/aws/smithy-go v1.22.1
github.com/bitnami/go-version v0.0.0-20231130084017-bb00604d650c
github.com/bmatcuk/doublestar/v4 v4.8.0
Expand Down Expand Up @@ -130,7 +129,7 @@ require (
google.golang.org/protobuf v1.36.3
gopkg.in/yaml.v3 v3.0.1
helm.sh/helm/v3 v3.17.0
k8s.io/api v0.32.0
k8s.io/api v0.32.1
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738
modernc.org/sqlite v1.34.5
sigs.k8s.io/yaml v1.4.0
Expand Down Expand Up @@ -178,7 +177,7 @@ require (
github.com/anchore/go-struct-converter v0.0.0-20221118182256-c68fdcfa2092 // indirect
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go v1.55.5 // indirect
github.com/aws/aws-sdk-go v1.55.6 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.24 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.28 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.28 // indirect
Expand All @@ -188,6 +187,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.9 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.11 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.10 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.9 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
Expand Down Expand Up @@ -415,14 +415,14 @@ require (
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
k8s.io/apiextensions-apiserver v0.32.0 // indirect
k8s.io/apimachinery v0.32.0 // indirect
k8s.io/apimachinery v0.32.1 // indirect
k8s.io/apiserver v0.32.0 // indirect
k8s.io/cli-runtime v0.32.0 // indirect
k8s.io/client-go v0.32.0 // indirect
k8s.io/component-base v0.32.0 // indirect
k8s.io/cli-runtime v0.32.1 // indirect
k8s.io/client-go v0.32.1 // indirect
k8s.io/component-base v0.32.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f // indirect
k8s.io/kubectl v0.32.0 // indirect
k8s.io/kubectl v0.32.1 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
Expand Down
32 changes: 16 additions & 16 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -810,17 +810,17 @@ github.com/aquasecurity/trivy-db v0.0.0-20241209111357-8c398f13db0e h1:O5j5SeCNB
github.com/aquasecurity/trivy-db v0.0.0-20241209111357-8c398f13db0e/go.mod h1:gS8VhlNxhraiq60BBnJw9kGtjeMspQ9E8pX24jCL4jg=
github.com/aquasecurity/trivy-java-db v0.0.0-20240109071736-184bd7481d48 h1:JVgBIuIYbwG+ekC5lUHUpGJboPYiCcxiz06RCtz8neI=
github.com/aquasecurity/trivy-java-db v0.0.0-20240109071736-184bd7481d48/go.mod h1:Ldya37FLi0e/5Cjq2T5Bty7cFkzUDwTcPeQua+2M8i8=
github.com/aquasecurity/trivy-kubernetes v0.6.7-0.20241101182546-89bffc3932bc h1:/mFBYIK9RY+L8s1CIbQbJ5B3v0YmoDSu5eAzavvMa+Y=
github.com/aquasecurity/trivy-kubernetes v0.6.7-0.20241101182546-89bffc3932bc/go.mod h1:ctlibFXOQyjWybeVVQI6NLG6GJoPWZJ4cIirQ/wPCQs=
github.com/aquasecurity/trivy-kubernetes v0.7.0 h1:0pRJFSslUYd9xzQIEw1c0mS7k1Vv489nH/LsxeU6yME=
github.com/aquasecurity/trivy-kubernetes v0.7.0/go.mod h1:O6JZMicTmZrsjEpGzsnBMhPTHAfpnTMqXTAMidG6M+M=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go v1.44.122/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
github.com/aws/aws-sdk-go v1.55.5 h1:KKUZBfBoyqy5d3swXyiC7Q76ic40rYcbqH7qjh59kzU=
github.com/aws/aws-sdk-go v1.55.5/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
github.com/aws/aws-sdk-go v1.55.6 h1:cSg4pvZ3m8dgYcgqB97MrcdjUmZ1BeMYKUxMMB89IPk=
github.com/aws/aws-sdk-go v1.55.6/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
github.com/aws/aws-sdk-go-v2 v1.33.0 h1:Evgm4DI9imD81V0WwD+TN4DCwjUMdc94TrduMLbgZJs=
github.com/aws/aws-sdk-go-v2 v1.33.0/go.mod h1:P5WJBrYqqbWVaOxgH0X/FYYD47/nooaPOZPlQdmiN2U=
github.com/aws/aws-sdk-go-v2/config v1.29.1 h1:JZhGawAyZ/EuJeBtbQYnaoftczcb2drR2Iq36Wgz4sQ=
Expand Down Expand Up @@ -2780,26 +2780,26 @@ honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.1.3/go.mod h1:NgwopIslSNH47DimFoV78dnkksY2EFtX0ajyb3K/las=
k8s.io/api v0.32.0 h1:OL9JpbvAU5ny9ga2fb24X8H6xQlVp+aJMFlgtQjR9CE=
k8s.io/api v0.32.0/go.mod h1:4LEwHZEf6Q/cG96F3dqR965sYOfmPM7rq81BLgsE0p0=
k8s.io/api v0.32.1 h1:f562zw9cy+GvXzXf0CKlVQ7yHJVYzLfL6JAS4kOAaOc=
k8s.io/api v0.32.1/go.mod h1:/Yi/BqkuueW1BgpoePYBRdDYfjPF5sgTr5+YqDZra5k=
k8s.io/apiextensions-apiserver v0.32.0 h1:S0Xlqt51qzzqjKPxfgX1xh4HBZE+p8KKBq+k2SWNOE0=
k8s.io/apiextensions-apiserver v0.32.0/go.mod h1:86hblMvN5yxMvZrZFX2OhIHAuFIMJIZ19bTvzkP+Fmw=
k8s.io/apimachinery v0.32.0 h1:cFSE7N3rmEEtv4ei5X6DaJPHHX0C+upp+v5lVPiEwpg=
k8s.io/apimachinery v0.32.0/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
k8s.io/apimachinery v0.32.1 h1:683ENpaCBjma4CYqsmZyhEzrGz6cjn1MY/X2jB2hkZs=
k8s.io/apimachinery v0.32.1/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
k8s.io/apiserver v0.32.0 h1:VJ89ZvQZ8p1sLeiWdRJpRD6oLozNZD2+qVSLi+ft5Qs=
k8s.io/apiserver v0.32.0/go.mod h1:HFh+dM1/BE/Hm4bS4nTXHVfN6Z6tFIZPi649n83b4Ag=
k8s.io/cli-runtime v0.32.0 h1:dP+OZqs7zHPpGQMCGAhectbHU2SNCuZtIimRKTv2T1c=
k8s.io/cli-runtime v0.32.0/go.mod h1:Mai8ht2+esoDRK5hr861KRy6z0zHsSTYttNVJXgP3YQ=
k8s.io/client-go v0.32.0 h1:DimtMcnN/JIKZcrSrstiwvvZvLjG0aSxy8PxN8IChp8=
k8s.io/client-go v0.32.0/go.mod h1:boDWvdM1Drk4NJj/VddSLnx59X3OPgwrOo0vGbtq9+8=
k8s.io/component-base v0.32.0 h1:d6cWHZkCiiep41ObYQS6IcgzOUQUNpywm39KVYaUqzU=
k8s.io/component-base v0.32.0/go.mod h1:JLG2W5TUxUu5uDyKiH2R/7NnxJo1HlPoRIIbVLkK5eM=
k8s.io/cli-runtime v0.32.1 h1:19nwZPlYGJPUDbhAxDIS2/oydCikvKMHsxroKNGA2mM=
k8s.io/cli-runtime v0.32.1/go.mod h1:NJPbeadVFnV2E7B7vF+FvU09mpwYlZCu8PqjzfuOnkY=
k8s.io/client-go v0.32.1 h1:otM0AxdhdBIaQh7l1Q0jQpmo7WOFIk5FFa4bg6YMdUU=
k8s.io/client-go v0.32.1/go.mod h1:aTTKZY7MdxUaJ/KiUs8D+GssR9zJZi77ZqtzcGXIiDg=
k8s.io/component-base v0.32.1 h1:/5IfJ0dHIKBWysGV0yKTFfacZ5yNV1sulPh3ilJjRZk=
k8s.io/component-base v0.32.1/go.mod h1:j1iMMHi/sqAHeG5z+O9BFNCF698a1u0186zkjMZQ28w=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f h1:GA7//TjRY9yWGy1poLzYYJJ4JRdzg3+O6e8I+e+8T5Y=
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f/go.mod h1:R/HEjbvWI0qdfb8viZUeVZm0X6IZnxAydC7YU42CMw4=
k8s.io/kubectl v0.32.0 h1:rpxl+ng9qeG79YA4Em9tLSfX0G8W0vfaiPVrc/WR7Xw=
k8s.io/kubectl v0.32.0/go.mod h1:qIjSX+QgPQUgdy8ps6eKsYNF+YmFOAO3WygfucIqFiE=
k8s.io/kubectl v0.32.1 h1:/btLtXLQUU1rWx8AEvX9jrb9LaI6yeezt3sFALhB8M8=
k8s.io/kubectl v0.32.1/go.mod h1:sezNuyWi1STk4ZNPVRIFfgjqMI6XMf+oCVLjZen/pFQ=
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 h1:M3sRQVHv7vB20Xc2ybTt7ODCeFj6JSWYFzOFnYeS6Ro=
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
lukechampine.com/uint128 v1.1.1/go.mod h1:c4eWIwlEGaxC/+H1VguhU4PHXNWDCDMUlWdIWl2j1gk=
Expand Down
51 changes: 51 additions & 0 deletions integration/k8s_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,4 +106,55 @@ func TestK8s(t *testing.T) {
return len(*r.Dependencies) > 0
}))
})

t.Run("limited user test", func(t *testing.T) {
// Set up the output file
outputFile := filepath.Join(t.TempDir(), "output.json")

osArgs := []string{
"--cache-dir",
cacheDir,
"k8s",
"limitedcontext",
"--kubeconfig", "limitedconfig",
"--report",
"summary",
"-q",
"--timeout",
"5m0s",
"--include-namespaces", "limitedns",
"--format",
"json",
"--output",
outputFile,
}

// Run Trivy
err := execute(osArgs)
require.NoError(t, err)

var got report.ConsolidatedReport
f, err := os.Open(outputFile)
require.NoError(t, err)
defer f.Close()

err = json.NewDecoder(f).Decode(&got)
require.NoError(t, err)

// Flatten findings
results := lo.FlatMap(got.Findings, func(resource report.Resource, _ int) []types.Result {
return resource.Results
})

// Has vulnerabilities
assert.True(t, lo.SomeBy(results, func(r types.Result) bool {
return len(r.Vulnerabilities) > 0
}))

// Has misconfigurations
assert.True(t, lo.SomeBy(results, func(r types.Result) bool {
return len(r.Misconfigurations) > 0
}))

})
}
18 changes: 18 additions & 0 deletions integration/testdata/fixtures/k8s/kube-config-template
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: {{CA}}
server: {{URL}}
name: kind-kind-test
contexts:
- context:
cluster: kind-kind-test
namespace: limitedns
user: limiteduser
name: limitedcontext
kind: Config
preferences: {}
users:
- name: limiteduser
user:
token: {{TOKEN}}
15 changes: 15 additions & 0 deletions integration/testdata/fixtures/k8s/limited-binding.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: limited-binding
namespace: limitedns

subjects:
- kind: ServiceAccount
name: limiteduser
namespace: default

roleRef:
kind: Role
name: limited-role
apiGroup: rbac.authorization.k8s.io
11 changes: 11 additions & 0 deletions integration/testdata/fixtures/k8s/limited-pod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: v1
kind: Pod
metadata:
name: limited-pod
namespace: limitedns
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
12 changes: 12 additions & 0 deletions integration/testdata/fixtures/k8s/limited-role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: limited-role
namespace: limitedns
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["list"]
- apiGroups: ["apps", "batch", "networking.k8s.io","rbac.authorization.k8s.io"]
resources: ["*"]
verbs: ["list"]
61 changes: 61 additions & 0 deletions magefiles/magefile.go
Original file line number Diff line number Diff line change
Expand Up @@ -307,13 +307,74 @@ func (t Test) K8s() error {
defer func() {
_ = sh.RunWithV(ENV, "kind", "delete", "cluster", "--name", "kind-test")
}()
// wait for the kind cluster is running correctly
err = sh.RunWithV(ENV, "kubectl", "wait", "--for=condition=Ready", "nodes", "--all", "--timeout=300s")
if err != nil {
return fmt.Errorf("can't wait for the kind cluster: %w", err)
}

err = sh.RunWithV(ENV, "kubectl", "apply", "-f", "./integration/testdata/fixtures/k8s/test_nginx.yaml")
if err != nil {
return fmt.Errorf("can't create a test deployment: %w", err)
}

// create an environment for limited user test
err = initk8sLimitedUserEnv()
if err != nil {
return fmt.Errorf("can't create environment for limited user: %w", err)
}

// print all resources for info
err = sh.RunWithV(ENV, "kubectl", "get", "all", "-A")
if err != nil {
return err
}

return sh.RunWithV(ENV, "go", "test", "-v", "-tags=k8s_integration", "./integration/...")
}

func initk8sLimitedUserEnv() error {
commands := [][]string{
{"kubectl", "create", "namespace", "limitedns"},
{"kubectl", "create", "-f", "./integration/testdata/fixtures/k8s/limited-pod.yaml"},
{"kubectl", "create", "serviceaccount", "limiteduser"},
{"kubectl", "create", "-f", "./integration/testdata/fixtures/k8s/limited-role.yaml"},
{"kubectl", "create", "-f", "./integration/testdata/fixtures/k8s/limited-binding.yaml"},
{"cp", "./integration/testdata/fixtures/k8s/kube-config-template", "./integration/limitedconfig"},
}

for _, cmd := range commands {
if err := sh.RunV(cmd[0], cmd[1:]...); err != nil {
return err
}
}
envs := make(map[string]string)
var err error
envs["CA"], err = sh.Output("kubectl", "config", "view", "-o", "jsonpath=\"{.clusters[?(@.name == 'kind-kind-test')].cluster.certificate-authority-data}\"", "--flatten")
if err != nil {
return err
}
envs["URL"], err = sh.Output("kubectl", "config", "view", "-o", "jsonpath=\"{.clusters[?(@.name == 'kind-kind-test')].cluster.server}\"")
if err != nil {
return err
}
envs["TOKEN"], err = sh.Output("kubectl", "create", "token", "limiteduser", "--duration=8760h")
if err != nil {
return err
}
commandsWith := [][]string{
{"sed", "-i", "-e", "s|{{CA}}|$CA|g", "./integration/limitedconfig"},
{"sed", "-i", "-e", "s|{{URL}}|$URL|g", "./integration/limitedconfig"},
{"sed", "-i", "-e", "s|{{TOKEN}}|$TOKEN|g", "./integration/limitedconfig"},
}
for _, cmd := range commandsWith {
if err := sh.RunWithV(envs, cmd[0], cmd[1:]...); err != nil {
return err
}
}
return nil
}

// Module runs Wasm integration tests
func (t Test) Module() error {
mg.Deps(t.FixtureContainerImages, t.GenerateExampleModules)
Expand Down
Loading