Secret scanning in images isn't working as expected.

[bnevis-i]

Description

I am trying to get secret scanning on docker images working. I have gone so far as to clone the github repo and do some step-by-step debugging.

What I did was to create a docker image with an RSA private key in it.

I the did a "docker save" to export a tarball, and then scan it with trivy image --input image.tar

What I noticed was that in the following code in pkg/fanal/artifact/image/image.go

	if err = a.inspect(ctx, missingImageKey, missingLayers, baseDiffIDs, layerKeyMap, configFile); err != nil {
		return types.ArtifactReference{}, xerrors.Errorf("analyze error: %w", err)
	}

That missingLayers is empty and it doesn't run the parallel secret scanning task. If I put layerKeyMap instead of missingLayers then I get debug messages about Missing diff ID in cache.

I don't quite understand yet the internal logic of what the code is doing, but it seems to me that it isn't scanning the image layers at all and that secret scanning of images is completely broken. (fs scanning seems to work)

I am wondering if there is a test case that the developers run to feed a known secret-infested image into trivy and get a known output, so that I can compare the known result with my own.

Desired Behavior

I expect secret scanning to output known secrets that I have seeded into the docker image.

Actual Behavior

No secrets are reported.

Reproduction Steps

1. Use openssl to generate an RSA keypair
2. Create Dockerfile that copies the private key into the image
3. Build the image
4. Export the image to a tar file using docker save
5. Run `trivy image --debug --scanners secret --input image.tar`
6. No secrets detected
...

Target

Container Image

Scanner

Secret

Output Format

None

Mode

Standalone

Debug Output

$ `trivy image --debug --scanners secret --input image.tar`
2023-06-14T22:09:08.123-0700	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-06-14T22:09:08.126-0700	DEBUG	cache dir:  /home/redacted/.cache/trivy
2023-06-14T22:09:08.126-0700	INFO	Secret scanning is enabled
2023-06-14T22:09:08.126-0700	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-06-14T22:09:08.126-0700	INFO	Please see also https://aquasecurity.github.io/trivy/v0.42/docs/secret/scanning/#recommendation for faster secret detection
2023-06-14T22:09:15.651-0700	INFO	Loading trivy-secret.yaml for secret scanning...
2023-06-14T22:09:15.654-0700	DEBUG	Image ID: sha256:ca652e80043891f3d6e25df37d651d9c85f7bd8f4891506e24dc1424ea7f59af
2023-06-14T22:09:15.654-0700	DEBUG	Diff IDs: [sha256:37afa14f948c17b5ba163973372966786c28b38558ad68d9d8c976a980ed64b2 sha256:d8600cba362549b5bc1f25de4168ad48305ba314e800cc14eb5cac9add0c7249 sha256:969ec887a72631770f331d5bb5c9bd5adcf6785cc99bc92271023b3dc818f006 sha256:d6690fb3afcdeab7764b1a03bd018d8a00b9a2b27a2ed9d8caf164b7b5391d0b sha256:c2760438b87d6e45f4e41855ac64edc452492614da24ed085c80293d38bd0dd7 sha256:828dc9f1f1aff29661e6df6a480041f845954742e09b220ccac24743ac94ccbb sha256:2e687e597b6704563a33e84be27a0bb92ecf5e5ca04669634ec8cc1dc7b284c9 sha256:0b3caf5951325399fdf5fff72d381d440ba325e7ed66eb203cae38436a5cad89 sha256:bdfbfde68ac710b6fc984a2461062874dd7fe7aee6b7cb6fd1284838e40045f3 sha256:03c5108fad6044627017e75e2da68a80beccb413612938dc8e2e5fb443c01d28 sha256:2a41f18a33dde5039aaadd56eb214e4980da3ea0e39455126f6fdd368e2f74c3 sha256:853f0eeadf7235d1f7f255f96a09dffd15062c57fc8fed63695b19033ef03477]
2023-06-14T22:09:15.654-0700	DEBUG	Base Layers: [sha256:37afa14f948c17b5ba163973372966786c28b38558ad68d9d8c976a980ed64b2]

After trivy image --reset the output is slightly different but still does not find the private key:
(Also deleted a trivy-secret.yaml that had an aggressive wildcard rule that suddenly started producing false positives after the reset).

2023-06-14T22:14:04.415-0700	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-06-14T22:14:04.424-0700	DEBUG	cache dir:  /home/bnevis/.cache/trivy
2023-06-14T22:14:04.424-0700	INFO	Secret scanning is enabled
2023-06-14T22:14:04.424-0700	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-06-14T22:14:04.424-0700	INFO	Please see also https://aquasecurity.github.io/trivy/v0.42/docs/secret/scanning/#recommendation for faster secret detection
2023-06-14T22:14:11.687-0700	DEBUG	No secret config detected: trivy-secret.yaml
2023-06-14T22:14:11.688-0700	DEBUG	Image ID: sha256:ca652e80043891f3d6e25df37d651d9c85f7bd8f4891506e24dc1424ea7f59af
2023-06-14T22:14:11.688-0700	DEBUG	Diff IDs: [sha256:37afa14f948c17b5ba163973372966786c28b38558ad68d9d8c976a980ed64b2 sha256:d8600cba362549b5bc1f25de4168ad48305ba314e800cc14eb5cac9add0c7249 sha256:969ec887a72631770f331d5bb5c9bd5adcf6785cc99bc92271023b3dc818f006 sha256:d6690fb3afcdeab7764b1a03bd018d8a00b9a2b27a2ed9d8caf164b7b5391d0b sha256:c2760438b87d6e45f4e41855ac64edc452492614da24ed085c80293d38bd0dd7 sha256:828dc9f1f1aff29661e6df6a480041f845954742e09b220ccac24743ac94ccbb sha256:2e687e597b6704563a33e84be27a0bb92ecf5e5ca04669634ec8cc1dc7b284c9 sha256:0b3caf5951325399fdf5fff72d381d440ba325e7ed66eb203cae38436a5cad89 sha256:bdfbfde68ac710b6fc984a2461062874dd7fe7aee6b7cb6fd1284838e40045f3 sha256:03c5108fad6044627017e75e2da68a80beccb413612938dc8e2e5fb443c01d28 sha256:2a41f18a33dde5039aaadd56eb214e4980da3ea0e39455126f6fdd368e2f74c3 sha256:853f0eeadf7235d1f7f255f96a09dffd15062c57fc8fed63695b19033ef03477]
2023-06-14T22:14:11.688-0700	DEBUG	Base Layers: [sha256:37afa14f948c17b5ba163973372966786c28b38558ad68d9d8c976a980ed64b2]
2023-06-14T22:14:11.689-0700	DEBUG	Missing image ID in cache: sha256:ca652e80043891f3d6e25df37d651d9c85f7bd8f4891506e24dc1424ea7f59af
2023-06-14T22:14:11.689-0700	DEBUG	Missing diff ID in cache: sha256:37afa14f948c17b5ba163973372966786c28b38558ad68d9d8c976a980ed64b2
2023-06-14T22:14:11.689-0700	DEBUG	Missing diff ID in cache: sha256:c2760438b87d6e45f4e41855ac64edc452492614da24ed085c80293d38bd0dd7
2023-06-14T22:14:11.689-0700	DEBUG	Missing diff ID in cache: sha256:d8600cba362549b5bc1f25de4168ad48305ba314e800cc14eb5cac9add0c7249
2023-06-14T22:14:11.689-0700	DEBUG	Missing diff ID in cache: sha256:969ec887a72631770f331d5bb5c9bd5adcf6785cc99bc92271023b3dc818f006
2023-06-14T22:14:11.689-0700	DEBUG	Missing diff ID in cache: sha256:d6690fb3afcdeab7764b1a03bd018d8a00b9a2b27a2ed9d8caf164b7b5391d0b
2023-06-14T22:14:11.760-0700	DEBUG	Skipping directory: dev
2023-06-14T22:14:11.766-0700	DEBUG	Skipping directory: proc
2023-06-14T22:14:11.766-0700	DEBUG	Skipping directory: sys
2023-06-14T22:14:11.784-0700	DEBUG	Missing diff ID in cache: sha256:828dc9f1f1aff29661e6df6a480041f845954742e09b220ccac24743ac94ccbb
2023-06-14T22:14:12.754-0700	DEBUG	Missing diff ID in cache: sha256:2e687e597b6704563a33e84be27a0bb92ecf5e5ca04669634ec8cc1dc7b284c9
2023-06-14T22:14:12.799-0700	DEBUG	Missing diff ID in cache: sha256:0b3caf5951325399fdf5fff72d381d440ba325e7ed66eb203cae38436a5cad89
2023-06-14T22:14:12.822-0700	DEBUG	Missing diff ID in cache: sha256:bdfbfde68ac710b6fc984a2461062874dd7fe7aee6b7cb6fd1284838e40045f3
2023-06-14T22:14:12.873-0700	DEBUG	Missing diff ID in cache: sha256:03c5108fad6044627017e75e2da68a80beccb413612938dc8e2e5fb443c01d28
2023-06-14T22:14:12.902-0700	DEBUG	Missing diff ID in cache: sha256:2a41f18a33dde5039aaadd56eb214e4980da3ea0e39455126f6fdd368e2f74c3
2023-06-14T22:14:12.980-0700	DEBUG	Missing diff ID in cache: sha256:853f0eeadf7235d1f7f255f96a09dffd15062c57fc8fed63695b19033ef03477
2023-06-14T22:14:18.249-0700	DEBUG	Unable to parse "var/lib/dpkg/available" file: file open error: open var/lib/dpkg/available: file does not exist
2023-06-14T22:14:18.691-0700	DEBUG	No secrets found in container image config

Operating System

Ubuntu 22.04 x86_64

Version

Version: 0.42.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-06-15 00:13:18.735419495 +0000 UTC
  NextUpdate: 2023-06-15 06:13:18.735419295 +0000 UTC
  DownloadedAt: 2023-06-15 02:33:35.862396557 +0000 UTC

Checklist

• Run trivy --reset
• Read the troubleshooting

[knqyf263]

@afdesk Can you please take a look?

[afdesk]

sure, I'll take a look

[afdesk]

@bnevis-i thanks for your report!
I'm trying to reproduce the issue and cannot do it.

my steps are next:

1. Generate an RSA keypair:

$ ssh-keygen -f key1 -N ""

2. Create a simple Dockerfile

FROM alpine:3.18.0
COPY key1 /pr/key1

3. Build an image

$ docker build -t demo:4638 .
Sending build context to Docker daemon  7.168kB
Step 1/2 : FROM alpine:3.18.0
 ---> 5e2b554c1c45
Step 2/2 : COPY key1 /pr/key1
 ---> be502e4497a7
Successfully built be502e4497a7
Successfully tagged demo:4638

4. Scan an image

$ trivy i --clear-cache && trivy -d i --scanners secret demo:4638
...
/pr/key1 (secrets)

Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

HIGH: AsymmetricPrivateKey (private-key)
══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Asymmetric Private Key
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 /pr/key1:1 (added by 'COPY file:75061f588f08ddcf167e0e815ed4a7')
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 [ -----BEGIN OPENSSH PRIVATE KEY-----*************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************-----END OPENSSH PRIVATE KEY-----
   2   
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

5. Store a tar archive

$ docker save demo:4638 > demo.tar

6. A new scan of the tar file:

$ trivy i --clear-cache && trivy -d i --scanners secret --input demo.tar
2023-06-15T22:56:12.346+0600	INFO	Removing artifact caches...
2023-06-15T22:56:13.041+0600	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
...

/pr/key1 (secrets)

Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

HIGH: AsymmetricPrivateKey (private-key)
══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Asymmetric Private Key
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 /pr/key1:1 (added by 'COPY file:75061f588f08ddcf167e0e815ed4a7')
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 [ -----BEGIN OPENSSH PRIVATE KEY-----*************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************-----END OPENSSH PRIVATE KEY-----
   2   
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[afdesk]

@bnevis-i I have several ideas:

1. There is using trivy-secret.yaml in the first debug output. Could you share this file?

2023-06-14T22:09:15.651-0700	INFO	Loading trivy-secret.yaml for secret scanning...

2. Where did you store the private key? maybe Trivy skips this folder...

[afdesk]

or maybe I miss something )

[bnevis-i]

I feel so bad.

With your counter example and a good night's sleep, I can see what I did wrong now.

Would you believe that I copied the key to /example

I'm so sorry for wasting your time.

And the trivy-secret.yaml I deleted without saving a copy when I went through the discussion posting checklist and noticed I had left it around. It wasn't relevant to my goof.