SBOM container tools comparison

[jakub-bochenski]

I put together a comparison of container scanning tools: https://github.com/jakub-bochenski/container-sbom-shootout

I'm looking for feedback — if you have ideas on improving the comparison or know how any of the tools could be configured for better results, I’d love to hear your thoughts!

The comparison uses some popular public images, using specially crafted images with controlled contents would be better, but I didn’t find time for it.

[prabhu]

I'm generally not a fan of such simple comparisons. The tools work differently with a range of analysis techniques and require careful implementation and rollout. Users should not expect out-of-the-box magic for something as critical as SBOMs.

Generates many file components, which are not useful for license compliance or vulnerability detection.

You have no idea how many licence violations and potential security vulnerabilities have been caught in the field using these components. You can thank us later!

identifies the fewest components for Python and Go

For both Python and Go, cdxgen users scan the source to generate an accurate SBOM. We need a buildable environment to generate build SBOMs.

Points 3 and 4 are devops questions. cdxgen is indeed available as single executable application binaries requiring no installations.

Least user-friendly tool

Thank you for this feedback. The command is cdxgen -t docker <image name>. When scanning source code, all arguments including -t and the path could be ignored, leaving with just cdxgen. Not sure how simpler this can become.

[jakub-bochenski]

You have no idea how many licence violations and potential security vulnerabilities have been caught in the field using these components. You can thank us later!

I took a closer look at some of the reported files, e.g.

   {
      "name": "libutil.so.1",
      "type": "file",
      "purl": "pkg:generic/libutil.so.1",
      "bom-ref": "pkg:generic/libutil.so.1",
      "hashes": [
        {
          "alg": "MD5",
          "content": "5436f45543bc6335b6d4c730004bb73e"
        },
        {
          "alg": "SHA-1",
          "content": "d9fcb3659504f1228d03cabfd20f9bf1194ba9c8"
        }
      ],
      "properties": [
        {
          "name": "SrcFile",
          "value": "/usr/lib/x86_64-linux-gnu/libutil.so.1"
        },
        {
          "name": "internal:is_shared_library",
          "value": "true"
        }
      ],
      "evidence": {
        "identity": [
          {
            "field": "purl",
            "confidence": 0,
            "methods": [
              {
                "technique": "filename",
                "confidence": 0,
                "value": "/usr/lib/x86_64-linux-gnu/libutil.so.1"
              }
            ],
            "concludedValue": "/usr/lib/x86_64-linux-gnu/libutil.so.1"
          }
        ]
      }
    },

Neither OSS Index nor Gihub Advisories return any entries for it.
In fact the PURL pkg:generic/libutil.so.1 seems to be wrong, this is part of libc6 package so the PURL should be something like pkg:deb/debian/libc6@2.36-9+deb12u6?arch=amd64

I got similar results looking into another reported file component libanl.so.1, which is in fact part of JDK.

Cdxgen reported 0 licenses for those components, can you explain how they can be useful for detecting license violations?

I've never seen Dependency-Track do anything useful with components like this:

For both Python and Go, cdxgen users scan the source to generate an accurate SBOM. We need a buildable environment to generate build SBOMs.

The comparison was about scanning container images.

Not sure how simpler this can become.

It would be simpler if one could use the docker image to scan other docker images as per #2209

[jakub-bochenski]

Interesting difference is that e.g. Syft is able to detect more components, because it also inspects nested components.
It finds gson bundled with /opt/owasp/dependency-track/dependency-track-apiserver.jar:WEB-INF/lib/nimbus-jose-jwt-10.0.2.jar (via maven bundle plugin)

[prabhu]

tldr; join the aboutcode community with this invite and join one of the community meetings and network with people to learn these things.

cdxgen is a small tool in a pipeline of tools that are needed for software transparency. Each tool does some work in a pipeline to continuously enhance and refine the BOM data. In this pkg:generic example, cdxgen sees a file, can calculate its hash (hashes are non-reproducible), but lacks a lookup database. There is some logic in cdxgen to utilise the OS package manager to identify which package provides a given file. This is used in case of an OBOM. In case of a container image, however, the host could be different and we cannot execute these commands without running the image in an interactive manner (Don't ask how to run Docker from within a cdxgen Docker image!).

There are two PURL lookup databases available right now and both require some effort, solid contributions, and sponsorships for use in enterprise environments.

1. purldb - from the creators of PURL (Recommended)
2. blint-db - my personal project. Not ready for external use.

Re: vulnerability databases - none of the ones you mentioned are perfect and complete. We have two databases (again requires significant support)

1. vulnerablecode - from the aboutcode people (Recommended)
2. vulnerability-db - my personal project.

Re: the specific Syft feature to look inside a jar, it sounds cool. There is no realistic way to implement that feature in cdxgen and it doesn't sound like a critical feature to me. Scancode could be used to implement this in a pipeline and might already support it, so feel free to use Syft or Scancode instead.

[jakub-bochenski]

thanks, I've added part of your explanation to the readme: jakub-bochenski/container-sbom-shootout@97d14f5

[prabhu]

You should also post my Slack comment that questions the validity and usefulness of these benchmarks. It is really easy for anyone to come up with a comparison report with a small, limited number of test cases and criticise tools. All these benchmarks really miss the point and don't even attempt to come up with anything actionable. Tool authors and tools can only do so much.