Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue #1227 - Get licenses for NuGet packages #3329

Open
wants to merge 61 commits into
base: main
Choose a base branch
from

Conversation

HeyeOpenSource
Copy link

@HeyeOpenSource HeyeOpenSource commented Oct 14, 2024

Description

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist:

  • I have added unit tests that cover changed behavior
  • I have tested my code in common scenarios and confirmed there are no regressions
  • I have added comments to my code, particularly in hard-to-understand sections

@spiffcs spiffcs self-requested a review October 14, 2024 15:41
@spiffcs spiffcs self-assigned this Oct 14, 2024
HeyeOpenSource and others added 14 commits October 15, 2024 09:03
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Improves the go cataloger semver extraction logic to include getting the
release version of traefik.  This is based off of the regex pattern that
already existed in the traefik binary classifier.

Signed-off-by: Weston Steimel <commits@weston.slmail.me>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: wagoodman <590471+wagoodman@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…options dotnetConfig struct.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…ons Catalog struct.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…re#3327)

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.12 to 3.26.13.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@c36620d...f779452)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…3326)

Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.17.2 to 0.17.3.
- [Release notes](https://github.com/anchore/sbom-action/releases)
- [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
- [Commits](anchore/sbom-action@61119d4...f5e124a)

---
updated-dependencies:
- dependency-name: anchore/sbom-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…09e9e5 (anchore#3331)

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: willmurphyscode <12529630+willmurphyscode@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…870434 (anchore#3332)

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: willmurphyscode <12529630+willmurphyscode@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…rs() function in syft/pkg/cataloger/dotnet.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
@HeyeOpenSource
Copy link
Author

HeyeOpenSource commented Oct 16, 2024

Just for the record:
I oriented myself at the golang cataloger.

The configuration can also be influenced by the following four main environment variables:

  • SYFT_DOTNET_SEARCH_LOCAL_LICENSES
    • 'true' | 'false'
    • Search for NuGet packages in all known local cache directories.
  • SYFT_DOTNET_LOCAL_CACHE_PATHS
    • Expects a comma-separated lists of local cache directories for NuGet packages to use when searching for local NuGet packages / licenses.
    • Defaults to all known local NuGet repository cache directories as retrieved from the dotnet SDK-tool if undefined.
  • SYFT_DOTNET_SEARCH_REMOTE_LICENSES
    • 'true' | 'false'
    • Search for NuGet packages in all enabled known remote NuGet package repositories eg. https://api.nuget.org/v3-flatcontainer/ as retrieved from https://api.nuget.org/v3/index.json (unless the NuGet package providers setting is overridden)
  • SYFT_DOTNET_PACKAGE_PROVIDERS
    • Expects a comma-separated lists of the URLs of remote NuGet package repositories to use when searching for remote NuGet packages / licenses.
    • Defaults to all enabled known remote NuGet repositories as retrieved from the dotnet SDK-tool if undefined.

NuGet package provider credentials:

These are only ever used, if a NuGet package repository returns the status code 401 Unauthorized when trying to retrieve a remote NuGet package.

  • SYFT_DOTNET_PACKAGE_PROVIDER_CREDENTIALS_USERNAME
    • Username for a credential for remote NuGet package repositories to use when searching for remote NuGet packages / licenses in a package repository requiring authentication.
  • SYFT_DOTNET_PACKAGE_PROVIDER_CREDENTIALS_PASSWORD
    • Password for a credential for remote NuGet package repositories to use when searching for remote NuGet packages / licenses in a package repository requiring authentication.

N.B.:

  • Credentials are only accepted if both username and password are given.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…age provider URLs terminated by '/'.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…hen accessing remote NuGet package repositories.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
  - Fix remote NuGet license retrieval.
  - Allow for NuGet package retrieval from package repositories requiring authentication.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
@HeyeOpenSource
Copy link
Author

FYI:
The feature for defining credentials should mostly be applicable to commercial CI/CD environments,
where access restriction is required by appropriate data security requirements.

Gitea for example allows to create such code- and NuGet package repositories.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…loger configuration.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…rich``` functionality.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
@HeyeOpenSource
Copy link
Author

HeyeOpenSource commented Oct 24, 2024

After tackling all review conversations up to now, the Validations action is all green once more:
Validations #27

I have removed any artifacts, which are not strictly neccessary.
As you correctly stated it does not really make sense to have obsolete images and binaries clogging up the repo.

Copy link
Contributor

@kzantow kzantow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the latest updates; I found a few other things after a finer tooth comb. Other than what's mentioned here, it's looking really good -- very appreciated!

internal/task/package_tasks.go Outdated Show resolved Hide resolved
cmd/syft/internal/options/dotnet.go Outdated Show resolved Hide resolved
cmd/syft/internal/options/dotnet.go Show resolved Hide resolved
cmd/syft/internal/options/dotnet.go Outdated Show resolved Hide resolved
cmd/syft/internal/options/dotnet.go Outdated Show resolved Hide resolved
syft/pkg/cataloger/dotnet/licenses.go Show resolved Hide resolved
syft/pkg/cataloger/dotnet/licenses.go Outdated Show resolved Hide resolved
syft/pkg/cataloger/dotnet/parse_dotnet_deps.go Outdated Show resolved Hide resolved
HeyeOpenSource and others added 7 commits October 30, 2024 17:07
…Oct 29th 2024.

Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
…3393)

Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.17.5 to 0.17.6.
- [Release notes](https://github.com/anchore/sbom-action/releases)
- [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
- [Commits](anchore/sbom-action@1ca97d9...251a468)

---
updated-dependencies:
- dependency-name: anchore/sbom-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Signed-off-by: HeyeOpenSource <opensource@heye-international.com>
Copy link
Author

@HeyeOpenSource HeyeOpenSource left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a sense, the localNuGetCacheResolvers are source content:
These are the folders ('sources') from where the external NuGet dependencies are linked...

Hence, I would currently refrain from changing from the file.Resolver implementation to an fs package implementation.

@kzantow : What do you say?

cmd/syft/internal/options/dotnet.go Outdated Show resolved Hide resolved
cmd/syft/internal/options/catalog.go Outdated Show resolved Hide resolved
syft/pkg/cataloger/dotnet/config.go Show resolved Hide resolved
syft/pkg/cataloger/dotnet/config.go Outdated Show resolved Hide resolved
syft/pkg/cataloger/dotnet/licenses.go Show resolved Hide resolved
syft/pkg/cataloger/dotnet/licenses.go Outdated Show resolved Hide resolved
syft/pkg/cataloger/dotnet/config.go Outdated Show resolved Hide resolved

type nugetLicenseResolver struct {
opts CatalogerConfig
localNuGetCacheResolvers []file.Resolver
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification about the decisions taken.
At the moment I am not quite sure, how I'd go about performing the switch towards the fs package...

Then again:
In a sense, the localNuGetCacheResolvers are source content:
These are the folders ('sources') from where the external NuGet dependencies are linked... 🤷‍♂️

@kzantow : What do you say?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Review
Development

Successfully merging this pull request may close these issues.

Get licenses for NuGet packages
4 participants