Resolving dependency / artifact in a codegen SPI implementation

[cescoffier]

Hello,

I believe it's mostly a question for @aloubyansky, but others may be interested.
I need to resolve (download) a specific artifact during the codegen phase.

This is about the gRPC code generator. We need 2 native dependencies: protoc and the protoc-gen-plugin.
Currently, these artifacts are dependencies. But it's causing issues (too many downloads, arch-based profiles not supported in Gradle...).

My idea was to:

1. if these artifacts are dependencies, use that (but it would require the application to depend on these instead of the code generator like right now
2. if these artifacts are not dependencies, we would resolve the two dependencies for the host OS/arch - of course, it will download them only once and then use the cached version from the local repository

But I need a way to work on Quarkus CLI, Maven, and Gradle. My idea was to inject the ModelResolver into the Code generator. But I'm not sure I can access it at that point.

Note that tomorrow, I will start another related discussion about the regular extensions (not code generators) that could add and resolve dependencies during a build step.

[aloubyansky]

You mean you want to download them depending on the requested build outcome?

[cescoffier]

I just want to download an arbitrary dependency. I'm not sure the model resolver is the best approach, but it seems to be able to resolve artifacts (not sure it supports classifiers)

[aloubyansky]

It'll raise quite a few points to consider.

• Resolver API: the Maven one? How would that be implemented for Gradle projects?
• Resolver configuration: should it be in-line with project's Maven/Gradle configuration?
• Dependency version alignment: should the resolved artifacts be allowed to conflict with the rest of the application dependencies and if not, how would that be enforced?
• Manifesting/SBOM: assuming these dependencies are not declared but resolved "arbitrarily" using a resolver API directly, how could they be manifested for vulnerability management?

Before expose resolver API to build steps, let's clearly state the use-cases we need to support. IMO, we should prefer a declarative approach to dependency configuration rather than the API at runtime.

[cescoffier]

Resolver API: the Maven one? How would that be implemented for Gradle projects?

I was hoping we have some abstraction on top of that. As I mentioned in the other discussion, the solution must work regardless the build tool.

Resolver configuration: should it be in-line with project's Maven/Gradle configuration?

I would say yes. Because we will potentially download artifacts, we need to enforce proxies and repository settings.

Dependency version alignment: should the resolved artifacts be allowed to conflict with the rest of the application dependencies, and if not, how would that be enforced?

My idea was to keep a single dependency in the extension, retrieve the resolved dependency, and use that specific version. Thus, it gets enforced.

Manifesting/SBOM: assuming these dependencies are not declared but resolved "arbitrarily" using a resolver API directly, how could they be manifested for vulnerability management?

Can the SBOM be extended using build items?

The use case is the one I described: how to handle cross-platform execution of JVM application. So far, we require the user to add the dependencies explicitly. Users may not be aware of what brotli or rocksdb is.

[aloubyansky]

AFAIU, the use-case you are describing can be implemented with a declarative approach, it doesn't require designing and implementing a new resolver API and making sure it works for Maven and Gradle respecting repository settings.

My idea was to keep a single dependency in the extension, retrieve the resolved dependency, and use that specific version. Thus, it gets enforced.

That doesn't explain how it gets enforced though. AFAIU, it's about allowing any extension to resolve any dependency during the build.

[cescoffier]

How would a declarative approach work? (without asking the user to explicitly declare the dependencies)

[aloubyansky]

That would be something to design. It would mean expressing conditions for a dependency to be added.
AFAIU, that would be an extension developer who would define a condition. It could be added to the extension metadata in a somewhat similar way we support conditional dependencies today.

BTW, another point I didn't mention is that, so far, today a user is still in control which dependencies are resolved (even the conditional ones), for example they can be excluded or a preferred version can be enforced by a user. That's another advantage of a declarative approach.

[cescoffier]

Conditional dependencies would work.
Basically, it would mean adding metadata to the extension metadata, right?

For the code generators, the target is always the host. So, it should not be a problem to resolve a conditional dependency for the host. Unfortunately, every native dependency uses classifiers differently to express the OS/Arch (but that's an implementation detail).

[cescoffier]

So, I've tried a few things with what we have right now. It's not particularly convincing (see #43992).

The problem is that we cannot use exclusions because of the packaging of the artifact to download (using classifier).

\CC @maxandersen (as you mention the idea to filter things out ... which unfortunately does not work)