Add private/auth awareness to datasource index

[rarkins]

To simplify datasource logic and particularly caching, we'd like that the centralized datasource dispatcher (index file) could be responsible for caching. i.e.

• Before a getReleases request is sent to a datasource, the dispatcher can look up the cache and return a result. This saves each datasource from implementing their own caching
• After a result is returned to the dispatcher, it caches the result if the request was not private.

Potentially, private results could be cached too if we can assure we only return them when the same credentials are provided as the original request.

Therefore the dispatcher needs the ability to know if a result is private. I think we can simplify part of that by defining private as "if custom authentication was included with the request", i.e. Bearer or Basic authentication.

The dispatcher can only know this if either (a) responsibility for reading hostRules moves from the http module into dispatcher, or (b) the http module has a way to "pass back" the information that a custom authentication header was applied.

Point (a) does not seem possible because currently authentication headers are sometimes inserted by certain datasources too, not just by the http module.

If using point (b) then it means the dispatcher can't know all possible auth for all datasources so can simply implement "only cache public packages". Datasources could return back an optional new boolean field private.

An improved future solution could look like:

• Remove all custom auth from datasources, including .npmrc parsing in npm. Instead if should be converted to packageRules and hostRules during the extract phase
• Have the datasource dispatcher understand hostRules, and apply them before passing to the individual datasource

This approach could also solve our desire for path-based hostRules (#5825).

Note: self-hosted users could also choose via a new config option to cache all results, private or not - if there is no trust boundary between repositories. Doing so could result in a performance improvement compared to today.

[rarkins]

Current use of hostRules in datasources:

• docker uses them to check for insecureRegistry
• npm applies a Bearer token manually. Not sure if this is even needed
• packagist applies and checks them but only to make sure it doesn't cache private

docker also does some ECR authentication and writes directly to opts.authorization

[rarkins]

Probably the biggest uncertainty is around .npmrc handling.

I can think of four scenarios for in-repo .npmrc:

• .npmrc has all auth credentials necessary. Renovate can read from it to get credentials for its own lookup, and leave it alone when updating lockfiles
• .npmrc is present but has environment variables as placeholders instead of credentials. Users would need to configure Renovate explicitly with credentials and have it replace them in .npmrc before npm is run (setting env variables is simplest, but we could also overwrite the file)
• .npmrc is present but credentials are completely absent. Renovate needs to either append credentials to the file prior to npm
• .npmrc is absent. Users need to add credentials in Renovate config and Renovate needs to write them to disk before updating lock files

Things to note:

• If a repo .npmrc contains environment variables but those variables aren't in env, then the lock file update will be aborted by npm, yarn, etc
• Contents of npmrc in pwd will be combined with any .npmrc within the current user home dir. This means we can potentially write out our generated .npmrc to /home/ubuntu/.npmrc in our renovate/node container. But it will fail if the pwd .npmrc contains unresolved variables
• Allowing users to specify custom env variables for child processes is probably something we should support anyway
• In theory it's possible to have different .npmrc files in different subdirectories each with different tokens. What's the chance we need to support that?

[rarkins]

First step is #8470, which returns authorization: true if any authorization header was set in the request.

Some datasources like GitHub, npm and Docker Hub might have authorization set for every request, but that doesn't mean every result returned is private. Instead, there needs to be secondary logic added to confirm private/public through other means if the registry itself doesn't directly indicate it in metadata.

Potentially such datasources could implement a new function isPrivate() that the datasource can call - and cache aggressively - and combine with the results. i.e. so that the same logic isn't duplicated across many datasource implementations.

Our approach should be either:

• Default to private: true if unknown, or
• Use "true", "false", and "unknown"

I think maybe keeping it as boolean but allowing null or undefined to mean "unknown" should work.

[Chumper]

An improved future solution could look like:

* Remove all custom auth from datasources, including `.npmrc` parsing in npm. Instead if should be converted to packageRules and hostRules during the extract phase
* Have the datasource dispatcher understand hostRules, and apply them before passing to the individual datasource

I like this. Additionally I would not base the decision if a result should be cached onto the isPrivate field.

Instead the datasource should be able to return a cacheKey for the getReleases request which can be used to decide if the case is being used or not.

In that case the datasource doesn't need to care about whether a datasource is public or private.
Instead it can return just a cacheKey which indicates what to cache based on what it think should be grouped.

E.g.

Lets simplify and say that the docker resource should always be cached, in that case the datasource could return docker/<image> as cacheKey, so that all repos with the same request would access the cache.

For another datasource it could incorporate the auth tokens into the cachekey, like npm/<shortened_md5_of_token>/package.
With that we would have additional benefit of supporting cached entries for kind of tenants.

The github release datasource could incorporate if the repository is private/public.

This would move the logic on what to cache to the datasource but the actual caching will happen in the layer above.

[rarkins]

Let's test the idea of the datasource index using hostRules to lookup token or username/password based on registryUrl.

Shortcomings:

• hostRules supports baseUrl and users may have configured it to be longer/more specific than the registryUrl. Therefore the datasource index may "miss" the credentials
• The same problem means you can't support different credentials per-host per-datasource, e.g. a custom GitHub token for one particular repo that overrides the platform token

Both the above can likely be addressed if credentials have the possibility to be host + package-aware, e.g.

• hostRules with package awareness (to be understood only by the datasource index)
• packageRules with host and credentials awareness

If we did the latter, would that replace hostRules with credentials, or even hostRules altogether?

Another challenge: hostRules supports domainName, hostName, and baseUrl. Would we need to support all for packageRules, or would registryUrl be enough? I think matchRegistryUrls should be enough. i.e. most likely people would add rules with matchDatasources only, or combined with matchRegistryURls, or potentially with matchPackagePatterns.