Adds support to spk commands for requests files

[dcookspi]

This adds support to spk commands (bake, env, and explain) to let them take files of package requests with a ` request on their command lines.

The aim is to allow a larger number of package requests to be passed to spk command than would normally fit on a single command line. We're seeing examples of tools calling spk bake or spk explain with 350+ detailed package requests, e.g. python/3.10.8+r3/ABCDEF or my-long-tool-name:{run,build,with-some-feature}/==1.2.3.4+r1, not just python or mypkg. Some of them are starting to hit command line limits and/or are becoming unwieldy to edit.

Each file must a yaml formatted list of package or var requests.

Example file (can have whatever is valid for a package request, json blob, '@' specifier, etc.):

- python-pytest-watch/4
- python-pytest-cov/>=3.0.0
- python-pytest-asyncio/~1.0.0
- python-pytest-timeout
- mypkg@build
- { pkg: python-pylint/2.0.0, preReleasePolicy: IncludeAll }

Example command lines:

# 2 packages on the command line
> spk explain python/3 python-pytest

# a number of packages in a file
> spk explain myfile.spk.yaml

# mixing packages on the command line and from files
> spk explain python myfile.spk.yaml python-pytest myotherfile.spk.yaml

The package requests in the requests files are added to the list of requests first, before the ones on the command line.

See also:

• Add list of layers file to spfs run command #1136

[rydrman]

From the meeting today:

• given that we want to eventually support spawn-like behaviour, could this be wrapped in an API object?

  api: v0/requirements
  requirements:
    - var: python.abi/cp39
    - pkg: python-pytest-watch/4

  We would need to update all the existing commands that take requests to read spec files even if @ is not in the name, but instead if it's not a valid request and ends in .spk.yaml. Then, we can handle adding whatever api object was loaded to the solver.

  Deprecate the existing options-file flag since this will be able to accomplish the same thing. You can still pass --opt on the command line and these will override anything in the file(s) - regardless of ordering they are processed last.

  # py.spk.yaml  -> contains opt for python.abi=cp27 debug=yes
  # pkg.spk.yaml -> used python.abi and debug in its template
  spk env --opt python.abi=cp37 py.spk.yaml pkg.spk.yaml@build

  In this case, we expect to get python.abi=cp37 and debug=yes because the requirements file is specified before the package spec file and the command line opts always take precedence. This means that we will need to incrementally build and rebuild the opts as things are loaded, maintaining the overrides each time.

• how do we handle this in spfs? this would work nicely into the existing live layer parsing with a new api: v0/platform or the like

• we need to document the spfs live layer file and basic workflow (unless I missed it) and include the above semantics as well in the docs somewhere for spk.

[dcookspi]

Updated this from the last meeting's feedback. Both pkg and var requests are supported now. The var requests are also used to make options for future templating renders inside parse_requests. The conversion from a var request to an OptionMap entry is a bit forced. Another way of doing that would be to separate out options into another field inthe requests file. I've put two commented out ways of doing that in the requests file struct as a starting point for further discussion on the options.

[rydrman]

From the meeting today:

• reads list of requests, both added to the solver
• vars are also added to the options for templating
• command line opt overrides anything in the file
• decisions
  • change --opt so that it doesn't add var request
  • add options field to file spec for options that don't become vars (same as new command line flag)
  • keep requirements field in spec for any kind of request (pkg or var)

Example:

api: v0/requirements
options: 
  pypk.version: 1.0.0
  python.abi: cp39
requirements:
  - pkg: python/3
  # we can't do this yet... but should be able to once #1135 is done
  - var: '{{ opts.python.abi }}' 
  - var: python.abi/cp39
  - {pkg: other, fromBuildEnv: true} # this is already an issue #1137

[dcookspi]

I've added the options field to the requests/requirements files, and updated the processing to keep command line opts as overrides with priority for templating and solver options. I've removed the lines that turned the command line options into var requests, and confirmed these options don't seem to influence the solve now.

This should be ready for another review.