Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optional Dependencies for conda v2 #55

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Optional Dependencies for conda v2 #55

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 125 additions & 0 deletions cep-draft.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
<table>
<tr><td> Title </td><td> Optional Dependencies</td>
<tr><td> Status </td><td> Draft </td></tr>
<tr><td> Author(s) </td><td> Travis Hathaway &lt;travis.j.hathaway@gmail.com&gt;</td></tr>
<tr><td> Created </td><td> June 7, 2023</td></tr>
<tr><td> Updated </td><td> June 7, 2023</td></tr>
<tr><td> Discussion </td><td> NA </td></tr>
<tr><td> Implementation </td><td> NA </td></tr>
</table>

[python-extras]: https://peps.python.org/pep-0508/#extras
[cargo-features]: https://doc.rust-lang.org/cargo/reference/features.html
[original-cep]: https://github.com/conda-incubator/ceps/pull/9
[conda-issue-7502]: https://github.com/conda/conda/issues/7502
[conda-issue-11053]: https://github.com/conda/conda/issues/11053
[mamba-issue-245]: https://github.com/mamba-org/boa/issues/245

## Abstract

This CEP proposes a new method for including optional dependencies when installing conda packages.
Optional dependencies are largely inspired by [Python's "extras"][python-extras] and
[Cargo's "features"][cargo-features]. When implemented, package builders will have a way to specify
dependencies which are not required but if installed would enable different features that are
not included by default.

## Motivation

Optional dependencies as a feature has been requested for conda compatible package managers for quite
some time (e.g. [conda-7502][conda-issue-7502], [conda-11053][conda-issue-11053], and [mamba-245][mamba-issue-245]).
Adding this feature to the conda ecosystem would also give package managers greater flexibility when
instructing users how to install and use their software. For example, a package could support multiple
variants by using this methodology. If a package supported either `seaborn` or `matplotlib`,
users could specify either `seaborn` or `matplotlib` as an optional dependency.


## Specification

This section will first show how this works from the end user point of view and then talks about
how it will work for package builders. Updates to the `repodata.json` format will also be discussed.

### User point of view

When a user goes to install a package, they will specify optional dependencies with the following syntax:

```bash
# For a single optional dependency
conda install package_name[optional_a]

# For multiple optional dependencies
conda install package_name[optional_a,optional_b]
```

Optional dependencies can also be specified by using the keyword argument syntax:

```bash
# For a single optional dependency
conda install package_name[extras=optional_a]

# For multiple optional dependencies
conda install package_name[extras=optional_a,optional_b]
Comment on lines +59 to +60
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the parsing here work? It looks like it would break the *args,**kwargs pattern proposed in conda/conda#11053 (which you're referencing already).

I'm also wondering if we really need two ways to do this? package_name[optional_a,optional_b] looks perfectly alright to me, and matches with what pip is doing.

While it's out of scope for this CEP, I'd really like to keep the door open to eventually extend the syntax to be able to use (exposition only, all kwargs need a proper design)

conda install package_name[optional_a,optional_b,cuda=True,license=no_gpl,cpu=avx512,...]

It seems to me that if we add the extras= way to specify optional dependencies, we don't gain much and risk making it much harder to eventually solve that problem.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for following the pip syntax.

```

Later, when running commands like `conda list` the packages will show up as "virtual metapackages", which
look like the following:

```bash
conda list

# Name Version Build Channel
package_name 0.1.0 h0c17e10_0 conda-forge
package_name@optional_a 0.1.0 pyhd8ed1ab_0 conda-forge
package_name@optional_b 0.1.0 pyhd8ed1ab_0 conda-forge
```

When a user wants to later remove an optional dependency, they do so by referencing
these "virtual metapackages":

```bash
conda remove package_name@optional_a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how to feel about the new syntax for an arguably odd case. Why do we need it to begin with? If one wants to remove the extras, they could use the same spec à la package_name[optional_a]? And if you just want to remove the extras, they can simply spell them out.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but I think there are good use cases for something like the “meta-packages” to exist:

  1. Install package_name[optional_deps]
  2. package_name gets an update and the extra “optional_deps” adds another dependency
  3. You upgrade the (automatically, with the other packages) package_name@optional_deps package and also get the new dependency that was added with the upgrade.

Otherwise, if there are no “virtual meta-packages” (or anything that keeps track of the extras you installed) the user wouldn't get the new dependency.

Also, another thing that I would love to have, but which would differ from the way pip does it (at least to my knowledge), is to not stop at optional dependencies, but go further with optional modules/files.
Let's say my project has the following structure:

src/package/...
src/package/core/...
src/package/feature_a/...
src/package/feature_b/...

And in the meta.yaml I could configure it like this (just an example):

# if files get specifically included in an extra,
# they would be excluded from the "core" package
extras:
    feature_a:
        include_files: ["feature_a/"]
        dependencies: ["optional1",..]
    feature_b:
        include_files: ["feature_b/"]
        dependencies: ["optional2",...]
# maybe also something like feature_c
    feature_c:
        include_extras: ["feature_a","feature_b"]
# optionally also include further files/dependencies etc.

Then conda install package, would only include /core.
And conda install package[feature_a] would include /core and /feature_a and feature_a's dependencies.
And conda install package[feature_c] would include everything.

```


### Packager point of view

**TBD**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Points to take into account here:

  • How will this be specified in the conda-build recipes
  • How will this be exposed in repodata.json
  • How will this be reflected in the info/ metadata folder inside the package



## Rationale

**TBD**


## Backwards compatibility

**TBD**


## Sample implementation

**TBD**


## FAQ

**TBD**


## Resolution

**TBD**


## References

- [Python extras][python-extras]
- [Cargo features][cargo-features]
- [First CEP for optional dependencies by @wolfv][original-cep]
- [Conda issue: "Optional groups of dependencies"][conda-issue-7502]
- [Conda issue: "ENH: More powerful syntax for build variants & optional package-extras"][conda-issue-11053]
- [Mamba issue: "[recipe-spec] Allow for arbitrary optional dependencies"][mamba-issue-245]


## Copyright

All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).