feat: Support for optional `python_site_packages_path` in repodata #3579

jjhelmus · 2024-11-06T20:18:08Z

Add support for the optional python_site_packages_path field that can appear in repodata for python packages as a means to specify the location of the site-packages directory.

This optional field is defined in CEP-17.

partial implementation for #3558

Please be kind as this is my first contribution to mamba and the first time digging into the source.

jjhelmus · 2024-11-06T20:29:15Z

This is likely not a complete implementation for CEP-17 in mamba but may be enough to unblock libmamba-conda-solver (see conda/conda-libmamba-solver#560).

This is rough and likely needs some work to finish things up but was hoping for initial feedback (or someone else to take over 😄).

With these changes I was able to get the field to show up in the output of repoquery search:

./build/micromamba/mamba repoquery search -c jjhelmus/label/sp_path python=3.99.99  --json | jq .result.pkgs
[
  {
    "build": "0",
    "build_number": 0,
    "build_string": "0",
    "channel": "jjhelmus",
    "constrains": [],
    "depends": [
      "pip"
    ],
    "fn": "python-3.99.99-0.tar.bz2",
    "license": "",
    "md5": "2570015e0cb5829cbdfa9780b4564db9",
    "name": "python",
    "noarch": "generic",
    "python_site_packages_path": "lib/python3.99t/site-packages",
    "sha256": "a2be7c56f9ce3d6fc25220a49185894d02cf03b832bb9fb55122e6a662bde910",
    "size": 2912,
    "subdir": "noarch",
    "timestamp": 1726861043,
    "track_features": "",
    "url": "https://conda.anaconda.org/t/jj-80e459ed-f2bc-4975-ae2e-68f8e339b666/jjhelmus/label/sp_path/noarch/python-3.99.99-0.tar.bz2",
    "version": "3.99.99"
  }
]

JohanMabille · 2024-11-20T11:52:44Z

Hi, thanks for tackling this! The implementation is neat and looks good to me, can you add some test to check this is correctly working?

Also running pre-commit run --all-files should fix the linter.

jjerphan · 2025-01-08T09:47:27Z

Hi @jjhelmus, the support of CEP-17 is important for supporting Python 3.13t builds.

Are you still interested in continuing this PR?

jjhelmus · 2025-01-15T15:37:15Z

Hi @jjhelmus, the support of CEP-17 is important for supporting Python 3.13t builds.

Are you still interested in continuing this PR?

I can still work on this but progress will be slow. If anyone wants to take over this work that would be fine.

When I original started for on this, supported free-threading was a high priority for the team I was on. Since then, free-threading support is no longer a priority so I have very little time to work on this.

jaimergp · 2025-03-12T15:58:05Z

Hi! I'm reviewing conda/conda-libmamba-solver#628 and in there we are already assuming that python_site_packages_path will be the name of the attribute used in the libmambapy PackageRecord object. Can that be guaranteed even without this being merged yet? IOW, will you change the name of the attribute in this PR? Thanks!

Add support for the optional python_site_packages_path field that can appear in repodata for python packages to specify the location of the site-packages directory. This optional field is defined in CEP-17.

Add the `python_site_packages_path` attribute to the PackageInfo class in the libmambapy bindings module.

Use the python_site_packages_path parameter from the python package in the environment as the site_package_path when available. When not specified fall back to setting this from the python version.

jjhelmus · 2025-03-19T22:38:42Z

I found some time to work on this again and have been making some progress. Micromamba will now copy the contents of noarch: python packages into the location specified by the python_site_packages_path field when available in the package's repodata entry.

This can be checked using:

./build/micromamba/mamba create -p ./blah python=3.13 "python_abi=*=*_cp313t" imagesize -c conda-forge --yes
ls -d blah/lib/python3.13*

There should be a single directory, blah/lib/python3.13t and its site-packages subdirectory should contain the file from the imagesize package.

I'm still working on adding some tests, should have these complete by the end of the week.

jjhelmus · 2025-03-20T16:55:30Z

This PR is ready for review.

With these changes, noarch: python packages are installed into the location specific by the python_site_packages_path field of the python package when present. When not specified packages are into into the site-packages path determined from the version of the python package as is the existing behavior.

This field is exposed in libmambapy so that it can be used by conda-libmamba-solver and in turn conda.

There are tests at a unit and integration level.

Happy to address any concerns.

jjhelmus · 2025-04-01T13:28:01Z

Bumping this PR. Is there anything additional needed for a review?

jjhelmus · 2025-04-15T15:42:08Z

Bump again. Looking for a review or guidance on if anything is needed to merge this change required to fully support free-threading Python in the conda ecosystem.

jjerphan · 2025-04-15T18:19:04Z

We have other constraints at the moment, but we will try to have a look soon.

jjerphan

Thank you for your patience and this important contribution, @jjhelmus.

I have a few comment and suggestions.

I think we also need some integration tests for installation and usage of python3.13t installation, usage and uninstallation with mamba, in particular in the previous presence of python3.13 (e.g. must normal python packages for python3.13 be uninstalled for safety or are they ignored?)

jjerphan · 2025-04-16T06:23:53Z

libmamba/ext/solv-cpp/include/solv-cpp/solvable.hpp

+        /**
+         * Set the python_site_packages_path of the solvable.
+         *
+         * This is not used by libsolv and is purely for data storing.


Can you motivate this API by mentioning some context from CEP-17 for reference?

jjerphan · 2025-04-16T06:25:52Z

libmamba/ext/solv-cpp/tests/src/test_solvable.cpp

Can you extend those tests for some free threading builds of CPython?

jjerphan · 2025-04-16T06:28:53Z

libmambapy/tests/test_specs.py

+    pkg.python_site_packages_path = "lib/python3.99t/site-packages"
+    assert pkg.python_site_packages_path == "lib/python3.99t/site-packages"


Can you assert its value before it is set?

jjerphan · 2025-04-16T06:46:45Z

micromamba/tests/test_repoquery.py

+    assert res["query"]["type"] == "search"
+    assert "conda-forge" in res["result"]["pkgs"][0]["channel"]
+    assert res["result"]["pkgs"][0]["name"] == "python"
+    assert res["result"]["pkgs"][0]["version"] == "3.13.1"
+    assert res["result"]["pkgs"][0]["python_site_packages_path"] == "lib/python3.13t/site-packages"


Suggested change

assert res["query"]["type"] == "search"

assert "conda-forge" in res["result"]["pkgs"][0]["channel"]

assert res["result"]["pkgs"][0]["name"] == "python"

assert res["result"]["pkgs"][0]["version"] == "3.13.1"

assert res["result"]["pkgs"][0]["python_site_packages_path"] == "lib/python3.13t/site-packages"

assert res["query"]["type"] == "search"

package_info = res["result"]["pkgs"][0]

assert "conda-forge" in package_info["channel"]

assert package_info["name"] == "python"

assert package_info["version"] == "3.13.1"

assert package_info["track_features"] == "free-threading"

assert package_info["python_site_packages_path"] == "lib/python3.13t/site-packages"

jjerphan · 2025-04-16T06:50:06Z

micromamba/tests/test_create.py

+    assert os.path.isdir(env_prefix)
+
+    if platform.system() == "Windows":
+        assert os.path.isdir(env_prefix / "lib" / "site-packages" / "imagesize")


So Windows stores all packages together indifferently from the version of Python and now from the free threaded builds?

Can explicit tests be added for python=3.13t for Windows for the presence or absence of folders?

jjerphan · 2025-04-16T06:50:18Z

micromamba/tests/test_install.py

+    helpers.install("-n", env_name, "imagesize")
+
+    if helpers.platform.system() == "Windows":
+        assert os.path.isdir(env_prefix / "lib" / "site-packages" / "imagesize")


Same comment as above.

Hind-M · 2025-05-14T13:24:48Z

libmamba/ext/solv-cpp/src/solvable.cpp

@@ -206,6 +206,22 @@ namespace solv
        return set_license(str.c_str());
    }

+    auto ObjSolvableViewConst::python_site_packages_path() const -> std::string_view
+    {
+        return ptr_to_strview(::solvable_lookup_str(const_cast<::Solvable*>(raw()), SOLVABLE_MEDIABASE)


Thank you for working on this!

One concern (raised in this issue) would be that SOLVABLE_MEDIABASE seems to be used for the channel here.
channel and set_channel don't seem to be used though other than in the tests, but still, I'm not sure how using that placeholder could interfere with what libsolv could be doing...
Maybe @AntoinePrv has more insight?

Good catch! Basically for most attribute we use (other name name, version...), this is only a key:value store and has little importance which key we use (the point of these methods is to hide them as implementation details).

I think since for the channel it is used with another struct, it does not matter. Alternatively one could use another obscure key with proper comment.

jezdez · 2025-06-04T06:52:04Z

Hi @AntoinePrv! We're looking to merge this in conda and conda-libmamba-solver, and I wondered if there is a chance we can merge this soon?

AntoinePrv · 2025-06-04T08:00:50Z

Absolutely! @jjhelmus do have spare time to handle @jjerphan comments? I can then take on the rebase/conflicts if you'd like.

jjhelmus · 2025-06-04T21:04:24Z

I may have some time to look at this in the next two days but if not I will not be able to update this until the week of June 16.

jjhelmus mentioned this pull request Nov 6, 2024

Use python_site_packages_path when specified conda/conda#14256

Merged

3 tasks

jjhelmus changed the title ~~add optional python_site_packages_path in repodata~~ support for optional python_site_packages_path in repodata Nov 6, 2024

jaimergp mentioned this pull request Nov 7, 2024

Implement CEP #90 #3558

Open

JohanMabille added the release::enhancements For enhancements PRs or implementing features label Nov 20, 2024

jjerphan changed the title ~~support for optional python_site_packages_path in repodata~~ feat: Support for optional python_site_packages_path in repodata Nov 20, 2024

isuruf mentioned this pull request Jan 7, 2025

Status of Conda ecosystem support Quansight-Labs/free-threaded-compatibility#46

Open

3 tasks

Hind-M mentioned this pull request Jan 28, 2025

Support for optional python_site_packages_path in repodata openSUSE/libsolv#579

Open

jjhelmus force-pushed the py_sp_path branch from 17e18a8 to 4500c1a Compare March 5, 2025 14:15

jjhelmus mentioned this pull request Mar 5, 2025

add support for python_site_packages_path as defined in CEP-17 conda/conda-libmamba-solver#628

Open

3 tasks

jjhelmus added 3 commits March 19, 2025 14:27

add optional python_site_packages_path in repodata

9547ac1

Add support for the optional python_site_packages_path field that can appear in repodata for python packages to specify the location of the site-packages directory. This optional field is defined in CEP-17.

add python_site_packages_path to libmambapy bindings

0c61996

Add the `python_site_packages_path` attribute to the PackageInfo class in the libmambapy bindings module.

use python_site_packages_path in transaction

5e67196

Use the python_site_packages_path parameter from the python package in the environment as the site_package_path when available. When not specified fall back to setting this from the python version.

jjhelmus force-pushed the py_sp_path branch from 4500c1a to 5e67196 Compare March 19, 2025 21:34

jjhelmus added 2 commits March 19, 2025 16:59

add python_site_packages_path as field to PackageInfo

f0d945c

lint

7687583

jjhelmus added 8 commits March 19, 2025 17:47

test for package_info module

ced339e

test for repo_data module

45d95ee

test for solvable module

a87dd30

test libmambapy

6985a14

repoquery integration test

dd2cc99

add test for pyc compilation in free-threading python

3adec2e

add test for python_site_packages_path with noarch: python

ed1e27d

add test for installing noarch: python into a free-threading env

c403553

h-vetinari mentioned this pull request Apr 9, 2025

unpause nogil migration conda-forge/conda-forge-pinning-feedstock#6673

Draft

jjerphan reviewed Apr 16, 2025

View reviewed changes

Hind-M reviewed May 14, 2025

View reviewed changes

AntoinePrv approved these changes May 14, 2025

View reviewed changes

		pkg.python_site_packages_path = "lib/python3.99t/site-packages"
		assert pkg.python_site_packages_path == "lib/python3.99t/site-packages"

feat: Support for optional python_site_packages_path in repodata #3579

Are you sure you want to change the base?

feat: Support for optional python_site_packages_path in repodata #3579

Conversation

jjhelmus commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjhelmus commented Nov 6, 2024

Uh oh!

JohanMabille commented Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjerphan commented Jan 8, 2025

Uh oh!

jjhelmus commented Jan 15, 2025

Uh oh!

jaimergp commented Mar 12, 2025

Uh oh!

jjhelmus commented Mar 19, 2025

Uh oh!

jjhelmus commented Mar 20, 2025

Uh oh!

jjhelmus commented Apr 1, 2025

Uh oh!

jjhelmus commented Apr 15, 2025

Uh oh!

jjerphan commented Apr 15, 2025

Uh oh!

jjerphan left a comment

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

jjerphan Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

Hind-M May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AntoinePrv May 14, 2025

Choose a reason for hiding this comment

Uh oh!

jezdez commented Jun 4, 2025

Uh oh!

AntoinePrv commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjhelmus commented Jun 4, 2025

Uh oh!

Uh oh!

feat: Support for optional `python_site_packages_path` in repodata #3579

feat: Support for optional `python_site_packages_path` in repodata #3579

jjhelmus commented Nov 6, 2024 •

edited

Loading

JohanMabille commented Nov 20, 2024 •

edited

Loading

Hind-M May 14, 2025 •

edited

Loading

AntoinePrv commented Jun 4, 2025 •

edited

Loading