chore(deps): upgrade pex to latest

[alexeagle]

No description provided.

[github-actions]

e2e/use_release folder: LCOV of commit 74fc16c during CI #1693

Summary coverage rate:
  lines......: 100.0% (2 of 2 lines)
  functions..: 100.0% (1 of 1 function)
  branches...: no data found

Files changed coverage rate: n/a

[alexeagle]

hmm @thesayyn looks like this hacky bit needs a re-hack https://github.com/aspect-build/rules_py/blob/main/py/tools/pex/main.py#L112

[thesayyn]

hmm @thesayyn looks like this hacky bit needs a re-hack https://github.com/aspect-build/rules_py/blob/main/py/tools/pex/main.py#L112

eh, i was hoping we wouldn't need to update it again. I'll TAL tomorrow

[jsirois]

Pex maintainer here. FWIW importing pex is absolutely unsupported. The Pex CLI will never break you but the non-existent API definitely will. If someone can give me a high-level confirmation of what you're trying to do, I can hopefully point out how you might acheive that with the CLI.

[thesayyn]

Pex maintainer here. FWIW importing pex is absolutely unsupported. The Pex CLI will never break you but the non-existent API definitely will. If someone can give me a high-level confirmation of what you're trying to do, I can hopefully point out how you might acheive that with the CLI.

What we were trying with that patch was to do two things.

Add expanded wheels into deps here:

rules_py/py/tools/pex/main.py

Lines 161 to 171 in 55fd502

	for dep in options.dependencies:
	dist = Distribution.load(dep)
	# TODO: explain which level of inferno is this!
	key = "%s-%s" % (dist.key, dist.version)
	dist_hash = pex_builder._add_dist(
	path= dist.location,
	dist_name = key
	)
	pex_info.add_distribution(key, dist_hash)
	pex_builder.add_requirement(dist.as_requirement())

And set some environment variables

• RUNFILES_DIR

  rules_py/py/tools/pex/main.py

  Line 117 in 55fd502

  INJECT_TEMPLATE=["os.environ['RUNFILES_DIR'] = __entry_point__"]

  for bazel-runfiles pip package to work.

And append some paths to sys.path to support py_binary#imports so that relative imports work.

[thesayyn]

FWIW importing pex is absolutely unsupported
Yes, we were aware of that in this case and went with a custom cli after finding some of related issues that went stale. (been a long time can't remember what i saw)

[jsirois]

Thanks @thesayyn. A few questions:

Add expanded wheels into deps here:

What is an "expanded wheel"? Just unzipped, or fully installed (unpack + spread as specified here: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#details) or something else?

And set some environment variables

I think --inject-env is what you want. That's new within the last several years. No real doc except CLI --help, but mentioned here: https://docs.pex-tool.org/recipes.html#uvicorn-and-other-customizable-application-servers

And append some paths to sys.path to support py_binary#imports so that relative imports work.

There is no CLI option to build this into the PEX file itself like --inject-env, but there is this env var knob: https://docs.pex-tool.org/api/vars.html#PEX_EXTRA_SYS_PATH. If you need the modification baked into the PEX you can use the very old --preamble option. For example:

:; echo 'import os; os.environ["PEX_EXTRA_SYS_PATH"] = "foo:bar"' > preamble

:; pex cowsay -c cowsay --preamble preamble -o cowsay.pex

:; PEX_VERBOSE=1 ./cowsay.pex -t Moo!
pex: Laying out PEX zipfile /home/jsirois/dev/pex-tool/pexcz/cowsay.pex: 0.2ms
pex: Executing installed PEX for /home/jsirois/dev/pex-tool/pexcz/./cowsay.pex at /home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e
pex:   Testing /home/jsirois/.pyenv/versions/3.11.11/bin/python3.11 can resolve PEX at /home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e: 2.0ms
pex: Using the current interpreter /home/jsirois/.pyenv/versions/3.11.11/bin/python3.11 since it matches constraints and PYTHONPATH is not set.
pex: Discarding site packages path: Platlib(/home/jsirois/.pyenv/versions/3.11.11/lib/python3.11/site-packages)
pex: Tainted path element: /home/jsirois/.pyenv/versions/3.11.11/lib/python3.11/site-packages
pex: Tainted path element: foo
pex: Scrubbing from user site: /home/jsirois/.local/lib/python3.11/site-packages
pex: Scrubbing from site-packages: /home/jsirois/.pyenv/versions/3.11.11/lib/python3.11/site-packages
pex: Scrubbing from site-packages: foo
pex: Scrubbing from site-packages: /home/jsirois/dev/pex-tool/pexcz/foo
pex: Adding foo:bar to sys.path
pex: New sys.path: ['/home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e/.bootstrap/pex/vendor/_vendored/attrs', '/home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e/.bootstrap', '/home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e', '/home/jsirois/.pyenv/versions/3.11.11/lib/python311.zip', '/home/jsirois/.pyenv/versions/3.11.11/lib/python3.11', '/home/jsirois/.pyenv/versions/3.11.11/lib/python3.11/lib-dynload', 'foo', 'bar']
pex: Activating PEX virtual environment from /home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e: 0.2ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: PYTHONPATH contains:
pex:     /home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e
pex:   * /home/jsirois/.pyenv/versions/3.11.11/lib/python311.zip
pex:     /home/jsirois/.pyenv/versions/3.11.11/lib/python3.11
pex:     /home/jsirois/.pyenv/versions/3.11.11/lib/python3.11/lib-dynload
pex:   * foo
pex:   * bar
pex:     /home/jsirois/.cache/pex/installed_wheels/0/274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a/cowsay-6.1-py3-none-any.whl
pex:     /home/jsirois/.cache/pex/unzipped_pexes/1/e18d55d51b2bbad12f8e13eccd0de5cb994eee1e/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport
  ____
| Moo! |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Note the verbose lines just above the cow that start with pex: PYTHONPATH contains:.

[chrisirhc]

What is an "expanded wheel"? Just unzipped, or fully installed (unpack + spread as specified here: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#details) or something else?

I believe it's unpack + spread. Does pex support this (I wasn't able to find out from doing reading pex --help)?

Here's the files in cowsay packaged referenced from the examples/py_pex_binary:

external/pypi_cowsay
├── bin
│   └── cowsay
├── BUILD.bazel
├── cowsay-6.1-py3-none-any.whl
├── MODULE.bazel
├── MODULE.bazel.lock
├── REPO.bazel
├── rules_python_wheel_entry_point_cowsay.py
├── site-packages
│   ├── __init__.py
│   ├── cowsay
│   │   ├── __init__.py
│   │   ├── __main__.py
│   │   ├── characters.py
│   │   ├── main.py
│   │   └── tests
│   │       ├── __init__.py
│   │       ├── solutions.py
│   │       ├── test_api.py
│   │       └── test_output.py
│   └── cowsay-6.1.dist-info
│       ├── entry_points.txt
│       ├── INSTALLER
│       ├── LICENSE.txt
│       ├── METADATA
│       ├── RECORD
│       ├── top_level.txt
│       └── WHEEL
└── WORKSPACE

There's a whl file in here but I believe it can't be used since there can be patches applied on top of it after unpack + spread.

[jsirois]

I believe it's unpack + spread. Does pex support this (I wasn't able to find out from doing reading pex --help)?

Pex internally uses unpacked + spread wheels in its cache ... which uv does too. And after reading the code a bit here, I see you're just using uv effectively via some of its crates. Right now you can build a PEX with:

1. A set of input requirements you let Pex resolve.
2. A Pex --lock you created earlier with pex3 lock create ... (or a subset of the lock via --lock my.lock req1 req2 where req1 and req2 must be contained in the lock ... then the transitive closure of these in the lock is used).
3. A --pylock PEP-751 lock which Pex will download sdists and wheels from.
4. A set of --pre-resolved-dists or a --pre-resolved-dists directory where the specified dists must be either sdists or wheels and must form a dependency closure.

You cannot build a Pex with an unpacked or unpacked + spread wheel from the CLI right now.

I think this issue tracks the machinery you need: pex-tool/pex#1361
That says build a PEX from a venv, but that is what you effectively have here with the site-packages / bin spread in the cowsay example.

For related uv issues see:

• materialize wheels (actual zipped up wheels) support: Support pip wheel subcommand astral-sh/uv#1681
• materialize sdists or wheels support: Support pip download astral-sh/uv#3163

[jsirois]

@chrisirhc, @arrdem and @thesayyn Pex 2.59.0 is now available and supports a new --venv-repository resolver. If you point pex at a venv with that new option, it will resolve any requirements you give it from that venv fully off line (unless the venv contains old-style .egg-info sdist installs - for those it does a fresh intransitive resolve using pip per-each - but you avoid any of those with your uv infra). If you hand Pex no requirements it turns all the distributions in the venv into a PEX; otherwise it uses the root requirements you give it to subset the venv portion needed to satisfy just those root requirements. Hopefully that feature can help you transition to just using the Pex CLI. Let me know if you have questions or there are any remaining missing knobs you might need.

[arrdem]

Thanks @jsirois! I've got some other stuff in the ruleset to wrap up first but that sounds like a much cleaner approach to PEX building than what we currently have, especially if we can leverage the venv building machinery I've been hacking on to create the required trees at build time.