ModuleNotFoundError when specifying relative dependency

[ianliu]

I've noticed that poetry fails to add a relative dependency with ModuleNotFoundError: No module named 'setuptools'. My test is as follows:

cd /tmp
poetry new foo
poetry new bar
cd foo
poetry add ../bar

This does not happen if I do exactly the same thing in a different directory, like /tmp/asdf/{foo,bar}. What I suspect is that /tmp/{foo,bar} where already used to create virtual environments in the past, as I usually test things on /tmp quite frequently.

I've tried to issue poetry env remove ... to remove any environment for /tmp/{foo,bar}, with no luck, the error always happens. What else could be interfering with this test?

Full error:

tmp$ poetry new bar
Created package bar in bar
tmp$ poetry new foo
Created package foo in foo
tmp$ cd foo/
foo$ poetry add ../bar

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing bar (0.1.0 /tmp/bar): Failed

  EnvCommandError

  Command ['/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/bin/pip', 'install', '--no-deps', '-U', '/tmp/bar'] errored with the following return code 2, and output: 
  Processing /tmp/bar
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
      status = run_func(*args)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
      return func(self, options, args)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 338, in run
      requirement_set = resolver.resolve(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
      collected = self.factory.collect_root_requirements(root_reqs)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
      req = self._make_requirement_from_install_req(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
      cand = self._make_candidate_from_link(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
      self._link_candidate_cache[link] = LinkCandidate(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
      super().__init__(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
      self.dist = self._prepare()
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
      dist = self._prepare_distribution()
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
      return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
      return self._prepare_linked_requirement(req, parallel_builds)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
      dist = _get_prepared_distribution(
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 47, in prepare_distribution_metadata
      self._install_build_reqs(finder)
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 106, in _install_build_reqs
      build_reqs = self._get_build_requires_wheel()
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 83, in _get_build_requires_wheel
      return backend.get_requires_for_build_wheel()
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "/tmp/pip-build-env-5z3l_yfv/overlay/lib/python3.10/site-packages/poetry/core/masonry/__init__.py", line 10, in <module>
      from .builder import Builder
    File "/tmp/pip-build-env-5z3l_yfv/overlay/lib/python3.10/site-packages/poetry/core/masonry/builder.py", line 7, in <module>
      from .builders.sdist import SdistBuilder
    File "/tmp/pip-build-env-5z3l_yfv/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/__init__.py", line 2, in <module>
      from .wheel import WheelBuilder
    File "/tmp/pip-build-env-5z3l_yfv/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 23, in <module>
      from packaging.tags import sys_tags
    File "/tmp/pip-build-env-5z3l_yfv/overlay/lib/python3.10/site-packages/poetry/core/_vendor/packaging/tags.py", line 7, in <module>
      import distutils.util
    File "/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 92, in create_module
      return importlib.import_module('setuptools._distutils')
    File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ModuleNotFoundError: No module named 'setuptools'
  
  WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
  You should consider upgrading via the '/home/ian/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/bin/python -m pip install --upgrade pip' command.
  

  at /usr/lib/python3.10/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│ 
      1197│         return decode(output)
      1198│ 
      1199│     def execute(self, bin, *args, **kwargs):


Failed to add packages, reverting the pyproject.toml file to its original content.

[abn]

@ianliu what version of poetry are you using? I am unable to reproduce this in a clean environment. Maybe try using poetry config virtualenvs.in-project true to rule out environment reuse.

podman run --rm -i --entrypoint bash docker.io/python:3.10 <<EOF
set -xe
python -m pip --disable-pip-version-check install -q poetry
cd /tmp
poetry new foo
poetry new bar
pushd foo
poetry add ../bar
EOF

+ python -m pip --disable-pip-version-check install -q poetry
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ poetry new foo
Created package foo in foo
+ poetry new bar
Created package bar in bar
/foo /
+ pushd foo
+ poetry add ../bar
Creating virtualenv foo-b2TG5iYf-py3.10 in /root/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 9 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.7)
  • Installing attrs (21.4.0)
  • Installing more-itertools (8.12.0)
  • Installing packaging (21.3)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing wcwidth (0.2.5)
  • Installing bar (0.1.0 /bar)
  • Installing pytest (5.4.3)

[ianliu]

So, I think I've made some progress. I was playing with my environment, and found out that if I mv ~/.cache/pypoetry{,.bak}, things start working again. I also tested that manually removing the virtual envs in ~/.cache/pypoetry/virtualenv/foo* makes this work again.

While testing, I also stumbled in another old environment that was dangling in my ~/.cache/pypoetry for another python version. I had a foobar-W7NWSgbT-py3.9 for an older version of python, and this environment was being used in my newly created 3.10 poetry project, causing version constraint problems when adding dependencies.

I will try making an isolated test case for the later example, as the first example is too complicated to replicate. I guess it involves different versions of setuptool and whatnot.

Maybe poetry could be more resilient with dangling environments in the user's cache dir.

[abn]

The issue you are encountering is related to #3153 #4946. The issue has to do with the pip and setuptools versions iirc, some of them being buggy when the distutils vendoring hack started getting included. Once a virtualenvironment is created the pip/setuptool versions are never really updated.

The changes in #2826 should make poetry avoid the issue as it no longer depends on the environment provided pip/setuptools.

[ianliu]

Thanks for clarifying! I will mark this as answered.