"File already exists" error installing jupyter-contrib-nbextensions on 0.10.0+

[davidhao3300]

🐞 bug report

Affected Rule

pip_parse

Is this a regression?

Yes - confirmed working on 0.9.0, and broke with 0.10.0

Description

🔬 Minimal Reproduction

All in a single folder:

WORKSPACE:

workspace(name = "example")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "56dc7569e5dd149e576941bdb67a57e19cd2a7a63cc352b62ac047732008d7e1",
    strip_prefix = "rules_python-0.10.0",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.10.0.tar.gz",
)

load("@rules_python//python:pip.bzl", "pip_parse")

# Create a central repo that knows about the dependencies needed from
# requirements_lock.txt.
pip_parse(
    name = "pip_deps",
    requirements_lock = "@example//:requirements.txt",
)

load("@pip_deps//:requirements.bzl", "install_deps")

install_deps()

BUILD.bazel

load("@pip_deps//:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_library")

py_library(
    name = "example",
    deps = [
        requirement("jupyter-contrib-nbextensions"),
    ],
)

requirements.txt:

argon2-cffi-bindings==21.2.0
argon2-cffi==21.3.0
async-generator==1.10
attrs==21.4.0
backcall==0.2.0
bleach==4.1.0
cffi==1.15.1
dataclasses==0.8
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
importlib-metadata==4.8.3
ipykernel==5.5.6
ipython-genutils==0.2.0
ipython==7.16.3
jedi==0.17.2
jinja2==3.0.3
jsonschema==3.2.0
jupyter-client==7.1.2
jupyter-contrib-core==0.4.0
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.9.2
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.5.0
jupyterlab-pygments==0.1.2
lxml==4.9.1
markupsafe==2.0.1
mistune==0.8.4
nbclient==0.5.9
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.5
notebook==6.4.10
packaging==21.3
pandocfilters==1.5.0
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.14.1
prompt-toolkit==3.0.30
ptyprocess==0.7.0
pycparser==2.21
pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.0
python-dateutil==2.8.2
pyyaml==6.0
pyzmq==23.2.0
send2trash==1.8.0
six==1.16.0
terminado==0.12.1
testpath==0.6.0
tornado==6.1
traitlets==4.3.3
typing-extensions==4.1.1
wcwidth==0.2.5
webencodings==0.5.1
zipp==3.6.0
setuptools==44.0.0

Run bazel build ...

🔥 Exception or Error

 (Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel
/__main__.py", line 4, in <module>
    main()
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel
/__init__.py", line 69, in main                                                                                                                                             bazel.extract_wheel(
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/rules_python/python/pip_install/extract_wheels/lib/bazel.py", line 368, in ext
ract_wheel
    whl.unzip(directory)
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/rules_python/python/pip_install/extract_wheels/lib/wheel.py", line 90, in unzi
p
    installer.install(
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/pypi__installer/installer/_core.py", line 109, in install
    record = destination.write_file(
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/pypi__installer/installer/destinations.py", line 203, in write_file
    return self.write_to_fs(
  File "/home/devbox/.cache/bazel/_bazel_devbox/9b5a4447dba2b43a5957aed0e30d112f/external/pypi__installer/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: ./bin/jupyter-contrib-nbextension
)

🌍 Your Environment

Operating System: Ubuntu 18.04

Output of bazel version:

Build label: 4.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Aug 30 15:17:47 2021 (1630336667)
Build timestamp: 1630336667
Build timestamp as int: 1630336667

Rules_python version:

0.10.2

[groodt]

This is a bug in the upstream package.
https://github.com/ipython-contrib/jupyter_contrib_nbextensions/blob/2248f6e15753d0eb4339215313ae21ba09cbdf74/setup.py#L103

In setuptools, scripts are deprecated in favour of entry_points. In this instance, this distribution package has 2 scripts with the same name. The fix is to remove the script in setup.py.

[davidhao3300]

Despite the upstream bug (which should be fixed), this did work before in rules_python 0.9.0. Not sure if there's other cases where this can happen - is there a reason why this used to work but now broke?

[davidhao3300]

And just for my own understanding, are the 2 same scripts caused by:
https://github.com/ipython-contrib/jupyter_contrib_nbextensions/blob/master/setup.py#L90 and https://github.com/ipython-contrib/jupyter_contrib_nbextensions/blob/master/setup.py#L103 ?

[groodt]

Yes, those 2 scripts overwrite. The console_script entry_point will overwrite the script. You can verify this yourself outside rules_python by installing that package into a venv.

We now use a PEP compliant installer. All PEP compliant installers will reject that distribution package. We do not support invalid distribution packages and the bug should be fixed upstream by removing the script entry.

[davidhao3300]

Great, thank you

[dougthor42]

I saw the same thing with jupyter-qtconsole.

Followup: if/when pypa/installer#216 gets merged, we can then patch pip_installer with some workarounds:

diff --git a/python/pip_install/tools/wheel_installer/wheel.py b/python/pip_install/tools/wheel_installer/wheel.py
index 750ebfc..51f1a28 100644
--- a/python/pip_install/tools/wheel_installer/wheel.py
+++ b/python/pip_install/tools/wheel_installer/wheel.py
@@ -606,6 +606,10 @@ class Wheel:
             "scripts": "/bin",
             "data": "/data",
         }
+
+        # Hack
+        overwrite = "qtconsole" in str(self.path)
+
         destination = installer.destinations.SchemeDictionaryDestination(
             installation_schemes,
             # TODO Should entry_point scripts also be handled by installer rather than custom code?
@@ -613,6 +617,7 @@ class Wheel:
             script_kind="posix",
             destdir=directory,
             bytecode_optimization_levels=[],
+            overwrite_existing = overwrite,
         )
 
         with installer.sources.WheelFile.open(self.path) as wheel_source: