Upgrading pip on 24.04 base fails

[carlcsaposs-canonical]

Bug Description

python3 -m pip install --upgrade pip==24.3.1 fails on Ubuntu 24.04

Building poetry-deps
:: + python3 -m pip install --upgrade pip==24.3.1
:: error: externally-managed-environment
::
:: × This environment is externally managed
:: ╰─> To install Python packages system-wide, try apt install
::     python3-xyz, where xyz is the package you are trying to
::     install.
::
::     If you wish to install a non-Debian-packaged Python package,
::     create a virtual environment using python3 -m venv path/to/venv.
::     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
::     sure you have python3-full installed.
::
::     If you wish to install a non-Debian packaged Python application,
::     it may be easiest to use pipx install xyz, which will manage a
::     virtual environment for you. Make sure you have pipx installed.
::
::     See /usr/share/doc/python3.12/README.venv for more information.
::
:: note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
:: hint: See PEP 668 for the detailed specification.
'override-build' in part 'poetry-deps' failed with code 1.

Expected behavior

charmcraft does not mark the environment as externally managed (https://packaging.python.org/en/latest/specifications/externally-managed-environments/#recommendations-for-distros), or the docs (e.g. https://canonical-charmcraft--2053.com.readthedocs.build/en/2053/howto/charm-to-poetry/#select-compatible-versions-of-pip-and-poetry) include instructions to use --break-system-packages

To Reproduce

git clone https://github.com/canonical/mysql-router-k8s-operator
cd mysql-router-k8s-operator
git checkout 294cf21ca596d5fda30416a2e9f945db0c8aa9fc
charmcraft pack -v --platform ubuntu@24.04:amd64

Environment

Ubuntu 22.04

$ snap list
Name              Version                  Rev    Tracking            Publisher          Notes
charmcraft        3.2.2.post133+g455880a4  5682   3.x/edge            canonical✓         classic
core18            20240920                 2846   latest/stable       canonical✓         base
core20            20240911                 2434   latest/stable       canonical✓         base
core22            20241119                 1722   latest/stable       canonical✓         base
core24            20240920                 609    latest/stable       canonical✓         base
juju              3.6.1                    29241  3/stable            canonical✓         -
lxd               5.0.4-497fe1e            31333  5.0/stable/…        canonical✓         -
microk8s          v1.29.11                 7435   1.29-strict/stable  canonical✓         -
snapcraft         8.5.1                    13181  latest/stable       canonical✓         classic
snapd             2.66.1                   23258  latest/stable       canonical✓         snapd

charmcraft.yaml

# Copyright 2022 Canonical Ltd.
# See LICENSE file for licensing details.

type: charm
platforms:
  ubuntu@22.04:amd64:
  ubuntu@22.04:arm64:
  ubuntu@24.04:amd64:
  ubuntu@24.04:arm64:
# Files implicitly created by charmcraft without a part:
# - dispatch (https://github.com/canonical/charmcraft/pull/1898)
# - manifest.yaml
#   (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L259)
# Files implicitly copied/"primed" by charmcraft without a part:
# - actions.yaml, config.yaml, metadata.yaml
#   (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L290-L293
#   https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L156-L157)
parts:
  # "poetry-deps" part name is a magic constant
  # https://github.com/canonical/craft-parts/pull/901
  poetry-deps:
    plugin: nil
    build-packages:
      - curl
    override-build: |
      python3 -m pip install --upgrade pip==24.3.1  # renovate: charmcraft-pip-latest
      curl -sSL https://install.python-poetry.org | python3 - --version 1.8.4  # renovate: charmcraft-poetry-latest
      ln -sf "$HOME/.local/bin/poetry" /usr/local/bin/poetry
  # "charm-poetry" part name is arbitrary; use for consistency
  # Avoid using "charm" part name since that has special meaning to charmcraft
  charm-poetry:
    # By default, the `poetry` plugin creates/primes these directories:
    # - lib, src
    #   (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/parts/plugins/_poetry.py#L76-L78)
    # - venv
    #   (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/parts/plugins/_poetry.py#L95
    #   https://github.com/canonical/craft-parts/blob/afb0d652eb330b6aaad4f40fbd6e5357d358de47/craft_parts/plugins/base.py#L270)
    plugin: poetry
    source: .
    after:
      - poetry-deps
    poetry-export-extra-args: ['--only', 'main,charm-libs']
    build-snaps:
      - rustup
    build-packages:
      - libffi-dev
      - libssl-dev
      - pkg-config
    override-build: |
      rustup default stable
      craftctl default
      # Include requirements.txt in *.charm artifact for easier debugging
      cp requirements.txt "$CRAFT_PART_INSTALL/requirements.txt"
  # "files" part name is arbitrary; use for consistency
  files:
    plugin: dump
    source: .
    build-packages:
      - git
    override-build: |
      # Workaround to add unique identifier (git hash) to charm version while specification
      # DA053 - Charm versioning
      # (https://docs.google.com/document/d/1Jv1jhWLl8ejK3iJn7Q3VbCIM9GIhp8926bgXpdtx-Sg/edit?pli=1)
      # is pending review.
      python3 -c 'import pathlib; import shutil; import subprocess; git_hash=subprocess.run(["git", "describe", "--always", "--dirty"], capture_output=True, check=True, encoding="utf-8").stdout; file = pathlib.Path("charm_version"); shutil.copy(file, pathlib.Path("charm_version.backup")); version = file.read_text().strip(); file.write_text(f"{version}+{git_hash}")'

      craftctl default
    prime:
      - LICENSE
      - charm_version
      - workload_version
      - scripts
      - templates

Relevant log output

$ charmcraft pack -v --platform ubuntu@24.04:amd64
Starting charmcraft, version 3.2.2.post133+g455880a4
Logging execution to '/home/ubuntu/.local/state/charmcraft/log/charmcraft-20241220-105657.967672.log'
Launching managed ubuntu 24.04 instance...
Starting instance 
Starting charmcraft, version 3.2.2.post133+g455880a4
Logging execution to '/tmp/charmcraft.log'
Initialising lifecycle
Installing build-packages
Installing build-snaps
Skipping pull for poetry-deps (already ran)
Skipping pull for charm-poetry (already ran)
Skipping pull for files (already ran)
Building poetry-deps
:: + python3 -m pip install --upgrade pip==24.3.1
:: error: externally-managed-environment
::
:: × This environment is externally managed
:: ╰─> To install Python packages system-wide, try apt install
::     python3-xyz, where xyz is the package you are trying to
::     install.
::
::     If you wish to install a non-Debian-packaged Python package,
::     create a virtual environment using python3 -m venv path/to/venv.
::     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
::     sure you have python3-full installed.
::
::     If you wish to install a non-Debian packaged Python application,
::     it may be easiest to use pipx install xyz, which will manage a
::     virtual environment for you. Make sure you have pipx installed.
::
::     See /usr/share/doc/python3.12/README.venv for more information.
::
:: note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
:: hint: See PEP 668 for the detailed specification.
'override-build' in part 'poetry-deps' failed with code 1.
Recommended resolution: Review the scriptlet and make sure it's correct.
Failed to execute charmcraft in instance.
Full execution log: '/home/ubuntu/.local/state/charmcraft/log/charmcraft-20241220-105657.967672.log'

[carlcsaposs-canonical]

It appears that --break-system-packages doesn't work

$ charmcraft pack -v --platform ubuntu@24.04:amd64
Starting charmcraft, version 3.2.2.post133+g455880a4
Logging execution to '/home/ubuntu/.local/state/charmcraft/log/charmcraft-20241220-110811.708973.log'
Launching managed ubuntu 24.04 instance...
Starting instance 
Starting charmcraft, version 3.2.2.post133+g455880a4
Logging execution to '/tmp/charmcraft.log'
Initialising lifecycle
Installing build-packages
Installing build-snaps
Skipping pull for poetry-deps (already ran)
Skipping pull for charm-poetry (already ran)
Skipping pull for files (already ran)
Building poetry-deps
:: + python3 -m pip install --upgrade pip==24.3.1 --break-system-packages
:: Collecting pip==24.3.1
::   Using cached pip-24.3.1-py3-none-any.whl.metadata (3.7 kB)
:: Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
:: Installing collected packages: pip
::   Attempting uninstall: pip
::     Found existing installation: pip 24.0
:: ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian.
'override-build' in part 'poetry-deps' failed with code 1.
Recommended resolution: Review the scriptlet and make sure it's correct.
Failed to execute charmcraft in instance.
Full execution log: '/home/ubuntu/.local/state/charmcraft/log/charmcraft-20241220-110811.708973.log'

what is the recommended way to upgrade pip on a 24.04 base?

[carlcsaposs-canonical]

fyi, this is a significant issue for charms with a 22.04 and 24.04 base, because pip needs to be upgraded on 22.04 or the build fails

:: + /usr/bin/python3.10 -m pip --python=/root/parts/charm-poetry/install/venv/bin/python3 install --no-deps --no-binary=:all: --requirement=/root/parts/charm-poetry/build/requirements.txt
::
:: Usage:
::   /usr/bin/python3.10 -m pip <command> [options]
::
:: no such option: --python

[lengau]

Thanks for the report! It's probably worth making it clearer in the docs that you can't replace pip on noble and newer, but I don't think we can do anything about the underlying pip-on-noble issue.

I think the following workaround in your override-build script would work though:

    override-build: |
      . /etc/os-release
      if [[ "${VERSION_CODENAME}" == "jammy" ]]; then
        python3 -m pip install --upgrade pip==24.3.1  # renovate: charmcraft-pip-latest
      fi
      curl -sSL https://install.python-poetry.org | python3 - --version 1.8.4  # renovate: charmcraft-poetry-latest
      ln -sf "$HOME/.local/bin/poetry" /usr/local/bin/poetry

That doesn't solve it for a case where we would need a newer pip on a base >= noble, but I think in that case what we'd probably have to do is set up another virtual environment in a temporary directory somewhere and then override PARTS_PYTHON_INTERPRETER for the part that uses the poetry plugin. I can write up a demo if you're interested, but I'm going to caveat it that right now I'd recommend against that demo :-)

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3850.

This message was autogenerated

[carlcsaposs-canonical]

Upgrading pip with --user and --break-system-packages seems to work