Tag :latest is broken when pip3 install with the last build

[BlueTogepi]

Hi
With the last build, the command pip3 install xxx can't seem to install system-wide library, I'm not sure if this is intended or not.

Here's my cloudbuild.yaml:

steps:
  - id: 'run-python-gcloud'
    name: gcr.io/google.com/cloudsdktool/google-cloud-cli:latest
    entrypoint: bash
    args:
      - -c
      - |
        pip3 install google-auth
        python3 example.py
    waitFor: ['-']

And here's my example.py python script:

import google.auth

which will throw out ModuleNotFoundError: No module named 'google' error.

Here's the full cloudbuild error log:

error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Traceback (most recent call last):
File "/workspace/example.py", line 1, in <module>
import google.auth
ModuleNotFoundError: No module named 'google'

I know that either apt install-ing or using an environment manager as suggested would work, but I don't believe this breaking change is intentional.

Also, is the gcp document updated recently? I've noticed it is recommended to migrate to :stable, and if I'm to migrate as suggested, are all python packages from public pypi accessible from COMPONENTS env variables?

[anindyatahsin]

@BlueTogepi This is happening because from version 496.0.0 we have started building the latest docker image with debian-12. The latest debian image (debian-12) comes with python 3.11 instead of python 3.9 (which was default for debian-11).

As you mentioned you can use apt-get to install the google-auth with the following command:
apt-get install -y python3-google-auth

About out documentation update, yes we recommend migrating to the stable image as it is the most secure gcloud cli docker image. Also it is significantly smaller than the latest image (~560 MB compared to 3.15 GB for latest).

The COMPONENT environment variable is actually used for installing any gcloud components, however, we have the APT_PACKAGES env variable for installing any apt packages. And yes all packages from public pypi should be accessible using the APT_PACKAGES env variable (please let us know if any package is not accessible).

[BlueTogepi]

@anindyatahsin Thanks for clearing this up!

Currently, I'm trying to migrate to :stable but I'm struggling with using python3 as the cloudbuild step's entrypoint. I'm not sure why it's always saying 'python3': No such file or directory even though python3 should be the default system python in debian-12.

Here's my current cloudbuild.yaml:

steps:
  - id: 'python-exec'
    name: gcr.io/google.com/cloudsdktool/google-cloud-cli:497.0.0-stable
    entrypoint: bash
    args:
      - -c
      - >-
        ./example.py
    env:
      - 'APT_PACKAGES="python3-google-auth python3-requests"'

and here's my current example.py (I've already chmod +x it before commiting to my repo):

#!/usr/bin/env python3
import google.auth
import requests

I've already tried using

• entrypoint: python3 with args: ./example.py
• entrypoint: bash with args: -c >- python3 ./example.py

and it was always the same 'python3': No such file or directory.

I'm not sure if it is either from the APT_PACKAGES="python3-google-auth python3-requests" env not working or cloudbuild trying to run entrypoint & args before the image could run libraries installation from env (so there is no python3?).

Please note that the cloudbuild log does not show anything about installing apt-get packages; whereas when I tried running with

docker run --rm -it -e APT_PACKAGES="python3-google-auth python3-requests" gcr.io/google.com/cloudsdktool/google-cloud-cli:497.0.0-stable python3

on my local computer, it does show a lot of debian libs installation and was working as expected.

Also, would you mind adding examples for using this image inside a cloudbuild step in the documentation? There are only docker run examples, and somehow it does not work the same with cloudbuild.yaml in this case.