SNOW-874654: Not able to build dockerfile if I included snowflake-connector-python

[Ramlksasi]

Python version

python3.8

Operating system and processor architecture

Docker base image : python:3.8-alpine3.14

Installed packages

pip install --upgrade pip
pip install numpy==1.21 cython==0.29.23
pip install snowflake-connector-python

What did you do?

Example dockerfile used to reproduce this issue:
FROM python:3.8-alpine3.14

RUN apk update && apk add --no-cache \
    build-base \
    libffi-dev \
    openssl-dev \
    libc-dev \
    linux-headers \
    postgresql-dev

RUN pip install --upgrade pip

RUN pip install numpy==1.21 cython==0.29.23

RUN pip install snowflake-connector-python


Logs:
[+] Building 14.8s (8/8) FINISHED                                                                                                       docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                         0.0s
 => [internal] load build definition from dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 422B                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3.8-alpine3.14                                                                                1.3s
 => [1/5] FROM docker.io/library/python:3.8-alpine3.14@sha256:9688d36309a9d72d346ba5d1e9af94cd129fa9c6a985e6dfdd089536116b2db1                          0.0s
 => CACHED [2/5] RUN apk update && apk add --no-cache     build-base     libffi-dev     openssl-dev     libc-dev     linux-headers     postgresql-dev   0.0s
 => CACHED [3/5] RUN pip install --upgrade pip                                                                                                          0.0s
 => CACHED [4/5] RUN pip install numpy==1.21 cython==0.29.23                                                                                            0.0s
 => ERROR [5/5] RUN pip install snowflake-connector-python                                                                                             13.5s
------                                                                                                                                                       
 > [5/5] RUN pip install snowflake-connector-python:                                                                                                         
1.008 Collecting snowflake-connector-python                                                                                                                  
1.590   Downloading snowflake-connector-python-3.0.4.tar.gz (405 kB)                                                                                         
1.956      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 405.8/405.8 kB 1.2 MB/s eta 0:00:00                                                                       
2.023   Installing build dependencies: started
13.42   Installing build dependencies: finished with status 'error'
13.43   error: subprocess-exited-with-error
13.43   
13.43   × pip subprocess to install build dependencies did not run successfully.
13.43   │ exit code: 1
13.43   ╰─> [122 lines of output]
13.43       Collecting setuptools>=40.6.0
13.43         Obtaining dependency information for setuptools>=40.6.0 from https://files.pythonhosted.org/packages/c7/42/be1c7bbdd83e1bfb160c94b9cafd8e25efc7400346cf7ccdbdb452c467fa/setuptools-68.0.0-py3-none-any.whl.metadata
13.43         Downloading setuptools-68.0.0-py3-none-any.whl.metadata (6.4 kB)
13.43       Collecting wheel
13.43         Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/17/11/f139e25018ea2218aeedbedcf85cd0dd8abeed29a38ac1fda7f5a8889382/wheel-0.41.0-py3-none-any.whl.metadata
13.43         Downloading wheel-0.41.0-py3-none-any.whl.metadata (2.2 kB)
13.43       Collecting cython
13.43         Obtaining dependency information for cython from https://files.pythonhosted.org/packages/a5/71/46f6e3c0ccc55216cf1830b7cc69f0f21a30cb08e6488a191bd293b95ecb/Cython-3.0.0-cp38-cp38-musllinux_1_1_aarch64.whl.metadata
13.43         Downloading Cython-3.0.0-cp38-cp38-musllinux_1_1_aarch64.whl.metadata (3.1 kB)
13.43       Collecting pyarrow<10.1.0,>=10.0.1
13.43         Downloading pyarrow-10.0.1.tar.gz (994 kB)
13.43            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 994.1/994.1 kB 2.7 MB/s eta 0:00:00
13.43         Installing build dependencies: started
13.43         Installing build dependencies: finished with status 'error'
13.43         error: subprocess-exited-with-error
13.43       
13.43         × pip subprocess to install build dependencies did not run successfully.
13.43         │ exit code: 1
13.43         ╰─> [93 lines of output]
13.43             Collecting cython>=0.29.22
13.43               Obtaining dependency information for cython>=0.29.22 from https://files.pythonhosted.org/packages/a5/71/46f6e3c0ccc55216cf1830b7cc69f0f21a30cb08e6488a191bd293b95ecb/Cython-3.0.0-cp38-cp38-musllinux_1_1_aarch64.whl.metadata
13.43               Using cached Cython-3.0.0-cp38-cp38-musllinux_1_1_aarch64.whl.metadata (3.1 kB)
13.43             Collecting oldest-supported-numpy>=0.14
13.43               Downloading oldest_supported_numpy-2022.11.19-py3-none-any.whl (4.9 kB)
13.43             Collecting setuptools_scm
13.43               Downloading setuptools_scm-7.1.0-py3-none-any.whl (43 kB)
13.43                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 kB 449.9 kB/s eta 0:00:00
13.43             Collecting setuptools>=40.1.0
13.43               Obtaining dependency information for setuptools>=40.1.0 from https://files.pythonhosted.org/packages/c7/42/be1c7bbdd83e1bfb160c94b9cafd8e25efc7400346cf7ccdbdb452c467fa/setuptools-68.0.0-py3-none-any.whl.metadata
13.43               Using cached setuptools-68.0.0-py3-none-any.whl.metadata (6.4 kB)
13.43             Collecting wheel
13.43               Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/17/11/f139e25018ea2218aeedbedcf85cd0dd8abeed29a38ac1fda7f5a8889382/wheel-0.41.0-py3-none-any.whl.metadata
13.43               Using cached wheel-0.41.0-py3-none-any.whl.metadata (2.2 kB)
13.43             Collecting numpy==1.19.2 (from oldest-supported-numpy>=0.14)
13.43               Downloading numpy-1.19.2.zip (7.3 MB)
13.43                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 3.5 MB/s eta 0:00:00
13.43               Installing build dependencies: started
13.43               Installing build dependencies: finished with status 'done'
13.43               Getting requirements to build wheel: started
13.43               Getting requirements to build wheel: finished with status 'done'
13.43               Preparing metadata (pyproject.toml): started
13.43               Preparing metadata (pyproject.toml): finished with status 'error'
13.43               error: subprocess-exited-with-error
13.43       
13.43               × Preparing metadata (pyproject.toml) did not run successfully.
13.43               │ exit code: 1
13.43               ╰─> [55 lines of output]
13.43                   warning: _philox.pyx:19:0: The 'DEF' statement is deprecated and will be removed in a future Cython version. Consider using global variables, constants, and in-place literals instead. See https://github.com/cython/cython/issues/4310
13.43       
13.43                   Error compiling Cython file:
13.43                   ------------------------------------------------------------
13.43                   ...
13.43                               self.rng_state.ctr.v[i] = counter[i]
13.43       
13.43                           self._reset_state_variables()
13.43       
13.43                           self._bitgen.state = <void *>&self.rng_state
13.43                           self._bitgen.next_uint64 = &philox_uint64
13.43                                                      ^
13.43                   ------------------------------------------------------------
13.43       
13.43                   _philox.pyx:195:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'
13.43                   Processing numpy/random/_bounded_integers.pxd.in
13.43                   Processing numpy/random/_philox.pyx
13.43                   Traceback (most recent call last):
13.43                     File "/tmp/pip-install-jsynbsqu/numpy_0a7931fd35644a3991edb0a6ce351d0c/tools/cythonize.py", line 235, in <module>
13.43                       main()
13.43                     File "/tmp/pip-install-jsynbsqu/numpy_0a7931fd35644a3991edb0a6ce351d0c/tools/cythonize.py", line 231, in main
13.43                       find_process_files(root_dir)
13.43                     File "/tmp/pip-install-jsynbsqu/numpy_0a7931fd35644a3991edb0a6ce351d0c/tools/cythonize.py", line 222, in find_process_files
13.43                       process(root_dir, fromfile, tofile, function, hash_db)
13.43                     File "/tmp/pip-install-jsynbsqu/numpy_0a7931fd35644a3991edb0a6ce351d0c/tools/cythonize.py", line 188, in process
13.43                       processor_function(fromfile, tofile)
13.43                     File "/tmp/pip-install-jsynbsqu/numpy_0a7931fd35644a3991edb0a6ce351d0c/tools/cythonize.py", line 77, in process_pyx
13.43                       subprocess.check_call(
13.43                     File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
13.43                       raise CalledProcessError(retcode, cmd)
13.43                   subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'cython', '-3', '--fast-fail', '-o', '_philox.c', '_philox.pyx']' returned non-zero exit status 1.
13.43                   Running from numpy source directory.
13.43                   setup.py:470: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
13.43                     run_build = parse_setuppy_commands()
13.43                   Cythonizing sources
13.43                   Traceback (most recent call last):
13.43                     File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
13.43                       main()
13.43                     File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
13.43                       json_out['return_val'] = hook(**hook_input['kwargs'])
13.43                     File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
13.43                       return hook(metadata_directory, config_settings)
13.43                     File "/tmp/pip-build-env-zgub1b2m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
13.43                       self.run_setup()
13.43                     File "/tmp/pip-build-env-zgub1b2m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 248, in run_setup
13.43                       super(_BuildMetaLegacyBackend,
13.43                     File "/tmp/pip-build-env-zgub1b2m/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 142, in run_setup
13.43                       exec(compile(code, __file__, 'exec'), locals())
13.43                     File "setup.py", line 499, in <module>
13.43                       setup_package()
13.43                     File "setup.py", line 479, in setup_package
13.43                       generate_cython()
13.43                     File "setup.py", line 274, in generate_cython
13.43                       raise RuntimeError("Running cythonize failed!")
13.43                   RuntimeError: Running cythonize failed!
13.43                   [end of output]
13.43       
13.43               note: This error originates from a subprocess, and is likely not a problem with pip.
13.43             error: metadata-generation-failed
13.43       
13.43             × Encountered error while generating package metadata.
13.43             ╰─> See above for output.
13.43       
13.43             note: This is an issue with the package mentioned above, not pip.
13.43             hint: See above for details.
13.43             [end of output]
13.43       
13.43         note: This error originates from a subprocess, and is likely not a problem with pip.
13.43       error: subprocess-exited-with-error
13.43       
13.43       × pip subprocess to install build dependencies did not run successfully.
13.43       │ exit code: 1
13.43       ╰─> See above for output.
13.43       
13.43       note: This error originates from a subprocess, and is likely not a problem with pip.
13.43       [end of output]
13.43   
13.43   note: This error originates from a subprocess, and is likely not a problem with pip.
13.43 error: subprocess-exited-with-error
13.43 
13.43 × pip subprocess to install build dependencies did not run successfully.
13.43 │ exit code: 1
13.43 ╰─> See above for output.
13.43 
13.43 note: This error originates from a subprocess, and is likely not a problem with pip.
------
dockerfile:15
--------------------
  13 |     RUN pip install numpy==1.21 cython==0.29.23
  14 |     
  15 | >>> RUN pip install snowflake-connector-python
  16 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install snowflake-connector-python" did not complete successfully: exit code: 1

What did you expect to see?

I need a successful docker build image to be hosted to AWS lambda.

Can you set logging to DEBUG and collect the logs?

import logging
import os

for logger_name in ('snowflake.connector',):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

[Ramlksasi]

Please let me know if there is any way to resolve this issue.

[zhanaral1]

@Ramlksasi have you resolved this problem?

[Ramlksasi]

No, not yet.

[williamli2021]

If it helps, I use the following approach to run this in container. It runs fine:

RUN apt-get update && apt-get install --no-install-recommends -y python3.9 python3.9-dev python3.9-venv python3-pip python3-wheel build-essential unixodbc-dev cron libpq5 git && \
	apt-get clean && rm -rf /var/lib/apt/lists/* 

RUN pip3 install --no-cache-dir -r requirements.txt

And in the requirements.txt, it has this:

snowflake-connector-python==3.1.0

[sfc-gh-yixie]

Thanks @williamli2021

@Ramlksasi Does it help you?
This error happened when it's compiling numpy's _philox.pyx instead of snowflake-connector-python.
I'm curious why numpy 1.19.2 was downloaded to build.

[sfc-gh-yixie]

I'm trying in my environment. Will update you.

[Ramlksasi]

Numpy is dowloading as a build dependency of snowflake connector. I will test this let here know. Thanks everyone for helping.

[sfc-gh-yixie]

Hi @Ramlksasi

I tried in my local environment. On Alpine Linux image, pip install downloads the source code to compile.
pip install snowfalke-connector-python will create an isolated environment to compile snowfalke-connector-python, which requires pyarrow in the same isolated env to compile. pyarrow then requires numpy. Your error happened when numpy was being built in the isolated env. It's not the numpy you installed previously. Actually even if you pass compiling numpy you will hit an error with compiling `pyarrow.

I tried two other ways aside from @williamli2021's workaround:

• option 1: Use --no-build-isolation when pip install snowflake-connector-python. Then you need to make sure pyarrow and numpy have been successfully installed in the same environment before connector. It seems that installing pyarrow on the alpine linux image isn't trivial either.
• option 2: Use pip install snowflake-connector-python==3.1.0a2. This alpha version no longer needs pyarrow and numpy. Instead it uses nanoarrow. We'll GA this feature in the near future. But if you need to use the pandas related APIs like to_pandas, you still need to install pandas and pyarrow.

  FROM python:3.8-alpine3.14
  
  RUN apk update && apk add --no-cache \
      build-base \
      libffi-dev \
      openssl-dev \
      libc-dev \
      linux-headers \
      postgresql-dev
  
  RUN pip install --upgrade pip
  
  RUN pip install snowflake-connector-python==3.1.0a2

[Ramlksasi]

Thank you very much for writing.