vcpkg doesn't provide python numpy component

[thomasnal]

Describe the bug
vcpkg cannot find python that vcpkg installed.

Environment

• OS: Windows
• Compiler: Visual Studio 16 2019
• CMake: 3.25
• Azure build agent

To Reproduce

1. cmake -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake ` ..
2. Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy)

vcpkg.json

{
  "name": "stopt",
  "version": "5.0",
  "dependencies": [
    {
      "name": "python3",
      "version>=": "3.8.3#2"
    },
    "pybind11",
    "boost", 
    "boost-mpi",
    "osi",
    "clp",
    "coinutils",
    "eigen3"
  ],
  "builtin-baseline": "69478c5caafcde4c490bb1fccb960296801dbb5f",

  "overrides": [
    { "name": "python3", "version": "3.8.3#2" }
  ]
}

CMakeLists.txt

...
  FIND_PACKAGE(Python REQUIRED COMPONENTS Interpreter Development NumPy)
...

Expected behavior
cmake finds the python installed by vcpkg.

Failure logs

cmake -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake ` ..
...
    python3[core]:x64-windows -> 3.8.3#2 -- C:\vcpkg\buildtrees\versioning_\versions\python3\a27ac0326c21362d6a324236c772161ac6130ef5
...
Installing 98/163 python3:x64-windows...
Building python3[core]:x64-windows...
-- Installing port from location: C:\vcpkg\buildtrees\versioning_\versions\python3\a27ac0326c21362d6a324236c772161ac6130ef5
-- Downloading https://github.com/python/cpython/archive/v3.8.3.tar.gz -> python-cpython-v3.8.3.tar.gz...
-- Extracting source C:/vcpkg/downloads/python-cpython-v3.8.3.tar.gz
-- Using source at C:/vcpkg/buildtrees/python3/src/v3.8.3-43b03f2bf0.clean
-- Building C:/vcpkg/buildtrees/python3/src/v3.8.3-43b03f2bf0.clean-Lib-Win/PCBuild/pythoncore.vcxproj for Release
-- Building C:/vcpkg/buildtrees/python3/src/v3.8.3-43b03f2bf0.clean-Lib-Win/PCBuild/pythoncore.vcxproj for Debug
-- Up-to-date: C:/vcpkg/packages/python3_x64-windows/include/python3.8
-- Installing: C:/vcpkg/packages/python3_x64-windows/include/python3.8/abstract.h
...
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "C:\Users\VssAdministrator\AppData\Local\vcpkg\archives\5f\5fcb4db6cb8c6ae049ca834ce75baa49163f3d56638ca5f3f4ca61fac3a8bf99.zip"
Elapsed time to handle python3:x64-windows: 3.186 min
...
CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy) (found
  version "2.7.18")
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.25/Modules/FindPython.cmake:561 (find_package_handle_standard_args)
  C:/vcpkg/scripts/buildsystems/vcpkg.cmake:829 (_find_package)
  CMakeLists.txt:216 (FIND_PACKAGE)
-- Configuring incomplete, errors occurred!
See also "D:/a/1/s/build/CMakeFiles/CMakeOutput.log".
See also "D:/a/1/s/build/CMakeFiles/CMakeError.log".

Additional context
Alternatives tried
Set PATH to system installation of Python,

PATH=/c/Users/VssAdministrator/AppData/Roaming/Python/Python38/Scripts:/c/hostedtoolcache/windows/Python/3.8.10/x64/Scripts:/c/hostedtoolcache/windows/Python/3.8.10/x64

Failed in the same way.

[thomasnal]

Setting Python3_ROOT_DIR according to CMake docs has no effect,

          SET Python3_ROOT_DIR=C:/hostedtoolcache/windows/Python/3.8.3/x64
          cmake -DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake `
                -DPython3_ROOT_DIR=C:/hostedtoolcache/windows/Python/3.8.3/x64 `
                ..
...
CMake Error at C:/Program Files/CMake/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python (missing: Python_NumPy_INCLUDE_DIRS NumPy)

Content of C:/hostedtoolcache/windows/Python/3.8.3/x64,

...
C:/hostedtoolcache/windows/Python/3.8.3/x64/libs/python3.lib
C:/hostedtoolcache/windows/Python/3.8.3/x64/libs/python38.lib
C:/hostedtoolcache/windows/Python/3.8.3/x64/libs/_tkinter.lib
C:/hostedtoolcache/windows/Python/3.8.3/x64/LICENSE.txt
C:/hostedtoolcache/windows/Python/3.8.3/x64/NEWS.txt
C:/hostedtoolcache/windows/Python/3.8.3/x64/python-3.8.3-amd64.exe
C:/hostedtoolcache/windows/Python/3.8.3/x64/python.exe
C:/hostedtoolcache/windows/Python/3.8.3/x64/python3.dll
C:/hostedtoolcache/windows/Python/3.8.3/x64/python3.exe
C:/hostedtoolcache/windows/Python/3.8.3/x64/python38.dll
C:/hostedtoolcache/windows/Python/3.8.3/x64/pythonw.exe
...

More context before calling cmake,

>>> python3 -V
Python 3.8.3

>>> pip3 list
Package    Version
---------- -------
numpy      1.23.5
pip        22.3.1
setuptools 41.2.0

[dg0yt]

I guess vcpkg can find python that python installs. It just doesn't provide the numpy component you ask for.

[huhuang03]

Any workaround on this? Manually install numpy and manually find and link it?

[huhuang03]

cmake seems use this code to find numpy

    execute_process(COMMAND ${${_PYTHON_PREFIX}_INTERPRETER_LAUNCHER} "${_${_PYTHON_PREFIX}_EXECUTABLE}" -c
                            "import sys\ntry: import numpy; sys.stdout.write(numpy.get_include())\nexcept:pass\n"
                    RESULT_VARIABLE _${_PYTHON_PREFIX}_RESULT
                    OUTPUT_VARIABLE _${_PYTHON_PREFIX}_NumPy_PATH
                    ERROR_QUIET
                    OUTPUT_STRIP_TRAILING_WHITESPACE)

So first install pip for vcpkg python

path_to_vcpkg/installed/tools/python3/python -m ensurepip --upgrade

Then use this pip to install numpy

path_to_vcpkg/installed/tools/python3/Scripts/pip install numpy

This should work.

[huhuang03]

Use system python is the prefer way!

[bennahugo]

Not familiar with vcpkg or the Windows directory structures, but Just making a note that once installed it should be reasonably simple to detect where the library file is (it should be in the site-packages right next to the library). On a Debian install it is

In [1]: import numpy as n; print(n.__version__); print(n.get_include())
1.13.3
/usr/lib/python3/dist-packages/numpy/core/include

Cmake can be made to rely on this. E.g. https://github.com/casacore/casacore/blob/master/cmake/FindNUMPY.cmake#LL54C1-L54C67

[dg0yt]

IIUC asking python for numpy is what CMake find module does:
https://github.com/Kitware/CMake/blob/ae21371e701ecfb88ec34306b50ebdfe00b26578/Modules/FindPython/Support.cmake#L3742-L3748

Does it need an extra action to have an external python actually provide the include files - maybe not just install runtime packages but also development packages?

[Roy-Kid]

Hi, I have another similar problem. I have both vcpkg and conda, but CMake always use vcpkg's python, which can not find numpy in conda environment. I try to use Python3_ROOT_DIR to specify it manually but not work. Do you have any idea with it?