%py-meson-python: respect jobserver

[haampie]

If Spack is running under a jobserver, packages should not pass -jN to ninja.

Spack's new installer creates a jobserver server by default, and ninja is a
client. If you pass it -jN, it ignores Spack's jobserver and spawns N
subprocesses, leading to oversubscription.

Without -jN, you get composable parallelism across multiple concurrent
package builds 🌈 🦄 get_effective_jobs runs None in case
of a jobserver.

@adamjstewart we could probably have better API for this. Is compile-args
standard? (Bit weird that it's a string, but well...)

[adamjstewart]

I don't think it's standard?

[haampie]

Convention in the sub-ecosystem? ;p

[rgommers]

compile-args is the standard key to put into a PEP 517 config_settings when using meson-python. It is not the same as for setuptools (which doesn't use Ninja) nor for scikit-build-core (docs FAQ):

# package uses meson-python:
pip install -Ccompile-args=-jN
# package uses scikit-build-core:
pip install -Ccmake.define.CMAKE_BUILD_PARALLEL_LEVEL=N

[rgommers]

The change looks reasonable to me. I don't even remember why there's Ccompile-args=-jN for NumPy and Matplotlib. For SciPy I think we had some issues with the Ninja default of 2*N + 2 processes due to memory usage, and wanted to change it to N. For Matplotlib and NumPy, which have lighter compiles, it could possibly just be copy/paste?

[haampie]

Maybe I could put the logic in the PythonPackage's defaults config_settings, conditional on %meson-python. Is that OK @adamjstewart?

[adamjstewart]

Worth a shot, but we would still need to override at least these packages anyway.

[haampie]

With the current patch, installing node-js next to py-numpy with spack -c config:installer:new install -j8:

python3─┬─python3─┬─python3───python3───ninja───2*[sh───ccache───g++───cc1plus]
        │         └─{python3}
        └─python3─┬─make───make─┬─ccache───gcc───cc1
                  │             ├─4*[ccache───g++───cc1plus]
                  │             └─ccache
                  └─{python3}

top branch is py-numpy, bottom branch node-js, total jobs is indeed 8 (2*g++ + make + 4*g++ + ccache)

Edit: seems to work well for all 3 packages.

[haampie]

I looked at scikit, which has different logic and some env variable, but that's a setuptools remnant (old versions). Also there it works fine. With the new installer

ninja: Jobserver mode detected: -j16 --jobserver-auth=fifo:/tmp/tmpneleew3q/jobserver_fifo

and with the old one it still passes -j16 explicitly.

The rest doesn't have -j anywhere. I checked these:

>>> from spack.repo import PATH
>>> pkgs = [pkg for pkg in PATH.all_package_classes() if any("py-meson-python" in d for d in pkg.dependencies.values())]
>>> [pkg.name for pkg in pkgs]
py-contourpy
py-fluidfft
py-fluidfft-fftw
py-fluidfft-fftwmpi
py-fluidfft-mpi-with-fftw
py-fluidfft-p3dfft
py-fluidfft-pfft
py-fluidsim
py-matplotlib
py-numpy
py-pandas
py-pywavelets
py-scikit-image
py-scikit-learn
py-scipy
py-xtb

So after this PR those packages should all have composable parallelism.

[rgommers]

Passing -jN for all packages that use meson-python seems fine to me. Using N processes is almost as fast as 2N+2 usually, and way safer regarding not running out of memory.

[haampie]

Users can oversubscribe as they like with spack install -j N, the N is shared over concurrent builds with the new installer.