Recommended workflow for using fastjet in environments with FastJet CPython bindings built?

[matthewfeickert]

I have what is probably an obvious question: If you have a Docker image that has the C++ FastJet built from source in it that also has the original Python bindings built and installed inside of a Python virtual environment setup for the user (/usr/local/venv)

...

ARG FASTJET_VERSION=3.4.0
RUN mkdir /code && \
    cd /code && \
    wget http://fastjet.fr/repo/fastjet-${FASTJET_VERSION}.tar.gz && \
    tar xvfz fastjet-${FASTJET_VERSION}.tar.gz && \
    cd fastjet-${FASTJET_VERSION} && \
    ./configure --help && \
    export CXX=$(which g++) && \
    export PYTHON=$(command -v python) && \
    export PYTHON_CONFIG=$(find /usr/local/ -iname "python-config.py") && \
    ./configure \
      --prefix=/usr/local/venv \
      --enable-pyext=yes && \
    make -j$(nproc --ignore=1) && \
    make check && \
    make install && \
    rm -rf /code

...

then if we also want the user to be able to have access (this is more important) to the scikit-hep/fastjet library, then as both the original bindings and this library have the name fastjet

docker@230caef77418:~$ command -v python
/usr/local/venv/bin/python
docker@230caef77418:~$ python -c 'import fastjet; print(fastjet)'
<module 'fastjet' from '/usr/local/venv/lib/python3.9/site-packages/fastjet.py'>
docker@230caef77418:~$ python -m pip --quiet install --upgrade pip setuptools wheel
docker@230caef77418:~$ python -m pip --quiet install fastjet
docker@230caef77418:~$ python -c 'import fastjet; print(f"{fastjet.__version__=}"); print(fastjet)'
fastjet.__version__='3.4.0.1'
<module 'fastjet' from '/usr/local/venv/lib/python3.9/site-packages/fastjet/__init__.py'>
docker@230caef77418:~$ 

the pip installed fastjet library will be the one that gets picked up by Python. So is there any reason why I might not want to skip building the Python bindings from source and instead just build the C++ library only and then

python -m pip install fastjet

immediately afterwards?

Another way of asking this is maybe "Is there any reason why I would ever want to favor having access to the original Python bindings over the scikit-hep/fastjet Python library?"

[jpivarski]

Since the original Python bindings are included within this fastjet package, I think you'd never want the original package to overshadow this one. Since this package is the only one named fastjet in PyPI, there would only be a conflict if the original bindings were manually inserted into the Python path.

[matthewfeickert]

Thanks. This is what I thought but wanted verification. So people like me who are building Docker images with FastJet should do the following:

...

ARG FASTJET_VERSION=3.4.0
RUN mkdir /code && \
    cd /code && \
    wget http://fastjet.fr/repo/fastjet-${FASTJET_VERSION}.tar.gz && \
    tar xvfz fastjet-${FASTJET_VERSION}.tar.gz && \
    cd fastjet-${FASTJET_VERSION} && \
    ./configure --help && \
    export CXX=$(which g++) && \
    ./configure \
      --prefix=/usr/local/venv \
    make -j$(nproc --ignore=1) && \
    make check && \
    make install && \
    rm -rf /code && \
    python -m pip install --upgrade pip && \
    python -m pip install --upgrade 'fastjet~=3.4.0.0'

...

[matthewfeickert]

Okay, so the answer is as of v3.4.0.1 that if you ever want to get paths from fastjet-config then you do need to install FastJet from source and then install fastjet as a Python library as the paths values from fastjet's fastjet-config will be invalid for any user given those are being set at build time. c.f. #148 (comment)

But that's for fastjet-config which is separate from the Python-bindings, and so @jpivarski's answer to my question still addresses the question asked. Issue #148 is demonstrating a different problem about source file distribution and setting config paths at build vs. install time (which might be/probably is well out of project scope here).

[lgray]

@matthewfeickert are you building for arm and that why you can't just use wheels?

[matthewfeickert]

@lgray I am also building for arm64, but I also realize that I wasn't thinking / forgot until @jpivarski reminded me that

Since the original Python bindings are included within this fastjet package

that of course this means the full package is included as well (doh!). So I've been forgetting that the entirety of fastjet-core is distributed in the sdist and wheel

$ docker run --rm -ti python:3.10 /bin/bash
root@c46cefbe24f0:/# python -m venv venv && . venv/bin/activate
(venv) root@c46cefbe24f0:/# python -m pip --quiet install --upgrade pip
(venv) root@c46cefbe24f0:/# python -m pip --quiet install fastjet
(venv) root@c46cefbe24f0:/# echo $(find venv/ -type d -iname "_fastjet_core")
venv/lib/python3.10/site-packages/fastjet/_fastjet_core
(venv) root@c46cefbe24f0:/# ls $(find venv/ -type d -iname "_fastjet_core")
bin  include  lib  share
(venv) root@c46cefbe24f0:/#

So now when I'm later on in my Dockerfile doing

...

# Install PYTHIA
ARG PYTHIA_VERSION=8307
# PYTHON_VERSION already exists in the base image
RUN mkdir /code && \
    cd /code && \
    wget --quiet "https://pythia.org/download/pythia${PYTHIA_VERSION:0:2}/pythia${PYTHIA_VERSION}.tgz" && \
    tar xvfz pythia${PYTHIA_VERSION}.tgz && \
    cd pythia${PYTHIA_VERSION} && \
    ./configure --help && \
    export PYTHON_MINOR_VERSION=${PYTHON_VERSION::3} && \
    if [[ "${TARGETARCH}" == "amd64" ]]; then \
        export CXX_COMMON="-O2 -m64 -pedantic -W -Wall -Wshadow -fPIC -std=c++17"; \
    elif [[ "${TARGETARCH}" == "arm64" ]]; then \
        export CXX_COMMON="-O2 -pedantic -W -Wall -Wshadow -fPIC -std=c++17"; \
    else \
      echo "TARGETARCH ${TARGETARCH} not supported. Exiting now."; \
      exit -1; \
    fi && \
    ./configure \
      --prefix=/usr/local/venv \
      --arch=Linux \
      --cxx=g++ \
      --enable-64bit \
      --with-gzip \
      --with-hepmc2=/usr/local/venv \
      --with-lhapdf6=/usr/local/venv \
      --with-fastjet3=/usr/local/venv \
      --with-python-bin=/usr/local/venv/bin/ \
      --with-python-lib=/usr/local/venv/lib/python${PYTHON_MINOR_VERSION} \
      --with-python-include=/usr/local/include/python${PYTHON_MINOR_VERSION} \
      --cxx-common="${CXX_COMMON}" \
      --cxx-shared="-shared -std=c++17" && \
    make -j$(nproc --ignore=1) && \
    make install && \
    unset CXX_COMMON && \
    rm -rf /code

...

I don't need to be going through a C++ source install of FastJet at all. As instead of building and installing FastJet under /usr/local/venv

docker@230caef77418:~$ ls /usr/local/venv/include/
HepMC  LHAPDF  Pythia8	Pythia8Plugins	fastjet  siscone

and directing PYTHIA to it during the build

--with-fastjet3=/usr/local/venv

I can just rely on the full package that is shipped with fastjet in the sdist and wheel

docker@230caef77418:~$ ls $(find /usr/local/venv/ -type d -iname "_fastjet_core")
bin  include  lib  share

and use

--with-fastjet3=$(find /usr/local/venv/ -type d -iname "_fastjet_core")

[matthewfeickert]

Ah, no actually this doesn't work as fastjet gives paths for --cxxflags that are invalid

Example:

$ docker run --rm -ti python:3.10 /bin/bash
root@aa1cddbea31d:/# python -m venv venv && . venv/bin/activate
(venv) root@aa1cddbea31d:/# python -m pip --quiet install --upgrade pip
(venv) root@aa1cddbea31d:/# python -m pip --quiet install 'fastjet==3.4.0.1'
(venv) root@aa1cddbea31d:/# command -v fastjet-config
(venv) root@aa1cddbea31d:/# find /venv/ -type f -iname "fastjet-config"
/venv/lib/python3.10/site-packages/fastjet/_fastjet_core/bin/fastjet-config
(venv) root@aa1cddbea31d:/# $(find /venv/ -type f -iname "fastjet-config") --cxxflags --libs --plugins
-I/project/src/fastjet/_fastjet_core/include -Wl,-rpath,/project/src/fastjet/_fastjet_core/lib -L/project/src/fastjet/_fastjet_core/lib -lfastjettools -lfastjet -lgmp -L/project/CGAL-5.5.1/lib -Wl,-rpath,/project/CGAL-5.5.1/lib -lm -lfastjetplugins -lsiscone_spherical -lsiscone
(venv) root@aa1cddbea31d:/# ls /project/src/fastjet/_fastjet_core/include
ls: cannot access '/project/src/fastjet/_fastjet_core/include': No such file or directory
(venv) root@aa1cddbea31d:/# find /venv/ -type d -iname "_fastjet_core"
/venv/lib/python3.10/site-packages/fastjet/_fastjet_core
(venv) root@aa1cddbea31d:/# 

where if you build the FastJet bindings from source you get valid paths.

$ docker run --rm -ti matthewfeickert/pythia-python:latest
root@794496c739a7:~/data# fastjet-config --cxxflags --libs --plugins
-I/usr/local/venv/include -Wl,-rpath,/usr/local/venv/lib -L/usr/local/venv/lib -lfastjettools -lfastjet -lm -lfastjetplugins -lsiscone_spherical -lsiscone
root@794496c739a7:~/data# ls /usr/local/venv/include
HepMC  LHAPDF  Pythia8	Pythia8Plugins	fastjet  siscone
root@794496c739a7:~/data#