feat: add pyproject.toml and cibuildwheel support to pythonbindings

[bellaz89]

What

This PR add a pip-compliant pyproject.toml project file. In addition the configuration for cibuildwheel is added (https://github.com/pypa/cibuildwheel). scikit-build is used as the pyproject.toml build backend. The normal way of building the package should be unaffected.

Why

This PR enables the deviceaccess-pythonbinding module to be installed via pip. Therefore, the integration of this module with a typical python development workflow is greatly improved. In addition, when built via cibuildwheel, distribution-independent portable .wheel packages are generated for CPython v3.8 to v3.13. The cibuildwheel configuration automatically fetch/compile/install the package dependencies inside a container thus simplifying the generation of the python package by third-parties.

How

pip

This requires the correct versions of the DeviceAccess-PythonBindings dependencies to be installed system-wide.

python -m pip install .

cibuildwheel

This requires cibuildwheel and docker/podman. Additionally, if cross compiled, binfmt/qemu support should be enabled

python -m pip install cibuildwheel
# export CIBW_CONTAINER_ENGINE=podman # only if podman is used
cibuildwheel

This generates the .wheel packages in ./wheelhouse at the end of the process

[mhier]

I fear this comes with a lot of caveats:

1. Building DeviceAccess etc. internally is fine as long as only the backends shipped with the DeviceAccess core library shall be used. In reality, we often have other backends like the DoocsBackend, which might get installed as a system package (e.g. Debian package) but will then be potentially binary incompatible with the DeviceAccess build used by the PythonBindings. This might lead to all kind of funny behaviour.
2. I am not really aware of the consequences of the buildwheel support. In my understanding, those wheels are meant as a way to distribute the binary build, so users don't have to compile everything from source. To make this work on many Linux systems, the "manylinux" definition uses rather old ABIs as a baseline. We require for DeviceAccess at least gcc-13 including its libstdc++. Also we need a fairly new BOOST library. Is all of that really included in the "manylinux_2_28" definition?

Maybe we can address the first point by somehow disabling the backend loading mechanism (let it print some error message explaining why it was disabled). This requires a change in DeviceAccess though.

Regarding the second point: Can we maybe just not support buildwheels and always force people to compile from source?

Also: you are installing system packages via dnf, which is Redhat specific. That has to be changed somehow. Is there an official way to require system dependencies in a pip build?

[mhier]

Ok, the dnf might be fine, since it will be running inside the buildwheel docker container, right?

Still, I see problems with binary compatibility, because BOOST will not necessarily be compatible with the system BOOST library and other python modules might bring in other BOOST versions. The same is true for libxml++, although less likely.

Also we think that the pure pip installation is rather useless, since you expect all dependencies to be present in the system already. If you can satisfy them, you could also install the Python bindings in the same way. Isn't it possible somehow to build the dependencies also when running python -m pip install .? You have to build the Python Bindings itself then, so building the dependencies should be possible in the same way?

[bellaz89]

Hi, thanks for looking into this

Building DeviceAccess etc. internally is fine as long as only the backends shipped with the DeviceAccess core library shall be used. In reality, we often have other backends like the DoocsBackend, which might get installed as a system package (e.g. Debian package) but will then be potentially binary incompatible with the DeviceAccess build used by the PythonBindings. This might lead to all kind of funny behaviour.

If needed, in the past I did some experiments on this. I wrote a script that builds all the plugins and patched the CMake file to force linking them to the deviceaccess-pythonbindings library so dload is automatically triggered at runtime. With this solution, all the plugins are loaded when deviceaccess is loaded by CPython (and therefore no additional manual loading is required ;) )

See: https://github.com/bellaz89/DeviceAccess-PythonBindings/blob/battery_included/CMakeLists.txt#L58-L63

Apparently, cibuildwheel also fixes these dependencies by pulling the libraries/dependencies required by the Doocs,EPICS,OPC-UA and modbus backends (Though, the amount of testing was limited).

Even if it is an inelegant and compilation-time consuming solution, this effectively allows delivering all the deviceaccess plugins/backends in one single package.

Ok, the dnf might be fine, since it will be running inside the buildwheel docker container, right?

Yes exactly. All the build process is containerized to enforce reproducibility. Even if it the container uses dnf the resulting .wheel is distribuition independent. Note that this is the 'official' way to build binary dependencies since, as far as I understand, this is the method enforced by the [PYthon Package Authority].(https://www.pypa.io/en/latest/)

Still, I see problems with binary compatibility, because BOOST will not necessarily be compatible with the system BOOST library > and other python modules might bring in other BOOST versions. The same is true for libxml++, although less likely.

Is this a problem though? If the module is linked to its private version of BOOST (and libxml++) embedded in the .wheel this won't be visible to other parts of the program no? If another part of the program uses another version of BOOST, both versions will be loaded.

The only problem is if one wants to load deviceaccess backends that were build with different versions of the dependencies. For them, the only "easy" solution I see is the one discussed above.

EDIT: I see that it is not easy as it seems, to be checked.

Also we think that the pure pip installation is rather useless, since you expect all dependencies to be present in the system ? already. If you can satisfy them, you could also install the Python bindings in the same way.

Yes. 'pure' pip installation (without cibuildwheel) is rather useless. It is there just because cibuildwheel needs it. It first compiles the dependencies and then it triggers the wheel building process with pip and different versions of CPython.

Anyways, a PEP compliant pyproject.toml is also required to generate the .wheel metadata used by pip. So, if not used, such metadata should be provided by other means.

Isn't it possible somehow to build the dependencies also when running python -m pip install .? You have to build the Python Bindings itself then, so building the dependencies should be possible in the same way?

I am not expert about that, but this should be possible through CMake (see https://cmake.org/cmake/help/latest/module/FetchContent.html)

[bellaz89]

Regarding the ABI version: I picked manylinux_2_28 because it was sufficiently new to build the package. Tough, if needed, more recent versions are available. See: https://github.com/pypa/manylinux

E.g. the latest image, manylinux_2_39, should be Almalinux 10 / GLIBC 2.39

[mhier]

To simplify the discussion I will concentrate on a single point for now:

If another part of the program uses another version of BOOST, both versions will be loaded.

This is not how the dynamic linker in Linux works. If a shared library depends on another shared library, it will first check whether a shared library of this name has been loaded already. If this is the case, no attempt will be made to load the dependency. There is no version checking in this.

Even if you somehow manage to load that other dependency version, only those symbols will be imported which are not already present. You cannot have the same symbol loaded twice in the process. So even linking in all our dependencies statically would not help.

I had done some extensive research in this direction a couple of years ago and discussed this on stackoverflow: https://stackoverflow.com/questions/55008283/dlmopen-and-c-libraries

In theory we could check if there has been any improvement in this direction, but using dlmopen in this context would be a huge hassle which IMO isn't worth it.

I think, if anything like this is done, we should aim for an official publication as a PyPI package, otherwise the benefit is too small. That sets a rather high requirement for portability (i.e. we really have to comply to manylinux) and interoperability with other packages.

PS: If you are simply concerned how to build the Python Bindings and its dependencies from source easily, I recommend dragon (https://gitlab.desy.de/msk-sw/utilities/workflow-scripts - might need modification for you to disable the Gitlab part, since you are only interested in ChimeraTK on Github).

PPS: We were not able to build the package as you documented it, so it seems not even to work for our main platform Ubuntu 24.04...

[bellaz89]

PPS: We were not able to build the package as you documented it, so it seems not even to work for our main platform Ubuntu 24.04...

If you are still interested in trying it. Could you please post a log? Thanks

[bellaz89]

So, I tried to read the SONAME of the embedded libraries by unzipping the generated .wheel:

for i in * ; do readelf -d $i | grep SONAME; done
 0x000000000000000e (SONAME)             Library soname: [libboost_atomic-00e6ede8.so.1.78.0]
 0x000000000000000e (SONAME)             Library soname: [libboost_chrono-5bc8051c.so.1.78.0]
 0x000000000000000e (SONAME)             Library soname: [libboost_filesystem-e9e1d9ac.so.1.78.0]
 0x000000000000000e (SONAME)             Library soname: [libboost_system-c0fa27da.so.1.78.0]
 0x000000000000000e (SONAME)             Library soname: [libboost_thread-f12e0bef.so.1.78.0]
 0x000000000000000e (SONAME)             Library soname: [libChimeraTK-DeviceAccess-bc5f9758.so.03.24.01]
 0x000000000000000e (SONAME)             Library soname: [libffi-3a37023a.so.6.0.2]
 0x000000000000000e (SONAME)             Library soname: [libglibmm-2-85c25c82.4.so.1.3.0]
 0x000000000000000e (SONAME)             Library soname: [libgmodule-2-6c5d2a3f.0.so.0.5600.4]
 0x000000000000000e (SONAME)             Library soname: [libgmp-d944b113.so.10.3.2]
 0x000000000000000e (SONAME)             Library soname: [libgnutls-e5fc1c5f.so.30.28.2]
 0x000000000000000e (SONAME)             Library soname: [libhogweed-cd4c53be.so.4.5]
 0x000000000000000e (SONAME)             Library soname: [libidn2-2f4a5893.so.0.3.6]
 0x000000000000000e (SONAME)             Library soname: [liblzma-51a76f52.so.5.2.4]
 0x000000000000000e (SONAME)             Library soname: [libnettle-37944285.so.6.5]
 0x000000000000000e (SONAME)             Library soname: [libp11-kit-ac9dcd7e.so.0.3.0]
 0x000000000000000e (SONAME)             Library soname: [libpcre-0dd207b5.so.1.2.10]
 0x000000000000000e (SONAME)             Library soname: [libsigc-2-86d3a688.0.so.0.0.0]
 0x000000000000000e (SONAME)             Library soname: [libtasn1-564de53e.so.6.5.5]
 0x000000000000000e (SONAME)             Library soname: [libunistring-05abdd40.so.2.1.0]
 0x000000000000000e (SONAME)             Library soname: [libxml2-39f609e7.so.2.9.7]
 0x000000000000000e (SONAME)             Library soname: [libxml++-2-48354e93.6.so.2.0.7]

It seems that the embedded libraries have a SONAME that embeds the version of the library used. ldd also confirms that the library inter-dependencies requires a libraries with these exact SONAMEs. It also seems that auditwheel runs patchelf to enforce SONAMEs uniqueness by adding an hash to libraries' names.

Even if you somehow manage to load that other dependency version, only those symbols will be imported which are not already present. You cannot have the same symbol loaded twice in the process. So even linking in all our dependencies statically would not help.

Are you sure about that? If two libraries have different SONAMEs and are loaded using dlopen with RLTD_LOCAL there shouldn't be a naming clash or? And this is the default policy when cpython loads shared libraries.

[mhier]

Are you sure about that? If two libraries have different SONAMEs and are loaded using dlopen with RLTD_LOCAL there shouldn't be a naming clash or? And this is the default policy when cpython loads shared libraries.

That doesn't work due to some long standing bug in glibc, please read the StackOverflow post :-)

[bellaz89]

Are you sure about that? If two libraries have different SONAMEs and are loaded using dlopen with RLTD_LOCAL there shouldn't be a naming clash or? And this is the default policy when cpython loads shared libraries.

That doesn't work due to some long standing bug in glibc, please read the StackOverflow post :-)

Correct me if I am wrong. In your stackoverflow post you try to load the same library twice since libxml++ it is linked both in the application and the plugin. However, as soon as you change the SONAME, the libraries become distinct. So I would say that the content of the post doesn't appy. Or?

EDIT: In addition, I also see that this issue is related to dlmopen. However CPython should use dlopen. So, again, I am not sure if your past observations applies.

[mhier]

Correct me if I am wrong. In your stackoverflow post you try to load the same library twice since libxml++ it is linked both in the application and the plugin. However, as soon as you change the SONAME, the libraries become distinct. So I would say that the content of the post doesn't appy. Or?

No, I was trying to load different OR identical versions (both must work) of the same library in the same process, such that they don't disturb each other. The use case back then was a DeviceAccess backend depending e.g. on DOOCS, which would be loaded through the DMAP file in a process, which is already using DOOCS. Everything is fine as long as the libraries are in the same version, now the problem to solve was when the backend depends on a different version of DOOCS than the main process. I tried to load the library in such a way that it would allow the the symbols of DOOCS in the version required by the backend to coexist with the symbols of DOOCS in the version required by the main process.

I tried a lot of different mechanisms, starting with dlopen, which does not allow any such separation. dlmopen in theory can do such separation through namespaces, but it is broken.

Now, if I think more about it, there may be a difference between what I attempted earlier and the situation now: We are now probably only concerned about conflicting symbols which are all imported via dlopen. I had the situation that one "combatant" of the conflict was loaded in the main executable, where RTLD_LOCAL could not be specified. Hence specifying RTLD_LOCAL when loading the backend was not helping, as all symbols loaded by the main executable are visible by the backend anyway (RTLD_LOCAL only separates from future imports).

So maybe we can give this a try, but we should really check this. The theory could be wrong. In pybind11 they are playing extra tricks to prevent different modules built with different versions of pybind11 to conflict with each other - this would be unnecessary if the separation by RTLD_LOCAL would be sufficient. (I am thinking of writing two small Python modules which depend on each one library which would conflict each other, similar to my segregated linking example of the stackoverflow post.)

[bellaz89]

If this might help, I wrote a small example for this in C:

https://github.com/bellaz89/test_patchelf

It requires the command patchelf and the headers for libxml2 and is compiled/executed with bash compile_execute.sh

Here the following things happen:

• A .so library and an executable are compiled. libxml-2.0 is linked in both.
• Two copies of the same .so library are made. patchelf is run to make their SONAMEs distinct.
• The executable is run. It loads the two versions of the library with RTLD_LOCAL | RTLD_NOW.
• It executes the same methods for test_A.so and test_B.so that write and read to/from a global variable of the libraries. It verifies that the address spaces of the two libraries are separated.
• An .xml file is loaded from main, test_A.so and test_B.so to exclude the issue you encountered when using dlmopen.
• It shows the executable memory mapping to confirm that both test_A.so and test_B.so are effectively loaded in main address space.

So I would say that this effectively proves that changing the SONAME of a library makes it a different library from the point of view of dlopen. And on top of that the libraries, as soon as they have different SONAMEs, can expose the same exact symbols and be used as long as RTLD_LOCAL is used.

[bellaz89]

No, I was trying to load different OR identical versions (both must work) of the same library in the same process, such that they don't disturb each other. The use case back then was a DeviceAccess backend depending e.g. on DOOCS, which would be loaded through the DMAP file in a process, which is already using DOOCS. Everything is fine as long as the libraries are in the same version, now the problem to solve was when the backend depends on a different version of DOOCS than the main process. I tried to load the library in such a way that it would allow the the symbols of DOOCS in the version required by the backend to coexist with the symbols of DOOCS in the version required by the main process.

Unrelated to this topic. Could this issue be due to the two different versions of DOOCS having the same SONAME? If it is the case, I would say that the right approach is asking MCS to embed the DOOCS release number inside the SONAME. If it is not already the case for the most recent DOOCS releases.

E.g. In my system libraries, if I run
readelf -d libboost_chrono.so | grep SONAME
I get
0x0000000e (SONAME) Library soname: [libboost_chrono.so.1.75.0]

So the SONAMEs are versioned exactly to make multiple versions of the same library coexists in the same process. In addition, RTLD_DEEPBIND could also help to prevent the process looking at the already defined symbols in the global scope first if a different version of the library is directly linked to the main executable.

[mhier]

No, doocs has the version number properly in the soname, otherwise we would have huge problems already.

Unfortunately I don't have time to go deeper into this topic right now, but I guess it is not so urgent right now.

Btw: Here is the error I get when running the cbuildwheel:

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 3.2.1

Build options:
  platform: linux
  allow_empty: False
  architectures: x86_64
  build_selector: 
    build_config: *
    skip_config: cp314t-* *-win32 *_i686 *-musllinux_*
    requires_python: >=3.8
    enable: []
  output_dir: /home/mhier/software/sources/ChimeraTK-DeviceAccess-PythonBindings/wheelhouse
  package_dir: /home/mhier/software/sources/ChimeraTK-DeviceAccess-PythonBindings
  test_selector: 
    skip_config: *
  before_all: bash pre_build_manylinux.sh
  before_build: 
  before_test: 
  build_frontend: build
  build_verbosity: 1
  config_settings: 
  container_engine: docker
  dependency_constraints: pinned
  environment: 
    BUILD_DIR="${HOME}/build"
    PKG_CONFIG_PATH="${BUILD_DIR}/local/lib/pkgconfig/:/usr/lib/pkgconfig:${PKG_CONFIG_PATH}"
    LD_LIBRARY_PATH="${BUILD_DIR}/local/lib/:${LD_LIBRARY_PATH}"
    PATH="${BUILD_DIR}/local/bin/:${PATH}"
  manylinux_images: 
    x86_64: manylinux_2_28_x86_64
    i686: quay.io/pypa/manylinux_2_28_i686:2025.10.10-1
    pypy_x86_64: quay.io/pypa/manylinux_2_28_x86_64:2025.10.10-1
    aarch64: manylinux_2_28_aarch64
    ppc64le: quay.io/pypa/manylinux_2_28_ppc64le:2025.10.10-1
    s390x: quay.io/pypa/manylinux_2_28_s390x:2025.10.10-1
    armv7l: quay.io/pypa/manylinux_2_31_armv7l:2025.10.10-1
    riscv64: quay.io/pypa/manylinux_2_39_riscv64:2025.10.10-1
    pypy_aarch64: quay.io/pypa/manylinux_2_28_aarch64:2025.10.10-1
    pypy_i686: quay.io/pypa/manylinux_2_28_i686:2025.10.10-1
  musllinux_images: 
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2025.10.10-1
    i686: quay.io/pypa/musllinux_1_2_i686:2025.10.10-1
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2025.10.10-1
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2025.10.10-1
    s390x: quay.io/pypa/musllinux_1_2_s390x:2025.10.10-1
    armv7l: quay.io/pypa/musllinux_1_2_armv7l:2025.10.10-1
    riscv64: quay.io/pypa/musllinux_1_2_riscv64:2025.10.10-1
  pyodide_version: None
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command: 
  test_environment: 
  test_extras: 
  test_groups: 
  test_requires: 
  test_sources: 
  xbuild_tools: None

Cache folder: /home/mhier/.cache/cibuildwheel

Here we go!

Starting container image manylinux_2_28_x86_64...

info: This container will host the build for cp38-manylinux_x86_64, cp39-manylinux_x86_64, cp310-manylinux_x86_64, cp311-manylinux_x86_64, cp312-manylinux_x86_64, cp313-manylinux_x86_64, cp314-manylinux_x86_64...
+ docker version -f '{{json .}}'
+ docker image inspect manylinux_2_28_x86_64 --format '{{.Os}}/{{.Architecture}}'
Error response from daemon: No such image: manylinux_2_28_x86_64:latest
Error response from daemon: pull access denied for manylinux_2_28_x86_64, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

                                                                       ✕ 1.45s
cibuildwheel: error: Command ['docker', 'create', '--env=CIBUILDWHEEL', '--env=SOURCE_DATE_EPOCH', '--name=cibuildwheel-7e08a095-9dd4-420e-890a-b065d131c24c', '--interactive', '--volume=/:/host', '--platform=linux/amd64', '--pull=always', 'manylinux_2_28_x86_64', '/bin/bash'] failed with code 1. 

[bellaz89]

Thanks. I fixed the address to download the image from quay.io/pypa.