Crossbar.io on FreeBSD

[jmgurney]

After spending way too long dealing w/ the mess that autobahn deps added over the years, I finally looked a bit closer at things, and found out that the default extras are just stupidly complex, and most of the do not even work on FreeBSD (and should therefore just be disabled).

As you can see here:
https://github.com/crossbario/autobahn-python/blob/master/setup.py#L258

it lists the following extras:
encryption, scram, nvx, dev, xbr and ui

And none of those are listed at:
https://autobahn.readthedocs.io/en/latest/installation.html

Nor does it document that installing by default requires you to install additional packages as dependencies.

I found the following deps were required:
cairo snappy gobject-introspection libargon2

but it's entirely possible that more deps are required as well.

[oberstet]

well, there are no "default extras", and you can do a plain pip install autobahn on python 3 and get a working websocket and wamp library. if you do pip install autobahn[all], that will install, well "all", which means you need all dependencies. there are different subsets of interest, and those are documented here https://github.com/crossbario/autobahn-python#extensions

ah, and a plain install has exactly 4 package dependencies

autobahn-python/setup.py

Line 251 in 3da0c26

install_requires=[

rgd FreeBSD packaging or similar: sorry, no clue. above is true when using a python venv eg

[jmgurney]

Ok, looks like the problem is that I'm installing crossbar:

crossbar 22.3.1 requires autobahn[compress,encryption,scram,serialization,twisted,xbr]>=22.2.1, but you have autobahn 17.9.3 which is incompatible.

and pip isn't always clear about what is getting installed when, and which package caused the install to happen.

(ignore the old version, I'm in the process of trying to get a newer version installed and working and it's a painful process.)

crossbar pulls in vmprof, which doesn't compile on FreeBSD:

      cc -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -I/usr/local/include -fPIC -I/home/freebsd/haws/p/incl
ude -I/usr/local/include/python3.9 -c src/_vmprof.c -o build/temp.freebsd-14.0-CURRENT-arm64-3.9/src/_vmprof.o -Wno-unused -DVMPROF_BSD=1 -I/usr/local/include -DVMPROF_UNIX=1 -I src/ -I src/
libbacktrace                                                                                   
      In file included from src/_vmprof.c:12:                                                  
      In file included from src/vmprof_common.h:19:                                
      src/vmprof_getpc.h:147:27: error: no member named 'mc_rip' in 'struct __mcontext'        
        return signal_ucontext->PC_FROM_UCONTEXT;   // defined in config.h                     
               ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~                                               
      src/vmprof_config.h:11:42: note: expanded from macro 'PC_FROM_UCONTEXT'                                                                                                                           #define PC_FROM_UCONTEXT uc_mcontext.mc_rip                                                                                                                                                                                        ^                                                                                                                                              
      src/_vmprof.c:42:6: error: "platform not supported"                                                                                                                                           #    error "platform not supported"                                                                                                                                                                ^                                                                                                                                                                                  

so, looks like the issue is pip prints it's dep tree only in case another package managed to be already satisfied, as I did manage to find this:

Requirement already satisfied: pytz in ./p/lib/python3.9/site-packages (from vmprof->crossbar->-r requirements.txt (line 4)) (2022.1)                                                         

[oberstet]

ok, autobahn for crossbar on FreeBSD, this is a different matter. this indeed requires autobahn with the extras you list, see

https://github.com/crossbario/crossbar/blob/8521972e03e20b56193046343a9f5209ac81ac5c/requirements-min.txt#L2

some pain is expected, as crossbar has over 100 deps in total (direct and indirect). if you are on CPython3 + Linux + pip/virtualenv (or on Docker), it is all good and smooth, other variants of Python, OS and package manager: your mileage will vary;)

anyways, two hints:

1. in autobahn 22.3.2, the (indirect) gobject/gtk dependencies have been removed, and hence this should be good

https://github.com/crossbario/autobahn-python/blob/master/docs/changelog.rst#2232

2. vmprof: this might be tricky. is vmprof the only dependency not working for you in FreeBSD? eg can you use everything but vmprof from this list?

https://github.com/crossbario/crossbar/blob/master/requirements-min.txt

if so, then we might think about adding run-time import guards for vmprof based features rather than adding it as a hard dependency ..

[oberstet]

rgd vmprof on FreeBSD:

vmprof/vmprof-python#168

if you have additional information, please consider commenting on that issue also for upstream to know ..

fwiw, there also is an issue for Python 3.10 with vmprof: vmprof/vmprof-python#240

finally, you will notice

https://github.com/crossbario/crossbar/blob/8521972e03e20b56193046343a9f5209ac81ac5c/requirements-min.txt#L17

refering forks .. sadly, didn't had time to get this fixed in upstream. obviously, if you strive to have all deps packaged in FreeBSD native packages themself, this is problematic ..

[jmgurney]

I have provided the errors that I'm running on that first vmprof issue. It likely could be that vmprof just does not support arm64 that I'm trying to get it to run on.

Also, all of this is not using native packages, this is using pip entirely to install and fetch everything. I've abandoned FreeBSD native packaging for Python in preference of venv's. Just wish python packages would fall back to python native code when they fail to compile, (within reason of course, like argon2 should fall back to a native python implementation if the C code failed).

The chances of getting the argon2 port fixed, which is another failure, looks like that won't happen, as it hasn't been touched in 6 years. I have opened up issues reporting it, and the fix is extremely simple.

I have also opened an issue for numpy as well: numpy/numpy#21386

That can be solved by pinning the version, but argon2 would require maintaining a fork, and I'm not interested in continuing to prop up a long dead bit of code when the correct solution is to remove it as a dep from crossbar/autobahn. Though, since it appears that crossbar is willing to maintain forks, maybe crossbar could?

But even fixing the other two modules, the vmprof port likely won't be fixed anytime soon.

[jmgurney]

btw, should I open a new issue to track the documentation not matching? or have you opened an issue to track that already?

[oberstet]

this is using pip entirely to install and fetch everything. I've abandoned FreeBSD native packaging for Python in preference of venv

+1! fwiw, I reached a similar conclusion over the years. OS-level packaging of Python packages (with or without native code) is wrong. sadly, Python native packaging (distutils, wheel, pip, ..) is ok'ish, but not great. The problems run deep .. eg CPyExt, that is the C-level API of the CPython language implementation isn't standardized, and by exposing the GIL, has pretty much formed the whole Python package universe. Anyways, one day I will learn Rust;)

---

argon2

yes, all Python packages should have Python-only versions, and we are actively selecting dependencies for following that, eg

autobahn-python/setup.py

Line 52 in 3da0c26

# C-based WebSocket acceleration (only use on CPython, not PyPy!)

only drags in native code on CPython, not PyPy, because the JIT of the latter will (most of the time) produce faster machine code on the fly than you old ahead-of-time C compiler anyways ..

PyPy is also the reason why we use argon2-cffi:

autobahn-python/setup.py

Line 101 in 3da0c26

'argon2_cffi>=20.1.0', # MIT license

here:

https://argon2-cffi.readthedocs.io/
https://github.com/hynek/argon2-cffi
https://github.com/hynek/argon2-cffi-bindings

which of above isn't supported on fbsd or unmaintained?

---

I have also opened an issue for numpy as well: numpy/numpy#21386

+1! fwiw, we couldn't (or wouldn't want to) remove numpy from crossbar, and numpy seems pretty important anyways .. for FreeBSD in general that is

---

But even fixing the other two modules, the vmprof port likely won't be fixed anytime soon.

as mentioned, we can make crossbar work without vmprof and remove the hard dependency. it's just that I was asking for "the rest" of deps on FreeBSD to figure out if fixing vmprof would then actually solve your actual issue: get crossbar running on freebsd

---

rgd issues, if you care, best would probably be 2 new issues on the crossbar repo: one for freebsd doc hints, and one for making vmprof a run-time detected soft dependency. thanks!

---

saw on the numpy issue you seem to target

FreeBSD/aarch64 (aka arm64), using clang

interesting! arm64 fbsd: for cloud/networking or embedded?

[jmgurney]

which of above isn't supported on fbsd or unmaintained?

argon2-cffi installs on FreeBSD perfectly fine. The problem is that argon2 is ALSO a dependency and some module isn't using argon2-cffi instead of argon2. I don't know where it's coming from.

$ pip list | grep argon
argon2-cffi        20.1.0                      /home/freebsd/haws/p/src/argon2-cffi/src

That is the official version w/o any modifications. I probably installed it manually as editable when I was trying to track down why argon2 wasn't compiling, because at the time I didn't know that crossbar/autobahn was depending upon two different argon2 python modules.

interesting! arm64 fbsd: for cloud/networking or embedded?

embedded... installing on a Rock64.

[oberstet]

The problem is that argon2 is ALSO a dependency and some module isn't using argon2-cffi instead of argon2. I don't know where it's coming from.

That's a bug (which we can and should fix), and it comes from cfxdb:

(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/zlmdb$ pipdeptree -p crossbar
...
crossbar==22.4.1.dev6
...
  - autobahn [required: >=22.3.2, installed: 22.4.1.dev6]
    - cryptography [required: >=3.4.6, installed: 36.0.2]
      - cffi [required: >=1.12, installed: 1.15.0]
        - pycparser [required: Any, installed: 2.21]
    - hyperlink [required: >=21.0.0, installed: 21.0.0]
      - idna [required: >=2.5, installed: 3.3]
    - setuptools [required: Any, installed: 62.1.0]
    - txaio [required: >=21.2.1, installed: 22.2.1]
...
  - cfxdb [required: >=22.3.1, installed: 22.4.1.dev4]
    - argon2 [required: >=0.1.10, installed: 0.1.10]
    - autobahn [required: >=22.3.2, installed: 22.4.1.dev6]
      - cryptography [required: >=3.4.6, installed: 36.0.2]
        - cffi [required: >=1.12, installed: 1.15.0]
          - pycparser [required: Any, installed: 2.21]
      - hyperlink [required: >=21.0.0, installed: 21.0.0]
        - idna [required: >=2.5, installed: 3.3]
      - setuptools [required: Any, installed: 62.1.0]
      - txaio [required: >=21.2.1, installed: 22.2.1]

[oberstet]

I've added 2 issues resulting from above:

• Depend on argon2-cffi, not argon2 cfxdb#86
• Make vmprof a soft-dependency crossbar#2002

[oberstet]

rgd gobject dependency: this comes from pip install .[all] via

autobahn-python/setup.py

Line 150 in 528f6a0

extras_require_ui = [

workaround to install "almost all", but not "ui":

pip install autobahn[twisted,accelerate,compress,serialization,encryption,scram,xbr]

---

Collecting PyGObject>=3.40.0
  Downloading PyGObject-3.42.2.tar.gz (719 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 719.5/719.5 kB 3.7 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      Collecting setuptools
        Downloading setuptools-65.3.0-py3-none-any.whl (1.2 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.6 MB/s eta 0:00:00
      Collecting wheel
        Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting pycairo
        Downloading pycairo-1.21.0.tar.gz (340 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 340.7/340.7 kB 1.4 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Building wheels for collected packages: pycairo
        Building wheel for pycairo (pyproject.toml): started
        Building wheel for pycairo (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
      
        × Building wheel for pycairo (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [12 lines of output]
            running bdist_wheel
            running build
            running build_py
            creating build
            creating build/lib.linux-aarch64-cpython-310
            creating build/lib.linux-aarch64-cpython-310/cairo
            copying cairo/__init__.py -> build/lib.linux-aarch64-cpython-310/cairo
            copying cairo/__init__.pyi -> build/lib.linux-aarch64-cpython-310/cairo
            copying cairo/py.typed -> build/lib.linux-aarch64-cpython-310/cairo
            running build_ext
            'pkg-config' not found.
            Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for pycairo
      Failed to build pycairo
      ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[om26er]

Regarding the above error, I believe sudo apt install pkg-config should help.