OpenMPI as build by default is slow

[davidedelvento]

On a Mellanox IB cluster, building OpenMPI in Spack with default settings results in bw and latency performance about one order of magnitude slower than expected, @scheibelp

On Spack's slack, Markus Janse of utsouthwestern.edu (not sure if it's the same user as @Markus92 ) kindly suggested to compile ucx@1.19 +rc simd=auto +ud +verbs +mlx5_dv to build ucx and have OpenMPI use that ucx version. However that ucx build defaults to ~thread_multiple whereas OpenMPI requires +thread_multiple

A side issue is that spack does not say "your ucx uses ~thread_multiple whereas OpenMPI requires +thread_multiple so I am not going to use it", but it either uses another version of ucx (which is slow) or it throws cryptic messages. I think this is known issue but if desired I can open a separate ticket on the spack repo.

When that thread_multiple issue is cleared up one can build ucx with spack install ucx@1.19 +rc simd=auto +ud +verbs +mlx5_dv +thread_multiple (which in my environment creates it with hash /bd6afih) and then build openmpi with spack install openmpi@5~vt fabrics=ucx ^/bd6afih which finally is fast (in production one may want to add the schedulers option to the openmpi build because otherwise some functionality will be missing)

By default, running configure outside of spack builds the "fast" version, probably because it detects IB presence and use it, whereas (I think) spack has a fixed set of default and maybe putting IB into it was not considered safe? If that is the case I am not sure how to solve it, since I see the dilemma between maximum compatibilities vs good performance, however arguably the most common use case for spack is a machine which does have IB.

Anyway, my pre-production full spec is (gcc@15.1.0 is spack-compiled and gcc@11.5.0 is OS default, the production spec will at least include slurm and probably a couple of other things)

[+]  teilp37  openmpi@5.0.8+atomics~cuda~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm~romio+rsh~static~two_level_namespace~vt+wrapper-rpath build_system=autotools fabrics:=ucx romio-filesystem:=none schedulers:=none arch=linux-rhel9-zen5 %c,cxx,fortran=gcc@15.1.0
[+]  ltryjx3      ^autoconf@2.72 build_system=autotools arch=linux-rhel9-zen4
[+]  ycexzea          ^gmake@4.4.1~guile build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  og7v2ze          ^m4@1.4.20+sigsegv build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  4kkqmxl              ^libsigsegv@2.14 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  ftbiojp      ^automake@1.16.5 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  rzsbkyn          ^compiler-wrapper@1.0 build_system=generic arch=linux-rhel9-zen4
[e]  nu6iakt          ^gcc@11.5.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel9-zen4
[+]  ba2t7ll          ^gcc-runtime@11.5.0 build_system=generic arch=linux-rhel9-zen4
[+]  vp4dznj      ^compiler-wrapper@1.0 build_system=generic arch=linux-rhel9-zen5
[+]  xtufstd      ^gcc@15.1.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  j3wjelk          ^diffutils@3.10 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  7zvjahb              ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  5shcgy6          ^gawk@5.3.1~nls build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  lhpq6rx              ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  cspea6h          ^gmp@6.3.0+cxx build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  tfh7kqq          ^libtool@2.4.7 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  nkuaety              ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  vrgqjnw          ^mpc@1.3.1 build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  s4if44g          ^mpfr@4.2.1 build_system=autotools libs:=shared,static patches:=3ec29a6 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  rpg6m2d              ^autoconf-archive@2023.02.20 build_system=autotools arch=linux-rhel9-zen4
[+]  4e6h3it          ^texinfo@7.1 build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  tfx2jaz              ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  uwic5ds                  ^tar@1.35 build_system=autotools zip=pigz arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  ta67jqa                      ^pigz@2.8 build_system=makefile arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  h6b5aah          ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  e5y5nqr      ^gcc-runtime@15.1.0 build_system=generic arch=linux-rhel9-zen5
[e]  a4mj5kw      ^glibc@2.34 build_system=autotools arch=linux-rhel9-zen4
[+]  lfatrna      ^gmake@4.4.1~guile build_system=generic arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  xcpgot3      ^hwloc@2.11.1~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  bcrt2md          ^libpciaccess@0.17 build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  7w7j5oa              ^util-macros@1.20.1 build_system=autotools arch=linux-rhel9-zen5
[+]  xk2b6uf          ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  q62gttj              ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  wd2ykyc          ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  spbodbk          ^pkgconf@2.3.0 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  qsxg7ms      ^libevent@2.1.12+openssl build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  b2xmvrr          ^openssl@3.4.1~docs+shared build_system=generic certs=mozilla arch=linux-rhel9-zen5 %c,cxx=gcc@14.3.0
[+]  synbm2z              ^ca-certificates-mozilla@2025-05-20 build_system=generic arch=linux-rhel9-zen5
[+]  fv5hjaq              ^gcc@14.3.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  jphgm36              ^gcc-runtime@14.3.0 build_system=generic arch=linux-rhel9-zen5
[+]  n2filrp      ^libtool@2.4.7 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  r6f3azb          ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  sfpfwoc              ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  6h335s3                  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  shsrphe                  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  hzg47yj                  ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  lftaoqh                      ^pkgconf@2.3.0 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  yzqxtfe                  ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  4zba2fv                  ^tar@1.35 build_system=autotools zip=pigz arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  75ondkw                      ^pigz@2.8 build_system=makefile arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  5dbgwvt                  ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[e]  sixxvdb          ^gcc@11.5.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel9-zen5
[+]  5sb45n2          ^gcc-runtime@15.1.0 build_system=generic arch=linux-rhel9-zen4
[+]  baoakpz          ^gmake@4.4.1~guile build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  2veasz3          ^m4@1.4.20+sigsegv build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  day4g3n              ^diffutils@3.10 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  ygdcno3              ^libsigsegv@2.14 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  bvcq7f3      ^numactl@2.0.18 build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  v4scskm          ^autoconf@2.72 build_system=autotools arch=linux-rhel9-zen4
[+]  bf5xurz              ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  2bzcgim                  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  ycgjn6s                  ^gdbm@1.23 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  3fqqn3q                      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  ym744c5          ^automake@1.16.5 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  zxc2hko      ^openssh@9.9p1+gssapi build_system=autotools arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  7zchf34          ^krb5@1.21.3+shared build_system=autotools arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  a24zau3              ^bison@3.8.2~color build_system=autotools arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  yuxg54c          ^libedit@3.1-20240808 build_system=autotools arch=linux-rhel9-zen5 %c=gcc@14.3.0
[+]  ivnzlwt          ^libxcrypt@4.4.38~obsolete_api build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  j5svvxd              ^gcc-runtime@11.5.0 build_system=generic arch=linux-rhel9-zen4
[+]  qd33je7              ^gmake@4.4.1~guile build_system=generic arch=linux-rhel9-zen5 %c=oneapi@2025.2.0
[+]  52tjpt5                  ^intel-oneapi-compilers@2025.2.0~amd+envmods~nvidia build_system=generic arch=linux-rhel9-zen5
[+]  jsrnxfn                  ^intel-oneapi-runtime@2025.2.0 build_system=generic arch=linux-rhel9-zen5
[+]  aqkyniu              ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  54bbtfz                  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  erd52c7                  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  gtalpai                      ^diffutils@3.10 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  al7yesp                          ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  gtloqno                  ^gdbm@1.23 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  mjpwruk                      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  en4ggz6                          ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  in4sbgk                              ^pkgconf@2.3.0 build_system=autotools arch=linux-rhel9-zen5 %c=oneapi@2025.2.0
[+]  qtupk5a      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  rkipsnf          ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  ysskcrn          ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  p4fwbhf          ^gdbm@1.23 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  4izsgk5      ^pkgconf@2.3.0 build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  smnow7f      ^pmix@5.0.5~munge~python build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  bd6afih      ^ucx@1.19.1~assertions~backtrace_detail~cma~cuda~dc~debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging+mlx5_dv+openmp+optimizations~parameter_checking+pic+rc~rdmacm~rocm+thread_multiple~ucg+ud+verbs~vfs~xpmem build_system=autotools libs:=shared,static opt=3 simd:=auto arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  cxasxkk          ^rdma-core@57.0~ipo+man_pages+pyverbs+static build_system=cmake build_type=Release generator=make arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  ghlpw2v              ^cmake@3.31.8~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  nuoc3fl                  ^curl@8.11.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  wzyr6ia                      ^nghttp2@1.65.0 build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  nnmwyvd                      ^openssl@3.4.1~docs+shared build_system=generic certs=mozilla arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0
[+]  4w2bxdh              ^libnl@3.3.0 build_system=autotools arch=linux-rhel9-zen5 %c=gcc@15.1.0
[+]  mgnk4s5                  ^bison@3.8.2~color build_system=autotools arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  e2dic7h                  ^flex@2.6.3+lex~nls build_system=autotools arch=linux-rhel9-zen5 %c,cxx=gcc@15.1.0
[+]  xs25qsv              ^py-docutils@0.20.1 build_system=python_pip arch=linux-rhel9-zen5
[+]  iatatgb                  ^py-pip@25.1.1 build_system=generic arch=linux-rhel9-zen5
[+]  pjtj7km                  ^py-setuptools@80.9.0 build_system=generic arch=linux-rhel9-zen5
[+]  nqxrasz                  ^py-wheel@0.45.1 build_system=generic arch=linux-rhel9-zen5
[+]  ehpcdnn                  ^python@3.13.5+bz2+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-rhel9-zen4 %c,cxx=gcc@15.1.0
[+]  erd36nx                      ^expat@2.7.1+libbsd build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@15.1.0
[+]  k5lfzi6                          ^libbsd@0.12.2 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@15.1.0
[+]  bdhpsaj                              ^libmd@1.1.0 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@15.1.0
[+]  p7p5pia                      ^libffi@3.4.8 build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@15.1.0
[+]  gc7uv44                      ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel9-zen4 %c=gcc@11.5.0
[+]  cjl6avv                      ^util-linux-uuid@2.41 build_system=autotools arch=linux-rhel9-zen4 %c=gcc@15.1.0
[+]  7ug7i7b                  ^python-venv@1.0 build_system=generic arch=linux-rhel9-zen5
[+]  dgsrt65      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rhel9-zen4 %c,cxx=gcc@11.5.0

[Markus92]

Yeah that's me. I'm very surprised though as I see the following in OpenMPI package definition:

    with when("fabrics=ucx"):
        depends_on("ucx")
        depends_on("ucx +cuda", when="+cuda")
        depends_on("ucx +thread_multiple", when="+thread_multiple")
        depends_on("ucx +thread_multiple", when="@3.0.0:")

And as you have OpenMPI 5, it should always build with the thread_multiple variant, not? That's how it looks like on our end.

The way I got it, and this might help, is to set the default variants correctly in packages.yaml. We got this that works with our Connect-X 3, 4 and 6 cards, also builds with lustre/gpfs/slurm bindings:

  openmpi:
    variants: +gpfs +lustre +internal-pmix fabrics=ucx schedulers=slurm
  ucx:
    variants: +mlx5_dv +verbs +dc +rc
  slurm:
    externals:
    - spec: slurm@22.05.9
      prefix: /usr
    buildable: false
  lustre:
    externals:
    - spec: lustre@2.14
      prefix: /usr

Slurm we got defined as an external as we use the EPEL versions for now. Lustre is also an external, GPFS we just do the 'pray it links with the right library' approach, which has kinda worked out so far.

Note we have to use the internal pmix as the external system one on our RHEL 9 system is too old and incompatible with OpenMPI 5.

One thing left: proper GPU Direct support. We haven't managed to get that one down yet without a bunch of segfaults.

[davidedelvento]

Thanks for chiming it. Sorry for being unclear, let me see if I rectify that.

And as you have OpenMPI 5, it should always build with the thread_multiple variant, not? That's how it looks like on our end.

Correct that is what OpenMPI wants and builds ucx on its own. But on its own, it builds it without mellanox, so I built it manually with

spack install ucx@1.19 +rc simd=auto +ud +verbs +mlx5_dv

which in turns defaults to ~thread_multiple and it is not (and cannot) be picked up by OpenMPI, hence my initial confusion (since spack doesn't really tell what the problem is).

[davidedelvento]

One thing left: proper GPU Direct support. We haven't managed to get that one down yet without a bunch of segfaults.

Yeah, let me know if you figure that out.