How to compare HPX perf between ETH and IB?

[Kihang-Youn]

HI, How can I compare the performance of Ethernet and InfiniBand in HPX?
I am testing on a cluster where both Ethernet and InfiniBand are connected. It seems that controlling it through OMPI parameters is not working, so I would like to ask for clarification.

Here's our commands:

ETH:
export OMPI_MCA_pml=ob1
export OMPI_MCA_btl=tcp,self,vader
hpx_test --hpx:print-bind --hpx:threads=1 --hpx:ini=hpx.logging.level=4 --hpx:debug-hpx-log --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.tcp.enable=0 --hpx:ini=hpx.parcel.mpi.multithreaded=0

IB:
export OMPI_MCA_pml=ucx
export UCX_TLS=all
hpx_test --hpx:print-bind --hpx:threads=1 --hpx:ini=hpx.logging.level=4 --hpx:debug-hpx-log --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.tcp.enable=0 --hpx:ini=hpx.parcel.mpi.multithreaded=0

Could it be that other HPX runtime options are taking precedence?
Thank you in advance for your support.

[hkaiser]

It seems that controlling it through OMPI parameters is not working

How do you know that it is 'not working'? What do you expect to see?

Also, (assuming hpx_test is your application), it doesn't seem that you are running your test in distributed mode.

[Kihang-Youn]

Hi @hkaiser,
Thank you for your support.

First, I'm executing alltoall, allgather, and allreduce in the HPX and measuring the execution time using tic-toc, but there is no change between ETH and IB.

Are you thinking that way because there are no commands like mpirun or mpiexec? I am submitting jobs through SLURM, so I am running them in a multi-process manner. Would it be correct to say that using the command below executes it in distributed mode?

mpirun -np 4 ./hpx_test
--hpx:print-bind
--hpx:threads=8
--hpx:ini=hpx.logging.level=4
--hpx:debug-hpx-log
--hpx:ini=hpx.parcel.mpi.enable=1
--hpx:ini=hpx.parcel.tcp.enable=0
--hpx:ini=hpx.parcel.mpi.multithreaded=1

[hkaiser]

Would it be correct to say that using the command below executes it in distributed mode?

Yes, that answers my question. This runs things in distrbuted. Depending on your srun setup this may run all four ranks on the same physical compute node however, in which case networking would go through shared memory in any case. Also, enabling logging will slow down the execution significantly, which might hide any performance differences you're trying to assess.

In any case, to answer your initial question - HPX does not interfere with the MPI environment settings you specify.

[Kihang-Youn]

Thank you for your prompt response. If there are any issues with my execution, I would like to correct them.
I have two questions regarding the runtime command. (*I was running OMB before executing HPX within the srun script, and since I observed the expected performance difference, I assumed it was working properly.)

1. I'm a bit confused about the option to increase the number of MPI processes. Is the following correct?
   If it's correct, could you let me know how many MPI ranks are being used? For example, 4 or 32
   mpirun -np 4 ./hpx_test
   --hpx:threads=8
   --hpx:ini=hpx.parcel.mpi.multithreaded=1

2. Could you please confirm if the following option is correct for comparing the performance difference between ETH and IB?
   IB: --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.tcp.enable=0
   ETH: --hpx:ini=hpx.parcel.mpi.enable=0 --hpx:ini=hpx.parcel.tcp.enable=1

[hkaiser]

Thank you for your prompt response. If there are any issues with my execution, I would like to correct them. I have two questions regarding the runtime command. (*I was running OMB before executing HPX within the srun script, and since I observed the expected performance difference, I assumed it was working properly.)

1. I'm a bit confused about the option to increase the number of MPI processes. Is the following correct?
   If it's correct, could you let me know how many MPI ranks are being used? For example, 4 or 32
   mpirun -np 4 ./hpx_test
   --hpx:threads=8
   --hpx:ini=hpx.parcel.mpi.multithreaded=1

This creates four ranks, each with 8 cores running.

2. Could you please confirm if the following option is correct for comparing the performance difference between ETH and IB?
   IB: --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.tcp.enable=0
   ETH: --hpx:ini=hpx.parcel.mpi.enable=0 --hpx:ini=hpx.parcel.tcp.enable=1

If you disable the MPI parcelport (hpx.parcel.mpi.enable=0), then it will not select ETH or IB. Also, I doubt that things will run correctly if you use mpirun to launch things while relying on the TCP parcelport (hpx.parcel.tcp.enable=1).