MPCD using MPI

[sgoh446]

I am trying to run the hoomd-mpcd routine using multiple CPUs or GPUs.
I managed to run mpcd simulations with a single rank, both on CPU and on GPU,
but the script I used does not run on multiple ranks.
As far as I understand, I can simply run the same script written for a single core, can't I?
I would be very grateful if you could help me.

Here is the code (test_hoomd_mpcd.py) that I am using:

import hoomd
import numpy

device = hoomd.device.GPU()
simulation = hoomd.Simulation(device=device, seed=1)

L=64
snapshot = hoomd.Snapshot()
snapshot.configuration.box = [L,L,L,0,0,0]

density = 10
rng = numpy.random.default_rng(seed=42)

snapshot.mpcd.types = ['A']
snapshot.mpcd.N = int(density * L * L * L)
snapshot.mpcd.position[:] = [0.5*L, 0.5*L, 0.5*L] * rng.uniform(low=-1,high=1,size=(snapshot.mpcd.N,3))

kT = 1
velocity = rng.normal(0.0, numpy.sqrt(kT), (snapshot.mpcd.N, 3))
velocity -= numpy.mean(velocity, axis=0)
snapshot.mpcd.velocity[:] = velocity

simulation.create_state_from_snapshot(snapshot)

integrator = hoomd.mpcd.Integrator(dt=0.02)
integrator.collision_method = hoomd.mpcd.collide.StochasticRotationDynamics(
    period=1, angle=130, kT=kT
)

integrator.streaming_method = hoomd.mpcd.stream.Bulk(
    period=integrator.collision_method.period
)

integrator.mpcd_particle_sorter = hoomd.mpcd.tune.ParticleSorter(trigger=2)
simulation.operations.integrator = integrator

simulation.run(2)
device.notice(f'{simulation.tps}')

On the HPC I am using, I run
srun --exact -n 4 --gres=gpu:4 python test_hoomd_mpcd.py

And the error messages I got:

Traceback (most recent call last):
  File "/p/project1/cdim/hoomd/test_hoomd_mpcd.py", line 14, in <module>
Traceback (most recent call last):
  File "/p/project1/cdim/hoomd/test_hoomd_mpcd.py", line 14, in <module>
Traceback (most recent call last):
  File "/p/project1/cdim/hoomd/test_hoomd_mpcd.py", line 14, in <module>
    snapshot.mpcd.types = ['A']
    snapshot.mpcd.types = ['A']
    ^^^^^^^^^^^^^
  File "/p/project1/cdim/hoomd/hoomd/snapshot.py", line 415, in mpcd
    ^^^^^^^^^^^^^
  File "/p/project1/cdim/hoomd/hoomd/snapshot.py", line 415, in mpcd
    snapshot.mpcd.types = ['A']
    ^^^^^^^^^^^^^
  File "/p/project1/cdim/hoomd/hoomd/snapshot.py", line 415, in mpcd
    raise RuntimeError("Snapshot data is only present on rank 0")
RuntimeError: Snapshot data is only present on rank 0
    raise RuntimeError("Snapshot data is only present on rank 0")
RuntimeError: Snapshot data is only present on rank 0
    raise RuntimeError("Snapshot data is only present on rank 0")
RuntimeError: Snapshot data is only present on rank 0
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

Thank you very much!!!

[joaander]

As the error message states, the snapshot data is present only on rank 0. The MPI tutorial shows how you can access global snapshot data in an if snapshot.communicator.rank == 0:.

[mphoward]

@sgoh446 Could you try compiling the branch for this draft pull request:

#1897

and let me know if it fixes the issue you were having for the L = 128 test case on 48 CPUs? Thanks!

[sgoh446]

Yes, it does, I was able to run L=128, density 50, and also L=256 with density 10.

Here is the speedup for L=256, density 10:
1 GPU: 17.584475932281656
2: 17.103018105468752
4: 23.089839622014704
8: 31.751673968096394
16: 40.11789445643914
32: 37.171577496698696
64: 55.32130058164815
128: 68.28837909337616
256: 101.95002358868672

which seems not as good as the CPU case.
It seems that the system size is still small, but I couldn't go beyond that due to the memory issue at initialization.

I don't have a particular use case, but I am doing MPCD benchmarks for my publication.
Could we try "make_random"?

Thanks a lot!

[mphoward]

Great to hear! We'll get the PR polished up and merged soon then.

Yes, it doesn't surprise me that the scaling to that many A100s is poor for this system size. If you are able to wait a week or so, I should be able to get the random initialization code path working again (the C++ code is still there, but it is missing a Python interface) so that you can try a larger system. I would also welcome a contribution from you if you would like to help implement it!

By the way, if you have performed any profiling, I would love to have that data to know what the bottleneck is, if you are willing to share it. I have not revisited the scaling behavior of this software since my 2018 publication and that test used K20x GPUs that are vastly different in terms of compute density than A100. At the time, I think the cell communication was the limiting factor, and I had some ideas about how to improve that... but they were not worth pursuing with the hardware at the time.

[sgoh446]

We are revising our manuscript and the deadline is approaching.
For now, the CPU scalability seems good enough for my comparison purposes, and it does not suffer too much from memory issues.

As for the profiling: I haven't done anything in this direction, sorry.

Anyway, thank you very much for your help!

[mphoward]

OK, no problem and you're welcome!