Configuring mpireport service for MPMD usecase?

[mdmosby]

Is there a way to pass a specific communicator to the mpireport service? My use case is that I have three applications that couple through MPMD, and when I try to get profile data from one of them using caliper+mpi, the programs all hang at exit/finalize, presumably due to an un-matched MPI_Allreduce in generating the report. A more-detailed example of the MPI launch is:

CALI_CONFIG=runtime-report mpirun -n 1 app-with-cali : -n 1 app-without-cali : -n 1 other-app-without-cali

My current workaround is to configure caliper without the mpireport service, but this will not be sufficient for profiling simulations "in the wild". Any suggestions on where I might look in the docs to make things function properly, or is this a missing feature?

[daboehme]

Hi @mdmosby,

There is a (not much publicized) way to flush performance data for a specific communicator with the Caliper ConfigManager API. I've attached an example below. Let me know if this helps!

#include <caliper/cali.h>
#include <caliper/cali-manager.h>
#include <caliper/MpiChannelManager.h>

// ...

int worldrank;
MPI_Comm subcomm;
MPI_Comm_rank(MPI_COMM_WORLD, &worldrank);
MPI_Comm_split(MPI_COMM_WORLD, worldrank % 2, worldrank, &subcomm);

cali::ConfigManager mgr("runtime-report");
cali::MpiChannelManager mpimgr(subcomm);
mpimgr.add(mgr);
mpimgr.start();

// ...

mpimgr.collective_flush(); // flush runs on subcomm

[mdmosby]

Thanks for your response! I'm finally getting back to this work and it looks like 90% of what I need. How would I make this work in the case that my current activation of caliper is through the environment? I can insert a singleton pattern to set the MpiChannelManager communicator, but it is unclear from the documentation how I might get access to all active ChannelControllers. Do I have to have Cali setup calls like your example to be able to make this work, or is there some API call I'm missing that would allow me to get the 'runtime-report' ConfigManager if it is setup or a NULL object if it isn't. I'm not ready to plumb Caliper activation/configuration through my app's parsing etc, but I need to make this work for MPMD or the optional performance monitoring/reporting is going to rot or get ripped out.

[mdmosby]

@daboehme any thoughts/suggestions?

[daboehme]

Hi @mdmosby, unfortunately it's not possible right now to access the internal ConfigManager that manages the channels created from the CALI_CONFIG environment variable. Another workaround for the MPI problem might be to disable the aggregation across ranks for the runtime-report. You can do that with runtime-report,aggregate_across_ranks=false. This way each rank that has this Caliper configuration is enabled will write its output separately. If multiple MPI ranks are doing that it's best to let each of them write into a separate file. You can do that with runtime-report,aggregate_across_ranks=false,profile.mpi,output=report_%mpi.rank%.txt. Hope this helps!