Trace Files aren't written out when using dyno gputrace

[adirajagopal]

I'm currently trying to run inference profiling on a cuda kernel that's launched from pytorch. I am inside a docker container based which has cuda 12.5. I run:

• dynolog --enable_ipc_monitor --enable_gpu_monitor --dcgm_lib_path=/usr/lib/x86_64-linux-gnu/libdcgm.so

• KINETO_DAEMON_INIT_DELAY_S=3 KINETO_USE_DAEMON=True python profiling/profile.py : which runs the kernel I would like to profile in a loop of 100 iterations.

• The kernel being profiled is wrapped in the following torch profiler command:
  with torch.profiler.profiler(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True)

• When I launch the script, I get the message: INFO:2024-07-25 20:57:39 4278:4278 init.cpp:131] Registering daemon config loader, cpuOnly = 0

• I then run dyno gputrace --log-file /home/trace.json which outputs Trace output files will be written to: /home/trace_7605.json

But I never end up seeing the trace files that are written. I am using the latest version of dynolog, i.e. dynolog_0.3.2-0-amd64.deb. Any help on how I could get these traces would be much appreciated.

[briancoutinho]

@adirajagopal quick note, you don't need to add any code to include torch profiler in your program ("with torch.profiler.profiler(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True)")

It can so happen that the torch profiler based trace collection conflicted with dyno based collection. Torch profiler will always take precedence and so you won't get an output trace.