[Option 2 Example] Dont land

[H-Huang]

Option 2 of #1682

In our custom overlap_f_b function we write run_forward() and run_backward(). run_backward() is run as a separate thread so that we can have both forward and backward running together side by side. Looks like this:

We added hooks before and after expert parallel dispatch and combine to signal boundary points, which now turns into

Now in each of these red blocks, we just need threading.Barrier(2).wait() so that the comm and compute are scheduled in lock-step.

[H-Huang]

Running with:

TORCH_NCCL_TRACE_BUFFER_SIZE=2000 TORCH_NCCL_DUMP_ON_TIMEOUT=true TORCH_FR_DUMP_TEMP_FILE=./nccl_trace_rank_ NGPU=4 CONFIG_FILE="./torchtitan/models/deepseek_v3/train_configs/debug_model.toml" ./run_train.sh

CUDA_LAUNCH_BLOCKING

TORCH_NCCL_TRACE_BUFFER_SIZE=2000 TORCH_NCCL_DUMP_ON_TIMEOUT=true TORCH_FR_DUMP_TEMP_FILE=./nccl_trace_rank_ NGPU=4 CONFIG_FILE="./torchtitan/models/deepseek_v3/train_configs/debug_model.toml" CUDA_LAUNCH_BLOCKING=1 ./run_train.sh