[Bugfix] Fix simple_rl_multiprocess.py to be runnable with recent vLLM version

[Lucaskabela]

Summary

We make a handful of QOL changes to make simple_rl_multiprocess.py work with recent vLLM. These are:

1. Updating the .gitignore to ignore the converted/ and models/ directories that are added by these scripts

2. pyrefly suppress to get precommit signal passing

3. Update to README.md to add monarch requirement

4. AttentionBackendEnum forwarded in locations needed (similar to [Experimental][rl][vllm compat] Update simple_rl example to work with vLLM nightly #2219)

5. Wiring through enable_gqa support in vllm_compat attention - this requires reshaping when q.shape[1] != k.shape[1] and using an output tensor explicitly with the proper shape

With these changes, we are able to run the simple_rl_multiprocess script again

Test Plan

VLLM_BATCH_INVARIANT=1 VLLM_ATTENTION_BACKEND=FLASH_ATTN python3 torchtitan/experiments/rl/unified/simple_rl_multiprocess.py

Before (Main)

  File "/home/lucaskabela/.conda/envs/vllm/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/lucaskabela/torchtitan/torchtitan/experiments/rl/unified/simple_rl_multiprocess.py", line 66, in main
    init_batch_invariance()
TypeError: init_batch_invariance() missing 1 required positional argument: 'attention_backend'

And after patching with the enum:

  File "/home/lucaskabela/pytorch/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lucaskabela/pytorch/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: VLLMCompatibleFlashAttention.forward() got an unexpected keyword argument 'enable_gqa'

After

Adding requests: 100%|█████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 2935.95it/s]
Processed prompts: 100%|███████████████| 40/40 [00:01<00:00, 27.61it/s, est. speed input: 193.29 toks/s, output: 552.25 toks/s]
[2026-02-09 16:11:59] INFO generator.py:446: [actor=<root>.<torchtitan.experiments.rl.unified.actors.generator.Generator generator{'gpus': 0/1}>] os.getpid()=300001 Generating finish generate (policy v0)...
[2026-02-09 16:11:59] INFO trainer.py:101: [actor=<root>.<torchtitan.experiments.rl.unified.actors.trainer.Trainer trainer{'gpus': 1/2}>] os.getpid()=300426 Trainer starts to train 0 on traj:
[2026-02-09 16:11:59] INFO trainer.py:101: [actor=<root>.<torchtitan.experiments.rl.unified.actors.trainer.Trainer trainer{'gpus': 0/2}>] os.getpid()=299553 Trainer starts to train 0 on traj:
NCCL version 2.28.9+cuda12.9
  ✓ vLLM-TorchTitan bitwise determinism verified: 20 tokens match exactly
  ✓ vLLM-TorchTitan bitwise determinism verified: 20 tokens match exactly

[acisseJZhong]

are you suggesting self.num_heads might be different from num_heads here?

[acisseJZhong]

nit: can we update to uv pip install as the rest of this doc uses uv?

[acisseJZhong]

Won't vllm handle GQA internally?