feat: Add debugger flag that can be turned on via RAY_DEBUG=legacy

[guyueh1]

What does this PR do ?

If user set RAY_DEBUG=legacy before running ray.sub, the debug mode will be turned on and user can attach to breakpoints via ray debug.

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

Summary by CodeRabbit

• New Features
  • Legacy debug mode now automatically enables external debugger support for both head and worker start commands. No extra steps are required; the appropriate argument is injected before the blocking startup flag. This streamlines attaching external debuggers during development and troubleshooting. Applies uniformly across nodes, improving parity with legacy tooling and reducing setup friction.

[coderabbitai]

📝 Walkthrough

Walkthrough

Introduces RAY_DEBUGGER_ARGS, set to "--ray-debugger-external" when RAY_DEBUG="legacy", and injects it into the ray start commands for head and worker nodes before the --block flag. Includes minor formatting updates within the same script.

Changes

Cohort / File(s)	Summary of Changes
Ray start script updates ray.sub	Added RAY_DEBUGGER_ARGS conditional on RAY_DEBUG="legacy"; inserted $RAY_DEBUGGER_ARGS into both head and worker ray start commands before --block; minor formatting adjustments.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User
  participant Script as ray.sub
  participant Ray as ray start

  User->>Script: Invoke ray start (head/worker)
  Script->>Script: Check RAY_DEBUG
  alt RAY_DEBUG="legacy"
    Script->>Script: Set RAY_DEBUGGER_ARGS="--ray-debugger-external"
  else
    Script->>Script: Set RAY_DEBUGGER_ARGS=""
  end
  Script->>Ray: Start with args + $RAY_DEBUGGER_ARGS + --block
  Ray-->>User: Process runs (blocked)

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Pre-merge checks and finishing touches

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly and accurately describes the primary change by noting the addition of a debugger flag toggled via RAY_DEBUG=legacy, making it clear and immediately informative for reviewers.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.
Test Results For Major Changes	✅ Passed	The PR only introduces a conditional debugger flag in a shell script, which is a small, non-breaking tweak without broader numeric or performance implications, so it does not qualify as a major change that would mandate documented test results. Given the limited scope, the absence of explicit testing information in the description is acceptable under the check’s guidance. Therefore the custom check passes.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

ray.sub (1)

67-70: Quote RAY_DEBUG in the condition and rely on existing docs
Documentation for RAY_DEBUG=legacy already exists in docs/debugging.md. Guard against unset variables by quoting in the condition:

- if [ $RAY_DEBUG = "legacy" ]; then
+ if [ "$RAY_DEBUG" = "legacy" ]; then

Optionally set a default or emit a warning when RAY_DEBUG is unset.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 57046a4 and 9fe5d7b.

📒 Files selected for processing (1)

• ray.sub (3 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)

• GitHub Check: sphinx-build / Build docs
• GitHub Check: Lint check
• GitHub Check: Lint check
• GitHub Check: Post submodule check comment / Comment on PR
• GitHub Check: Post automodel integration comment / Comment on PR

🔇 Additional comments (2)

ray.sub (2)

283-285: LGTM: Debugger args correctly positioned before --block.

The $RAY_DEBUGGER_ARGS is correctly inserted before the --block flag, ensuring proper command structure for the head node.

Note: Line 285 has a trailing space after --block, which is a minor formatting issue but does not affect functionality.

---

383-385: LGTM: Debugger args correctly positioned before --block.

The $RAY_DEBUGGER_ARGS is correctly inserted before the --block flag for worker nodes, maintaining consistency with the head node configuration.

Note: Line 385 has a trailing space after --block, which is a minor formatting issue but does not affect functionality.

[terrykong]

@guyueh1 for my education, what does this change enable that you couldn't do with https://docs.nvidia.com/nemo/rl/latest/debugging.html ?

[guyueh1]

@terrykong this is an alternative way, won't enable more than the vscode debugger; it is just a way to run ray debug even without IDE. The way to debug is

1. Set RAY_DEBUG=legacy before starting the job (ray.sub)
2. Add breakpoint() to your code
3. When the job has started, use another terminal to connect to the header node via bash <job-id>-attach.sh and run ray debug interactively. You will be able to see and attach to the breakpoint and use PDB-style commands to debug.

[terrykong]

oh interesting, so before when i set RAY_DEBUG=legacy uv run examples/run_grpo_math.py i was only able to get a breakpoint into the driver and needed to use the distributed debugger for breakpointing into actors/tasks, but you're saying when adding this to the head/worker startup args, you can breakpoint into actors as well?

if so, that's awesome. do you think you could update https://github.com/NVIDIA-NeMo/RL/blob/main/docs/debugging.md with this new method?

[guyueh1]

@terrykong yes, you need to set both RAY_DEBUG=legacy and --ray-debugger-external , then you can breakpoint into actors from header in a separate terminal that spins ray debug (also refer to verl's doc)

I have updated our doc.

[terrykong]

@jgerh to review

[jgerh]

Completed tech pubs review and left a few copyedits and suggested text revisions.

[guyueh1]

@terrykong i think this is ready for merge, if L0 test is enough