[misc,trainer,rollout] feat: add Prometheus metrics logging to experiment tracking

[guillemgt]

What does this PR do?

Adds the ability to query Prometheus metrics and log them to experiment tracking backends (WandB, TensorBoard, MLflow, etc.) during training. This allows users to correlate infrastructure metrics (GPU cache usage, throughput) with training metrics in a unified view.

Checklist Before Starting

• Search for similar PRs: prometheus metrics
• Format the PR title as [{modules}] {type}: {description}

Test

Testing was performed locally/internally during development. The feature has been validated to work correctly with:

• Prometheus HTTP API queries
• Ray head node auto-discovery
• Cache behavior
• Error handling (connection errors, timeouts, malformed responses)
• Graceful failure modes

Due to the complexity of mocking Ray and Prometheus infrastructure, comprehensive unit tests are not included in this PR. The feature can be validated end-to-end by configuring metrics_to_log and verifying metrics appear in experiment tracking backends.

Test

Tested with the example config below. Metric was successfully logged to rollout/vllm_generation_tokens_total in tensorboard over a 20 iteration run on 2 nodes.

API and Usage Example

actor_rollout_ref:
  rollout:
    prometheus:
      enable: True
      port: 9090
      metrics_to_log:
        - "vllm:generation_tokens_total"

Design & Code Changes

1. verl/workers/config/rollout.py: Add metrics_to_log field to PrometheusConfig
2. verl/experimental/agent_loop/prometheus_utils.py: Add PrometheusClient class (~180 lines)
3. verl/trainer/ppo/ray_trainer.py: Initialize client and query metrics before logging

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide
• Apply pre-commit checks
• Add documentation
• Add unit tests - Explained in Test section why tests are not included
• Notify ci-request Slack channel (will do after PR creation)
• Recipe submodule not affected

[gemini-code-assist]

Code Review

This pull request introduces a PrometheusClient to query and log metrics from Prometheus to experiment tracking backends. The implementation is well-structured, including features like Ray head node discovery, caching, and retry logic. However, I've identified two critical issues in the error handling within prometheus_utils.py. One issue in the Ray head node discovery defaults to localhost on any failure, which is problematic in a distributed setting. The other issue is in the metric querying loop, which silently swallows all exceptions. Addressing these will significantly improve the robustness and debuggability of this new feature.