Skip to content

ServerlessBackend creates a new W&B run per training step, causing "Run finished" alerts to fire on every step #638

@go5paopao

Description

@go5paopao

Hi, I ran into an issue while training with ServerlessBackend.

Problem

When using ServerlessBackend, each call to backend.train() creates a separate W&B run. This means that if "Run
finished" Slack notifications are enabled in W&B User Settings, a notification is sent on every training step — 50
steps means 50 Slack notifications, which is extremely noisy in practice.

Reproduction

for step in range(NUM_STEPS):
    finished = await art.gather_trajectory_groups(groups)
    result = await backend.train(model, finished, learning_rate=1e-5)
    # ↑ Each call creates and finishes a separate W&B run

Expected behavior

One of the following would address this:

  • Consolidate the entire training session (model.register() through backend.close()) into a single W&B run
  • Handle per-step runs in a way that does not trigger W&B alerts (e.g. via wandb.Settings)
  • Provide a user-configurable option to control this behavior

Workaround

Currently the only option is to disable "Run finished" notifications globally in W&B User Settings, but this silences
notifications for all projects — not ideal. W&B does not currently support per-project alert configuration.

Thanks for the great library!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions