Hi, I ran into an issue while training with ServerlessBackend.
Problem
When using ServerlessBackend, each call to backend.train() creates a separate W&B run. This means that if "Run
finished" Slack notifications are enabled in W&B User Settings, a notification is sent on every training step — 50
steps means 50 Slack notifications, which is extremely noisy in practice.
Reproduction
for step in range(NUM_STEPS):
finished = await art.gather_trajectory_groups(groups)
result = await backend.train(model, finished, learning_rate=1e-5)
# ↑ Each call creates and finishes a separate W&B run
Expected behavior
One of the following would address this:
- Consolidate the entire training session (
model.register() through backend.close()) into a single W&B run
- Handle per-step runs in a way that does not trigger W&B alerts (e.g. via
wandb.Settings)
- Provide a user-configurable option to control this behavior
Workaround
Currently the only option is to disable "Run finished" notifications globally in W&B User Settings, but this silences
notifications for all projects — not ideal. W&B does not currently support per-project alert configuration.
Thanks for the great library!