total_steps for using OneCycleLR lr scheduler

[exiawsh]

Hello, I have some question about the pytorch official One_cycle scheduler when using the pytorch lightning.
When using the One_cycle scheduler, I need provide the parameter total_steps.
Are the total_steps include the validation steps or only include training steps?
That's mean, if I have 3000 steps for training per epoch, and 2000 steps for validation per epoch, and the total epochs are 24.
I should give the parameter with 3000*24 or 5000*24?

[akihironitta]

@exiawsh It should be 3000*24. Are you seeing an issue around it?

Also, there's a useful property in the Trainer you might be interested in:

def configure_optimizers(self):
    optimizer = ...
    scheduler = torch.optim.lr_scheduler.OneCycleLR(
        optimizer, max_lr=1e-3, total_steps=self.trainer.estimated_stepping_batches
    )
    return [optimizer], [scheduler]

https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.trainer.trainer.Trainer.html#pytorch_lightning.trainer.trainer.Trainer.estimated_stepping_batches

[exiawsh]

And also how about using accumulate_grad_batches when using the OneCycleLR?

[exiawsh]

because i got something wrong with self.trainer.estimated_steeping_batches.

[akihironitta]

And I have another question about the amp training.

Let's not discuss it here as it's not related to your original question. Would you mind creating a new GitHub Discussion and asking your question there?

---

And also how about using accumulate_grad_batches when using the OneCycleLR?
because i got something wrong with self.trainer.estimated_steeping_batches.

Looks similar to #12317. Could you check if yours is the same as the issue?

• If yes, 1.6.4 has the patch included, so please try again with the new version (https://github.com/PyTorchLightning/pytorch-lightning/releases/tag/1.6.4).
• If it looks like another issue, would you mind creating a new issue with a minimal script using the BoringModel? https://github.com/PyTorchLightning/pytorch-lightning/tree/master/pl_examples/bug_report

[exiawsh]

Ah thank you! It's the same issue, I will update it later. For the amp training problem I will open a new issue. Your patience is greatly appreciated.

[JackCaster]

Hey @akihironitta I stumbled on this old issue while looking on how to set up Onecyclelr. Shouldn’t the interval of the scheduler be set to “step” (as suggested here #3051 (comment))? By default it is set to “epoch”. If so, the example in the docs https://lightning.ai/docs/pytorch/stable/api/pytorch_lightning.trainer.trainer.Trainer.html#lightning.pytorch.trainer.trainer.Trainer.estimated_stepping_batches should also be fixed.