Optimizer choices?

[Akamight]

In both of the jupyter notebooks and the paper, I noticed that instead of using Adam, the most commonly used optimizer for transformers, you used Adagrad for all of the experiments. Is there a reason behind this or simply a empirical observation?

Additionally, are other newly developed optimizers (RAdam, NovoGrad, DiffGrad, etc.) compatible with the method introduced or do they defeat the purpose?

[tbachlechner]

Thank you for your question! In our paper we use the LAMB optimizer (and hence Adam) for the Transformer experiments- we should say this in the main text, not just the appendix. There is no particular reason to use Adagrad in ReZero. We merely used Adagrad in the comparison experiments because it is robust against divergence for very deep networks that do not use ReZero.

The choice of optimizer is somewhat orthogonal to the ReZero contribution. ReZero ensures that at initialization an update of any weight will give rise to a reasonable change in the network output and hence finite gradients. This is a necessary condition for any gradient based optimizer to work.

Does this answer your question?

[Akamight]

Yes! Thank you!

[hitvoice]

By the way, which pytorch implementation of LAMB did you use in the experiments of the paper? Unlike Adam, it's not in the "torch.optim". Would you please share the link to the LAMB implementation you referred to?

[Akamight]

By the way, which pytorch implementation of LAMB did you use in the experiments of the paper? Unlike Adam, it's not in the "torch.optim". Would you please share the link to the LAMB implementation you referred to?

I found the one in the pytorch-optimizer pip library to be quite good with ReZero. Other methods, such as AdaMod, AdaBound and RAdam also works well with this method.

I would suggest you to try some of the methods in that library and see if they work well as a baseline. Then, if you are interested, changing the optimizer to CProp with Adam yielded around 3-4% of increase in perplexity compared to RAdam in many applications (NMT, audio applications, etc.) and acceleration in training without the hassle of fine tuning lr decay hyperparameter. Changing the activation function to Mish also speeded up the training by another 30% or so, and also yielded a better result.

[tbachlechner]

Thanks for those comments!

This is the LAMB opt we used in our experiments.

[hitvoice]

Wow, thank you so much for your detailed explanations!!! I'll try these options you mentioned and see how they perform.