Skip to content

Commit

Permalink
Add configurable beta parameters for adam
Browse files Browse the repository at this point in the history
  • Loading branch information
gobbleturk committed Apr 27, 2023
1 parent df3cd24 commit 7dc4aa3
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion MaxText/configs/base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,8 @@ prompt: "I love to "
enable_profiler: False
enable_checkpointing: True

# Adam optimizer parameters
adam_b1: 0.9 # Exponential decay rate to track the first moment of past gradients.
adam_b2: 0.95 # Exponential decay rate to track the second moment of past gradients.
adam_eps: 1.e-8 # A small constant applied to denominator outside of the square root.
adam_eps_root: 0. # A small constant applied to denominator inside the square root
adam_eps_root: 0. # A small constant applied to denominator inside the square root.

0 comments on commit 7dc4aa3

Please sign in to comment.