diff --git a/cs8850_15_SGD.html b/cs8850_15_SGD.html index 13071861..17bac2c6 100644 --- a/cs8850_15_SGD.html +++ b/cs8850_15_SGD.html @@ -5,7 +5,7 @@ - + @@ -403,6 +403,13 @@

Challenges

  • How to avoid local minima and saddle points +
    +

    Batch?

    + +

    Gradient descent optimization algorithms

    @@ -484,7 +491,7 @@

    RMSprop:

    Adadelta: - get everything right

    + get everything right

    \begin{align*} \vec{g}_t & = \nabla_{\theta} \prob{J}{\vec{\theta}} \\ @@ -529,6 +536,22 @@

    but use Adam if in doubt

    +
    +

    Momentum?

    + +
    + +
    + +
    + +

    The rise of the SGD

    Ravines