[MLP] network gets confused

[amaneureka]

In a long run, sometimes the network gets confused and plays lame from next game.

[amaneureka]

Possible solution: Exponentially decrease learning rate proportional to confidence of network.

[yask123]

Why does it get confused ?

[amaneureka]

Solution itself explains it.
It's not good idea to move saturated weights with small momentum when they are already saturated.
Hence it should decrease it's learning rate as network get confident on its decision.