Weight initialization.md

Weight initialization. When you instantiate a [[neural network]], you have to consider what the initial weight values are.

Example. Xavier initialization uses a normal distribution with mean 0 and variance $\frac{1}{d}$ where $d$ is the number of incoming connections.

Example. He/Kaiming initialization is used for [[ReLU]] and uses a normal distribution with mean 0 and variance $\frac{2}{d}$.