lecture2.md

class: middle, center, title-slide

Deep Learning

Lecture 2: Multi-layer perceptron

Prof. Gilles Louppe
g.louppe@uliege.be

---

Today

Explain and motivate the basic constructs of neural networks.

• From linear discriminant analysis to logistic regression
• Stochastic gradient descent
• From logistic regression to the multi-layer perceptron
• Vanishing gradients and rectified networks
• Universal approximation theorem

---

class: middle

Neural networks

---

Threshold Logic Unit

.grid[ .kol-3-5[ The Threshold Logic Unit (McCulloch and Pitts, 1943) $$f(\mathbf{x}) = 1_{\{\sum_i w_i x_i + b \geq 0\}},$$ with Boolean inputs $x_i$, weights $w_i$ and bias $b$, was the first mathematical model for a neuron.

This unit can implement

• $\text{or}(a,b) = 1_{\{a+b - 0.5 \geq 0\}}$,
• $\text{and}(a,b) = 1_{\{a+b - 1.5 \geq 0\}}$,
• $\text{not}(a) = 1_{\{-a + 0.5 \geq 0\}}$.

Therefore, any Boolean function can be built with such units. ] .kol-2-5.width-100[] ]

.footnote[Credits: McCulloch and Pitts, A logical calculus of ideas immanent in nervous activity, 1943.]

---

Perceptron

The perceptron (Rosenblatt, 1957) $$f(\mathbf{x}) = \begin{cases} 1 &\text{if } \sum_i w_i x_i + b \geq 0 \\ 0 &\text{otherwise} \end{cases}$$ is very similar, except that the inputs are real.

This model was originally motivated by biology, with $w_i$ being synaptic weights and $x_i$ and $f$ firing rates. .center.width-65[]

.footnote[Credits: Frank Rosenblatt, Mark I Perceptron operators' manual, 1960.]

???

A perceptron is a signal transmission network consisting of sensory units (S units), association units (A units), and output or response units (R units). The ‘retina’ of the perceptron is an array of sensory elements (photocells). An S-unit produces a binary output depending on whether or not it is excited. A randomly selected set of retinal cells is connected to the next level of the network, the A units. As originally proposed there were extensive connections among the A units, the R units, and feedback between the R units and the A units.

In essence an association unit is also an MCP neuron which is 1 if a single specific pattern of inputs is received, and it is 0 for all other possible patterns of inputs. Each association unit will have a certain number of inputs which are selected from all the inputs to the perceptron. So the number of inputs to a particular association unit does not have to be the same as the total number of inputs to the perceptron, but clearly the number of inputs to an association unit must be less than or equal to the total number of inputs to the perceptron. Each association unit's output then becomes the input to a single MCP neuron, and the output from this single MCP neuron is the output of the perceptron. So a perceptron consists of a "layer" of MCP neurons, and all of these neurons send their output to a single MCP neuron.

---

class: middle, center, black-slide

.grid[ .kol-1-2[.width-100[]] .kol-1-2[

.width-100[]] ]

The Mark I Percetron (Frank Rosenblatt).

---

class: middle, center, black-slide

<iframe width="600" height="450" src="https://www.youtube.com/embed/cNxadbrN_aI" frameborder="0" allowfullscreen></iframe>

The Perceptron

---

class: middle

Let us define the (non-linear) activation function:

$$\text{sign}(x) = \begin{cases} 1 &\text{if } x \geq 0 \\ 0 &\text{otherwise} \end{cases}$$ .center[]

The perceptron classification rule can be rewritten as $$f(\mathbf{x}) = \text{sign}(\sum_i w_i x_i + b).$$

---

class: middle

Computational graphs

.grid[ .kol-3-5[.width-90[]] .kol-2-5[ The computation of $$f(\mathbf{x}) = \text{sign}(\sum_i w_i x_i + b)$$ can be represented as a computational graph where

• white nodes correspond to inputs and outputs;
• red nodes correspond to model parameters;
• blue nodes correspond to intermediate operations. ] ]

???

Draw the NN diagram.

---

class: middle

In terms of tensor operations, $f$ can be rewritten as $$f(\mathbf{x}) = \text{sign}(\mathbf{w}^T \mathbf{x} + b),$$ for which the corresponding computational graph of $f$ is:

.center.width-70[]

???

Ask about their intuition on the intuitive meaning of $f(x)$ (i.e., the product as a similarity measure).

---

Linear discriminant analysis

Consider training data $(\mathbf{x}, y) \sim p_{X,Y}$, with

• $\mathbf{x} \in \mathbb{R}^p$,
• $y \in \{0,1\}$.

Assume class populations are Gaussian, with same covariance matrix $\Sigma$ (homoscedasticity):

$$p(\mathbf{x}|y) = \frac{1}{\sqrt{(2\pi)^p |\Sigma|}} \exp \left(-\frac{1}{2}(\mathbf{x} - \mathbf{\mu}_y)^T \Sigma^{-1}(\mathbf{x} - \mathbf{\mu}_y) \right)$$

???

Switch to blackboard.

---

Using the Bayes' rule, we have:

$$\begin{aligned} p(y=1|\mathbf{x}) &= \frac{p(\mathbf{x}|y=1) p(y=1)}{p(\mathbf{x})} \\\ &= \frac{p(\mathbf{x}|y=1) p(y=1)}{p(\mathbf{x}|y=0)p(y=0) + p(\mathbf{x}|y=1)p(y=1)} \\\ &= \frac{1}{1 + \frac{p(\mathbf{x}|y=0)p(y=0)}{p(\mathbf{x}|y=1)p(y=1)}}. \end{aligned}$$

--

count: false

It follows that with

$$\sigma(x) = \frac{1}{1 + \exp(-x)},$$

we get

$$p(y=1|\mathbf{x}) = \sigma\left(\log \frac{p(\mathbf{x}|y=1)}{p(\mathbf{x}|y=0)} + \log \frac{p(y=1)}{p(y=0)}\right).$$

---

class: middle

Therefore,

$$\begin{aligned} &p(y=1|\mathbf{x}) \\\ &= \sigma\left(\log \frac{p(\mathbf{x}|y=1)}{p(\mathbf{x}|y=0)} + \underbrace{\log \frac{p(y=1)}{p(y=0)}}_{a}\right) \\\ &= \sigma\left(\log p(\mathbf{x}|y=1) - \log p(\mathbf{x}|y=0) + a\right) \\\ &= \sigma\left(-\frac{1}{2}(\mathbf{x} - \mathbf{\mu}_1)^T \Sigma^{-1}(\mathbf{x} - \mathbf{\mu}_1) + \frac{1}{2}(\mathbf{x} - \mathbf{\mu}_0)^T \Sigma^{-1}(\mathbf{x} - \mathbf{\mu}_0) + a\right) \\\ &= \sigma\left(\underbrace{(\mu_1-\mu_0)^T \Sigma^{-1}}_{\mathbf{w}^T}\mathbf{x} + \underbrace{\frac{1}{2}(\mu_0^T \Sigma^{-1} \mu_0 - \mu_1^T \Sigma^{-1} \mu_1) + a}_{b} \right) \\\ &= \sigma\left(\mathbf{w}^T \mathbf{x} + b\right) \end{aligned}$$

---

class: middle, center

.width-100[]

---

count: false class: middle, center

.width-100[]

---

count: false class: middle, center

.width-100[]

---

class: middle

Note that the sigmoid function $$\sigma(x) = \frac{1}{1 + \exp(-x)}$$ looks like a soft heavyside:

.center[]

Therefore, the overall model $f(\mathbf{x};\mathbf{w},b) = \sigma(\mathbf{w}^T \mathbf{x} + b)$ is very similar to the perceptron.

---

class: middle, center

.center.width-70[]

This unit is the main primitive of all neural networks!

---

Logistic regression

Same model $$p(y=1|\mathbf{x}) = \sigma\left(\mathbf{w}^T \mathbf{x} + b\right)$$ as for linear discriminant analysis.

But,

• ignore model assumptions (Gaussian class populations, homoscedasticity);
• instead, find $\mathbf{w}, b$ that maximizes the likelihood of the data.

???

Switch to blackboard.

---

class: middle

We have,

$$\begin{aligned} &\arg \max_{\mathbf{w},b} p(\mathbf{d}|\mathbf{w},b) \\\ &= \arg \max_{\mathbf{w},b} \prod_{\mathbf{x}_i, y_i \in \mathbf{d}} p(y=y_i|\mathbf{x}_i, \mathbf{w},b) \\\ &= \arg \max_{\mathbf{w},b} \prod_{\mathbf{x}_i, y_i \in \mathbf{d}} \sigma(\mathbf{w}^T \mathbf{x}_i + b)^{y_i} (1-\sigma(\mathbf{w}^T \mathbf{x}_i + b))^{1-y_i} \\\ &= \arg \min_{\mathbf{w},b} \underbrace{\sum_{\mathbf{x}_i, y_i \in \mathbf{d}} -{y_i} \log\sigma(\mathbf{w}^T \mathbf{x}_i + b) - {(1-y_i)} \log (1-\sigma(\mathbf{w}^T \mathbf{x}_i + b))}_{\mathcal{L}(\mathbf{w}, b) = \sum_i \ell(y_i, \hat{y}(\mathbf{x}_i; \mathbf{w}, b))} \end{aligned}$$

This loss is an instance of the cross-entropy $$H(p,q) = \mathbb{E}_p[-\log q]$$ for $p=p_{Y|\mathbf{x}_i}$ and $q=p_{\hat{Y}|\mathbf{x}_i}$.

---

class: middle

When $Y$ takes values in $\{-1,1\}$, a similar derivation yields the logistic loss $$\mathcal{L}(\mathbf{w}, b) = -\sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \log \sigma\left(y_i (\mathbf{w}^T \mathbf{x}_i + b))\right).$$

.center[]

---

Multi-layer perceptron

So far we considered the logistic unit $h=\sigma\left(\mathbf{w}^T \mathbf{x} + b\right)$, where $h \in \mathbb{R}$, $\mathbf{x} \in \mathbb{R}^p$, $\mathbf{w} \in \mathbb{R}^p$ and $b \in \mathbb{R}$.

These units can be composed in parallel to form a layer with $q$ outputs: $$\mathbf{h} = \sigma(\mathbf{W}^T \mathbf{x} + \mathbf{b})$$ where $\mathbf{h} \in \mathbb{R}^q$, $\mathbf{x} \in \mathbb{R}^p$, $\mathbf{W} \in \mathbb{R}^{p\times q}$, $b \in \mathbb{R}^q$ and where $\sigma(\cdot)$ is upgraded to the element-wise sigmoid function.

.center.width-70[]

???

Draw the NN diagram.

---

class: middle

Similarly, layers can be composed in series, such that: $$\begin{aligned} \mathbf{h}_0 &= \mathbf{x} \\ \mathbf{h}_1 &= \sigma(\mathbf{W}_1^T \mathbf{h}_0 + \mathbf{b}_1) \\ ... \\ \mathbf{h}_L &= \sigma(\mathbf{W}_L^T \mathbf{h}_{L-1} + \mathbf{b}_L) \\ f(\mathbf{x}; \theta) = \hat{y} &= \mathbf{h}_L \end{aligned}$$ where $\theta$ denotes the model parameters $\{ \mathbf{W}_k, \mathbf{b}_k, ... | k=1, ..., L\}$.

This model is the multi-layer perceptron, also known as the fully connected feedforward network.

???

Draw the NN diagram.

---

class: middle, center

.width-100[]

---

class: middle

Output layer

• For binary classification, the width $q$ of the last layer $L$ is set to $1$, which results in a single output $h_L \in [0,1]$ that models the probability $p(y=1|\mathbf{x})$.
• For multi-class classification, the sigmoid activation $\sigma$ in the last layer can be generalized to produce a vector $\mathbf{h}_L \in \bigtriangleup^C$ of probability estimates $p(y=i|\mathbf{x})$.
  
  This activation is the $\text{Softmax}$ function, where its $i$-th output is defined as $$\text{Softmax}(\mathbf{z})_i = \frac{\exp(z_i)}{\sum_{j=1}^C \exp(z_j)},$$ for $i=1, ..., C$.

---

Regression

For regression problems, one usually starts with the assumption that $$p(y|\mathbf{x}) = \mathcal{N}(y; \mu=f(\mathbf{x}; \theta), \sigma^2=1),$$ where $f$ is parameterized with a neural network which last layer does not contain any final activation.

---

class: middle

We have, $$\begin{aligned} &\arg \max_{\theta} p(\mathbf{d}|\theta) \\ &= \arg \max_{\theta} \prod_{\mathbf{x}_i, y_i \in \mathbf{d}} p(y=y_i|\mathbf{x}_i, \theta) \\ &= \arg \min_{\theta} -\sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \log p(y=y_i|\mathbf{x}_i, \theta) \\ &= \arg \min_{\theta} -\sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \log\left( \frac{1}{\sqrt{2\pi}} \exp(-\frac{1}{2}(y_i - f(\mathbf{x};\theta))^2) \right)\\ &= \arg \min_{\theta} \sum_{\mathbf{x}_i, y_i \in \mathbf{d}} (y_i - f(\mathbf{x};\theta))^2, \end{aligned}$$ which recovers the common squared error loss $\ell(y, \hat{y}) = (y-\hat{y})^2$.

---

class: middle, center

(demo)

---

Training neural networks

• In general, the loss functions do not admit a minimizer that can be expressed analytically in closed form.
• However, a minimizer can be found numerically, using a general minimization technique such as gradient descent.

---

class: middle

Gradient descent

Let $\mathcal{L}(\theta)$ denote a loss function defined over model parameters $\theta$ (e.g., $\mathbf{w}$ and $b$).

To minimize $\mathcal{L}(\theta)$, gradient descent uses local linear information to iteratively move towards a (local) minimum.

For $\theta_0 \in \mathbb{R}^d$, a first-order approximation around $\theta_0$ can be defined as $$\hat{\mathcal{L}}(\epsilon; \theta_0) = \mathcal{L}(\theta_0) + \epsilon^T\nabla_\theta \mathcal{L}(\theta_0) + \frac{1}{2\gamma}||\epsilon||^2.$$

.center.width-60[]

???

Switch to blackboard.

---

class: middle

A minimizer of the approximation $\hat{\mathcal{L}}(\epsilon; \theta_0)$ is given for $$\begin{aligned} \nabla_\epsilon \hat{\mathcal{L}}(\epsilon; \theta_0) &= 0 \\ &= \nabla_\theta \mathcal{L}(\theta_0) + \frac{1}{\gamma} \epsilon, \end{aligned}$$ which results in the best improvement for the step $\epsilon = -\gamma \nabla_\theta \mathcal{L}(\theta_0)$.

Therefore, model parameters can be updated iteratively using the update rule $$\theta_{t+1} = \theta_t -\gamma \nabla_\theta \mathcal{L}(\theta_t),$$ where

• $\theta_0$ are the initial parameters of the model;
• $\gamma$ is the learning rate;
• both are critical for the convergence of the update rule.

---

class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

count: false class: center, middle

Example 1: Convergence to a local minima

---

class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

count: false class: center, middle

Example 2: Convergence to the global minima

---

class: center, middle

Example 3: Divergence due to a too large learning rate

---

count: false class: center, middle

Example 3: Divergence due to a too large learning rate

---

count: false class: center, middle

Example 3: Divergence due to a too large learning rate

---

count: false class: center, middle

Example 3: Divergence due to a too large learning rate

---

count: false class: center, middle

Example 3: Divergence due to a too large learning rate

---

count: false class: center, middle

Example 3: Divergence due to a too large learning rate

---

Stochastic gradient descent

In the empirical risk minimization setup, $\mathcal{L}(\theta)$ and its gradient decompose as $$\begin{aligned} \mathcal{L}(\theta) &= \frac{1}{N} \sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \ell(y_i, f(\mathbf{x}_i; \theta)) \\ \nabla \mathcal{L}(\theta) &= \frac{1}{N} \sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \nabla \ell(y_i, f(\mathbf{x}_i; \theta)). \end{aligned}$$ Therefore, in batch gradient descent the complexity of an update grows linearly with the size $N$ of the dataset. This is bad!

---

class: middle

Since the empirical risk is already an approximation of the expected risk, it should not be necessary to carry out the minimization with great accuracy.

---

Instead, stochastic gradient descent uses as update rule: $$\theta_{t+1} = \theta_t - \gamma \nabla \ell(y_{i(t+1)}, f(\mathbf{x}_{i(t+1)}; \theta_t))$$

• Iteration complexity is independent of $N$.
• The stochastic process $\{ \theta_t | t=1, ... \}$ depends on the examples $i(t)$ picked randomly at each iteration.

--

.grid.center.italic[ .kol-1-2[.width-100[]

Batch gradient descent] .kol-1-2[.width-100[]

Stochastic gradient descent ] ]

---

class: middle

Why is stochastic gradient descent still a good idea?

• Informally, averaging the update $$\theta_{t+1} = \theta_t - \gamma \nabla \ell(y_{i(t+1)}, f(\mathbf{x}_{i(t+1)}; \theta_t)) $$ over all choices $i(t+1)$ restores batch gradient descent.
• Formally, if the gradient estimate is unbiased, e.g., if $$\begin{aligned} \mathbb{E}_{i(t+1)}[\nabla \ell(y_{i(t+1)}, f(\mathbf{x}_{i(t+1)}; \theta_t))] &= \frac{1}{N} \sum_{\mathbf{x}_i, y_i \in \mathbf{d}} \nabla \ell(y_i, f(\mathbf{x}_i; \theta_t)) \\ &= \nabla \mathcal{L}(\theta_t) \end{aligned}$$ then the formal convergence of SGD can be proved, under appropriate assumptions (see references).
• If training is limited to single pass over the data, then SGD directly minimizes the expected risk.

---

class: middle

The excess error characterizes the expected risk discrepancy between the Bayes model and the approximate empirical risk minimizer. It can be decomposed as $$\begin{aligned} &\mathbb{E}\left[ R(\tilde{f}_*^\mathbf{d}) - R(f_B) \right] \\ &= \mathbb{E}\left[ R(f_*) - R(f_B) \right] + \mathbb{E}\left[ R(f_*^\mathbf{d}) - R(f_*) \right] + \mathbb{E}\left[ R(\tilde{f}_*^\mathbf{d}) - R(f_*^\mathbf{d}) \right] \\ &= \mathcal{E}_\text{app} + \mathcal{E}_\text{est} + \mathcal{E}_\text{opt} \end{aligned}$$ where

• $\mathcal{E}_\text{app}$ is the approximation error due to the choice of an hypothesis space,
• $\mathcal{E}_\text{est}$ is the estimation error due to the empirical risk minimization principle,
• $\mathcal{E}_\text{opt}$ is the optimization error due to the approximate optimization algorithm.

---

class: middle

A fundamental result due to Bottou and Bousquet (2011) states that stochastic optimization algorithms (e.g., SGD) yield the best generalization performance (in terms of excess error) despite being the worst optimization algorithms for minimizing the empirical risk.

---

Automatic differentiation (teaser)

To minimize $\mathcal{L}(\theta)$ with stochastic gradient descent, we need the gradient $$\nabla \mathcal{\ell}(\theta) = \begin{bmatrix} \frac{\partial \mathcal{\ell}}{\partial \theta_0}(\theta) \\ \\ \vdots \\ \\ \frac{\partial \mathcal{\ell}}{\partial \theta_{K-1}}(\theta) \end{bmatrix} $$ i.e., a vector that gathers the partial derivatives of the loss for each model parameter $\theta_k$ for $k=0, \ldots, K-1$.

These derivatives can be evaluated automatically from the computational graph of $\ell$ using automatic differentiation.

---

class: middle

In Leibniz notations, the chain rule states that $$ \begin{aligned} \frac{\partial \ell}{\partial \theta_i} &= \sum_{k \in \text{parents}(\ell)} \frac{\partial \ell}{\partial u_k} \underbrace{\frac{\partial u_k}{\partial \theta_i}}_{\text{recursive case}} \end{aligned}$$

---

class: middle

Backpropagation

• Since a neural network is a composition of differentiable functions, the total derivatives of the loss can be evaluated backward, by applying the chain rule recursively over its computational graph.
• The implementation of this procedure is called reverse automatic differentiation or backpropagation.

---

class: middle

Let us consider a simplified 1-hidden layer MLP and the following loss function: $$\begin{aligned} f(\mathbf{x}; \mathbf{W}_1, \mathbf{W}_2) &= \sigma\left( \mathbf{W}_2^T \sigma\left( \mathbf{W}_1^T \mathbf{x} \right)\right) \\ \mathcal{\ell}(y, \hat{y}; \mathbf{W}_1, \mathbf{W}_2) &= \text{cross\_ent}(y, \hat{y}) + \lambda \left( ||\mathbf{W}_1||_2 + ||\mathbf{W}_2||_2 \right) \end{aligned}$$ for $\mathbf{x} \in \mathbb{R^p}$, $y \in \mathbb{R}$, $\mathbf{W}_1 \in \mathbb{R}^{p \times q}$ and $\mathbf{W}_2 \in \mathbb{R}^q$.

---

class: middle

In the forward pass, intermediate values are all computed from inputs to outputs, which results in the annotated computational graph below:

.width-100[]

---

class: middle

The partial derivatives can be computed through a backward pass, by walking through all paths from outputs to parameters in the computational graph and accumulating the terms. For example, for $\frac{\partial \ell}{\partial \mathbf{W}_1}$ we have: $$\begin{aligned} \frac{\partial \ell}{\partial \mathbf{W}_1} &= \frac{\partial \ell}{\partial u_8}\frac{\partial u_8}{\partial \mathbf{W}_1} + \frac{\partial \ell}{\partial u_4}\frac{\partial u_4}{\partial \mathbf{W}_1} \\ \frac{\partial u_8}{\partial \mathbf{W}_1} &= ... \end{aligned}$$

.width-100[]

---

class: middle

.width-100[]

Let us zoom in on the computation of the network output $\hat{y}$ and of its derivative with respect to $\mathbf{W}_1$.

• Forward pass: values $u_1$, $u_2$, $u_3$ and $\hat{y}$ are computed by traversing the graph from inputs to outputs given $\mathbf{x}$, $\mathbf{W}_1$ and $\mathbf{W}_2$.
• Backward pass: by the chain rule we have $$\begin{aligned} \frac{\partial \hat{y}}{\partial \mathbf{W}_1} &= \frac{\partial \hat{y}}{\partial u_3} \frac{\partial u_3}{\partial u_2} \frac{\partial u_2}{\partial u_1} \frac{\partial u_1}{\partial \mathbf{W}_1} \\ &= \frac{\partial \sigma(u_3)}{\partial u_3} \frac{\partial \mathbf{W}_2^T u_2}{\partial u_2} \frac{\partial \sigma(u_1)}{\partial u_1} \frac{\partial \mathbf{W}_1^T \mathbf{x}}{\partial \mathbf{W}_1} \end{aligned}$$ Note how evaluating the partial derivatives requires the intermediate values computed forward.

---

Vanishing gradients

Training deep MLPs with many layers has for long (pre-2011) been very difficult due to the vanishing gradient problem.

• Small gradients slow down, and eventually block, stochastic gradient descent.
• This results in a limited capacity of learning.

.width-100[] .caption[Backpropagated gradients normalized histograms (Glorot and Bengio, 2010).
Gradients for layers far from the output vanish to zero. ]

---

class: middle

Let us consider a simplified 2-hidden layer MLP, with $x, w_1, w_2, w_3 \in\mathbb{R}$, such that $$f(x; w_1, w_2, w_3) = \sigma\left(w_3\sigma\left( w_2 \sigma\left( w_1 x \right)\right)\right). $$

Under the hood, this would be evaluated as $$\begin{aligned} u_1 &= w_1 x \\ u_2 &= \sigma(u_1) \\ u_3 &= w_2 u_2 \\ u_4 &= \sigma(u_3) \\ u_5 &= w_3 u_4 \\ \hat{y} &= \sigma(u_5) \end{aligned}$$ and its derivative $\frac{\partial\hat{y}}{\partial w_1}$ as $$\begin{aligned}\frac{\partial\hat{y}}{\partial w_1} &= \frac{\partial \hat{y}}{\partial u_5} \frac{\partial u_5}{\partial u_4} \frac{\partial u_4}{\partial u_3} \frac{\partial u_3}{\partial u_2}\frac{\partial u_2}{\partial u_1}\frac{\partial u_1}{\partial w_1}\\ &= \frac{\partial \sigma(u_5)}{\partial u_5} w_3 \frac{\partial \sigma(u_3)}{\partial u_3} w_2 \frac{\partial \sigma(u_1)}{\partial u_1} x \end{aligned}$$

---

class: middle

The derivative of the sigmoid activation function $\sigma$ is:

.center[]

$$\frac{\partial \sigma}{\partial x}(x) = \sigma(x)(1-\sigma(x))$$

Notice that $0 \leq \frac{\partial \sigma}{\partial x}(x) \leq \frac{1}{4}$ for all $x$.

---

class: middle

Assume that weights $w_1, w_2, w_3$ are initialized randomly from a Gaussian with zero-mean and small variance, such that with high probability $-1 \leq w_i \leq 1$.

Then,

$$\frac{\partial \hat{y}}{\partial w_1} = \underbrace{\frac{\partial \sigma(u_5)}{\partial u_5}}_{\leq \frac{1}{4}} \underbrace{w_3}_{\leq 1} \underbrace{\frac{\partial \sigma(u_3)}{\partial u_3}}_{\leq \frac{1}{4}} \underbrace{w_2}_{\leq 1} \underbrace{\frac{\sigma(u_1)}{\partial u_1}}_{\leq \frac{1}{4}} x$$

This implies that the derivative $\frac{\partial \hat{y}}{\partial w_1}$ exponentially shrinks to zero as the number of layers in the network increases.

Hence the vanishing gradient problem.

• In general, bounded activation functions (sigmoid, tanh, etc) are prone to the vanishing gradient problem.
• Note the importance of a proper initialization scheme.

---

Rectified linear units

Instead of the sigmoid activation function, modern neural networks are for most based on rectified linear units (ReLU) (Glorot et al, 2011):

$$\text{ReLU}(x) = \max(0, x)$$

.center[]

---

class: middle

Note that the derivative of the ReLU function is

$$\frac{\partial }{\partial x} \text{ReLU}(x) = \begin{cases} 0 &\text{if } x \leq 0 \\ 1 &\text{otherwise} \end{cases}$$ .center[]

For $x=0$, the derivative is undefined. In practice, it is set to zero.

---

class: middle

Therefore,

$$\frac{\partial \hat{y}}{\partial w_1} = \underbrace{\frac{\partial \sigma(u_5)}{\partial u_5}}_{= 1} w_3 \underbrace{\frac{\partial \sigma(u_3)}{\partial u_3}}_{= 1} w_2 \underbrace{\frac{\partial \sigma(u_1)}{\partial u_1}}_{= 1} x$$

This solves the vanishing gradient problem, even for deep networks! (provided proper initialization)

Note that:

• The ReLU unit dies when its input is negative, which might block gradient descent.
• This is actually a useful property to induce sparsity.
• This issue can also be solved using leaky ReLUs, defined as $$\text{LeakyReLU}(x) = \max(\alpha x, x)$$ for a small $\alpha \in \mathbb{R}^+$ (e.g., $\alpha=0.1$).

---

Activation functions

.center[

]

---

class: middle, center

(demo)

???

Don't forget the magic trick!

---

Universal approximation

Let us consider the 1-hidden layer MLP $$f(x) = \sum w_i \text{ReLU}(x + b_i).$$
This model can approximate any smooth 1D function, provided enough hidden units.

---

class: middle

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle count: false

.center[]

---

class: middle

.bold[Universal approximation theorem.] (Cybenko 1989; Hornik et al, 1991) Let $\sigma(\cdot)$ be a bounded, non-constant continuous function. Let $I_p$ denote the $p$-dimensional hypercube, and $C(I_p)$ denote the space of continuous functions on $I_p$. Given any $f \in C(I_p)$ and $\epsilon &gt; 0$, there exists $q &gt; 0$ and $v_i, w_i, b_i, i=1, ..., q$ such that $$F(x) = \sum_{i \leq q} v_i \sigma(w_i^T x + b_i)$$ satisfies $$\sup_{x \in I_p} |f(x) - F(x)| &lt; \epsilon.$$

• It guarantees that even a single hidden-layer network can represent any classification problem in which the boundary is locally linear (smooth);
• It does not inform about good/bad architectures, nor how they relate to the optimization procedure.
• The universal approximation theorem generalizes to any non-polynomial (possibly unbounded) activation function, including the ReLU (Leshno, 1993).

---

LEGO® Deep Learning

.center.width-50[]

---

class: middle

.center.circle.width-30[]

.italic[ People are now building a new kind of software by .bold[assembling networks of parameterized functional blocks] and by .bold[training them from examples using some form of gradient-based optimization]. ]

.pull-right[Yann LeCun, 2018.]

---

class: middle

DL as an architectural language

.width-100[]

---

class: middle

.center[

The toolbox ]

.footnote[Credits: Oriol Vinyals, 2020.]

---

class: black-slide

LEGO® Creator Expert

.center[ .width-60[] .width-60[]

AlphaStar, DeepMind 2019. ]

.footnote[Credits: Vinyals et al, 2019.]

---

class: black-slide, middle

.center[

<iframe width="600" height="450" src="https://www.youtube.com/embed/j0z4FweCy4M?start=4279" frameborder="0" allowfullscreen></iframe>

HydraNet, Tesla 2021. ]

---

class: end-slide, center count: false

The end.

---

count: false

References

• Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6), 386.
• Bottou, L., & Bousquet, O. (2008). The tradeoffs of large scale learning. In Advances in neural information processing systems (pp. 161-168).
• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. nature, 323(6088), 533.