week_4/activation-functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings: Intro to CBOW model, activation functions and working with Numpy\n",
    "\n",
    "In this lecture notebook you will be given an introduction to the continuous bag-of-words model, its activation functions and some considerations when working with Numpy. \n",
    "\n",
    "Let's dive into it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The continuous bag-of-words model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CBOW model is based on a neural network, the architecture of which looks like the figure below, as you'll recall from the lecture.\n",
    "\n",
    "<div style=\"width:image width px; font-size:100%; text-align:center;\"><img src='cbow_model_architecture.png' alt=\"alternate text\" width=\"width\" height=\"height\" style=\"width:917;height:337;\" /> Figure 1 </div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by implementing the activation functions, ReLU and softmax."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ReLU is used to calculate the values of the hidden layer, in the following formulas:\n",
    "\n",
    "\\begin{align}\n",
    " \\mathbf{z_1} &= \\mathbf{W_1}\\mathbf{x} + \\mathbf{b_1}  \\tag{1} \\\\\n",
    " \\mathbf{h} &= \\mathrm{ReLU}(\\mathbf{z_1})  \\tag{2} \\\\\n",
    "\\end{align}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fix a value for $\\mathbf{z_1}$ as a working example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.71320643],\n",
       "       [-4.79248051],\n",
       "       [ 1.33648235],\n",
       "       [ 2.48803883],\n",
       "       [-0.01492988]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a random seed so all random outcomes can be reproduced\n",
    "np.random.seed(10)\n",
    "\n",
    "# Define a 5X1 column vector using numpy\n",
    "z_1 = 10 * np.random.rand(5, 1) - 5\n",
    "\n",
    "# Print the vector\n",
    "z_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that using numpy's `random.rand` function returns a numpy array filled with values taken from a uniform distribution over [0, 1). Numpy allows vectorization so each value is multiplied by 10 and then substracted 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the ReLU of this vector, you want all the negative values to become zeros.\n",
    "\n",
    "First create a copy of this vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create copy of vector and save it in the 'h' variable\n",
    "h = z_1.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now determine which of its values are negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False],\n",
       "       [ True],\n",
       "       [False],\n",
       "       [False],\n",
       "       [ True]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Determine which values met the criteria (this is possible because of vectorization)\n",
    "h < 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now simply set all of the values which are negative to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Slice the array or vector. This is the same as applying ReLU to it\n",
    "h[h < 0] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it: you have the ReLU of $\\mathbf{z_1}$!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.71320643],\n",
       "       [0.        ],\n",
       "       [1.33648235],\n",
       "       [2.48803883],\n",
       "       [0.        ]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print the vector after ReLU\n",
    "h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now implement ReLU as a function.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'relu' function that will include the steps previously seen\n",
    "def relu(z):\n",
    "    result = z.copy()\n",
    "    result[result < 0] = 0\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**And check that it's working.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.        ],\n",
       "       [4.50714306],\n",
       "       [2.31993942],\n",
       "       [0.98658484],\n",
       "       [0.        ]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a new vector and save it in the 'z' variable\n",
    "z = np.array([[-1.25459881], [ 4.50714306], [ 2.31993942], [ 0.98658484], [-3.4398136 ]])\n",
    "\n",
    "# Apply ReLU to it\n",
    "relu(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output:\n",
    "\n",
    "    array([[0.        ],\n",
    "           [4.50714306],\n",
    "           [2.31993942],\n",
    "           [0.98658484],\n",
    "           [0.        ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second activation function that you need is softmax. This function is used to calculate the values of the output layer of the neural network, using the following formulas:\n",
    "\n",
    "\\begin{align}\n",
    " \\mathbf{z_2} &= \\mathbf{W_2}\\mathbf{h} + \\mathbf{b_2}   \\tag{3} \\\\\n",
    " \\mathbf{\\hat y} &= \\mathrm{softmax}(\\mathbf{z_2})   \\tag{4} \\\\\n",
    "\\end{align}\n",
    "\n",
    "To calculate softmax of a vector $\\mathbf{z}$, the $i$-th component of the resulting vector is given by:\n",
    "\n",
    "$$ \\textrm{softmax}(\\textbf{z})_i = \\frac{e^{z_i} }{\\sum\\limits_{j=1}^{V} e^{z_j} }  \\tag{5} $$\n",
    "\n",
    "Let's work through an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 9. ,  8. , 11. , 10. ,  8.5])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a new vector and save it in the 'z' variable\n",
    "z = np.array([9, 8, 11, 10, 8.5])\n",
    "\n",
    "# Print the vector\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll need to calculate the exponentials of each element, both for the numerator and for the denominator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 8103.08392758,  2980.95798704, 59874.1417152 , 22026.46579481,\n",
       "        4914.7688403 ])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save exponentials of the values in a new vector\n",
    "e_z = np.exp(z)\n",
    "\n",
    "# Print the vector with the exponential values\n",
    "e_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The denominator is equal to the sum of these exponentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "97899.41826492078"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save the sum of the exponentials\n",
    "sum_e_z = np.sum(e_z)\n",
    "\n",
    "# Print sum of exponentials\n",
    "sum_e_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the value of the first element of $\\textrm{softmax}(\\textbf{z})$ is given by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08276947985173956"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print softmax value of the first element in the original vector\n",
    "e_z[0] / sum_e_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is for one element. You can use numpy's vectorized operations to calculate the values of all the elements of the $\\textrm{softmax}(\\textbf{z})$ vector in one go.\n",
    "\n",
    "**Implement the softmax function.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'softmax' function that will include the steps previously seen\n",
    "def softmax(z):\n",
    "    e_z = np.exp(z)\n",
    "    sum_e_z = np.sum(e_z)\n",
    "    return e_z / sum_e_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now check that it works.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print softmax values for original vector\n",
    "softmax([9, 8, 11, 10, 8.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output:\n",
    "\n",
    "    array([0.08276948, 0.03044919, 0.61158833, 0.22499077, 0.05020223])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the sum of all these values is equal to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assert that the sum of the softmax values is equal to 1\n",
    "np.sum(softmax([9, 8, 11, 10, 8.5])) == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensions: 1-D arrays vs 2-D column vectors\n",
    "\n",
    "Before moving on to implement forward propagation, backpropagation, and gradient descent in the next lecture notebook, let's have a look at the dimensions of the vectors you've been handling until now.\n",
    "\n",
    "Create a vector of length $V$ filled with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define V. Remember this was the size of the vocabulary in the previous lecture notebook\n",
    "V = 5\n",
    "\n",
    "# Define vector of length V filled with zeros\n",
    "x_array = np.zeros(V)\n",
    "\n",
    "# Print vector\n",
    "x_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a 1-dimensional array, as revealed by the `.shape` property of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print vector's shape\n",
    "x_array.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform matrix multiplication in the next steps, you actually need your column vectors to be represented as a matrix with one column. In numpy, this matrix is represented as a 2-dimensional array.\n",
    "\n",
    "The easiest way to convert a 1D vector to a 2D column matrix is to set its `.shape` property to the number of rows and one column, as shown in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Copy vector\n",
    "x_column_vector = x_array.copy()\n",
    "\n",
    "# Reshape copy of vector\n",
    "x_column_vector.shape = (V, 1)  # alternatively ... = (x_array.shape[0], 1)\n",
    "\n",
    "# Print vector\n",
    "x_column_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of the resulting \"vector\" is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 1)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print vector's shape\n",
    "x_column_vector.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you now have a 5x1 matrix that you can use to perform standard matrix multiplication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this lecture notebook!** Hopefully you now have a better understanding of the activation functions used in the continuous bag-of-words model, as well as a clearer idea of how to leverage Numpy's power for these types of mathematical computations.\n",
    "\n",
    "In the next lecture notebook you will get a comprehensive dive into:\n",
    "\n",
    "- Forward propagation.\n",
    "\n",
    "- Cross-entropy loss.\n",
    "\n",
    "- Backpropagation.\n",
    "\n",
    "- Gradient descent.\n",
    "\n",
    "**See you next time!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}