6_LSTM_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Long Short Term Memory (LSTM) Models\n",
    "\n",
    "This tutorial walks through programming a Long Short Term Memory Model in Python from scratch using only the numpy library for matrix operations, with an application to language models. The backpropogation steps are explicitly derived.\n",
    "\n",
    "## Introduction \n",
    "\n",
    "In speech, recent words might be better to predict the next word in a sentence. But sometimes, words from longer time horizons in the past might be important for prediction. For example, if we start talking about the LA Lakers and I say, \"Kobe Bryant is the greatest of all -\" and then stop talking, a die hard fan might scream out \"TIME!\". Two things may have prompted this response: 1) the fact I had mentioned Kobe Bryant to begin with, making you think of the greatest of the greats to ever play the game, hence thinking \"of all time\". And 2) The word \"time\" might typically follow the phrase \"greatest of all time\". But what if we weren't talking about all players before, but our conversation was strictly about shooting guards. Then maybe the proper response would be \"greatest of all shooting guards to ever play the game\". \n",
    "\n",
    "The immediate past as well as the far past (beginning of our hour long basketball conversation) might be important to predict the next word.\n",
    "\n",
    "Long Short Term Memory Models seem properly fit to train a language model that can have a memory, i.e. forget unnecessary information in the conversation history and bring its attention to relevant information, be it from the immediate past or further back in time.\n",
    "\n",
    "## Reading \n",
    "\n",
    "Now would be a good time to mention that LSTM models have been around for a few years and there are a lot of very useful tutorials online which are much better at explaining the inner workings of the model. \n",
    "\n",
    "1. [Christopher Olah has a great explanation with clear diagrams](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n",
    "\n",
    "2. [Nico Jimenez's page is what motivated this speech based version of his code](http://nicodjimenez.github.io/2014/08/08/lstm.html)\n",
    "\n",
    "3. [Arun Mallya's clean presentation about LSTM's helped my understanding of the backpropogation](http://arunmallya.github.io/writeups/nn/lstm/index.html#/)\n",
    "\n",
    "After reading these articles, I think the rest of the tutorial will make more sense.\n",
    "\n",
    "### Differences with this tutorial\n",
    "I had a hard time understanding the backpropogation of LSTM's for speech models. The main difference with this tutorial is twofold  \n",
    "\n",
    "1. Illustrate how you can program LSTMs for sentences. Say you have a bunch of sentences, and you want to develop a LSTM model for language, maybe for a chatbot or generative robot. This tutorial is to show you one way of quantifying the words/sentences as examples and training the model.\n",
    "\n",
    "2. Because the focus is on speech, the output to the LSTM is a distribution over words in the vocabulary. Hence, the activation function on the output is a [softmax](https://en.wikipedia.org/wiki/Softmax_function) function. The backpropogration was not entirely clear from previous tutorials so I explicitly derive the formulas below for additional clarity.\n",
    "\n",
    "## Forward Propogation \n",
    "\n",
    "The main idea behind an LSTM is to have a cell state running throughout the entire sentence, and this cell state will be updated after the introduction of each new word to the sentence. The reason this will be useful is because it allows us to include memory into the model. \n",
    "\n",
    "The LSTM language model will start off by taking in a one-hot encoded vector for the first word in the sentence, $x_0$, say \"Kobe\". With this, we will create a hidden state and cell state for $t=0$.\n",
    "\n",
    "At each time period (or introduction of new word in our case), the LSTM model will compute three gates: a forget gate, an input gate, and an output gate. As an example, suppose we heard the word \"Kobe\" at $t=0$ and are now at $t=1$, where the word \"Bryant\" was just said. The one-hot encoded vector \"Bryant\" will be $x_1$.\n",
    "\n",
    "The forget gate tells us how much to forget from time $t=0$'s cell state. So, maybe the first word at $t=0$ was \"Kobe\", and the abstract cell space put a lot of weight on the words \"Beef\" and \"Bryant\". Once we get the word \"Bryant\" at $t=1$, we should forget anything in the cell state putting weight on \"Beef\". \n",
    "\n",
    "The input gate tells us how much weight should be put on the new information from the current word given. What if we are at $t=1$, and instead of \"Bryant\" we now hear \"uhhhhhhh\". We know this is a filler word so we should not put much weight on it. The input gate will be a vector of small weights so $x_1$ vector \"uhhhhhhh\" doesn't contribute largely to the cell state at $t=1$. \n",
    "\n",
    "Lastly, the output gate tells us how much of the current cell state should we pass onto the following word. This isn't completely necessary but it gives us a little extra dynamics for fitting the model.\n",
    "\n",
    "The input node will be a transformation into the hidden space of the new word and the past hidden state. \n",
    "\n",
    "Lastly, we will generate a new value (vector in our case) of the cell state using the forget gate, input gate, output gate, and new input vector. This cell state will be used to create the hidden state (we use as output), which is the distribution over words in our vocabulary. \n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\text{Input Gate}\\hspace{2em} & i_t = \\sigma(\n",
    "\\begin{pmatrix}\n",
    "\\Theta_{xi} \\Theta_{hi}\n",
    "\\end{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "x_t \\\\ h_{t-1}\n",
    "\\end{pmatrix}\n",
    "+ b_i) \\\\\n",
    "\\text{Forget Gate}\\hspace{2em}  & f_t = \\sigma(\n",
    "\\begin{pmatrix}\n",
    "\\Theta_{xf} \\Theta_{hf}\n",
    "\\end{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "x_t \\\\ h_{t-1}\n",
    "\\end{pmatrix}\n",
    "+ b_f) \\\\\n",
    "\\text{Output Gate}\\hspace{2em}  & o_t = \\sigma(\n",
    "\\begin{pmatrix}\n",
    "\\Theta_{xo} \\Theta_{ho}\n",
    "\\end{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "x_t \\\\ h_{t-1}\n",
    "\\end{pmatrix}\n",
    "+ b_o) \\\\\n",
    "\\text{Input Node}\\hspace{2em}  & g_t = \\tanh(\n",
    "\\begin{pmatrix}\n",
    "\\Theta_{xg} \\Theta_{hg}\n",
    "\\end{pmatrix}\n",
    "\\begin{pmatrix}\n",
    "x_t \\\\ h_{t-1}\n",
    "\\end{pmatrix}\n",
    "+ b_g) \\\\\n",
    "\\text{Cell State}\\hspace{2em}  & c_t = f_t\\circ c_{t-1}+i_t\\circ g_t \\\\\n",
    "\\text{Preactivation}\\hspace{2em}  & z_t = o_t \\circ tanh(c_t) \\\\\n",
    "\\text{Output}\\hspace{2em}  & h_t = softmax(\\Theta_h*z_t+b_h) \\\\\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Training the model with Backpropogation \n",
    "\n",
    "Backpropogation is similar to a plain vanilla neural network, but the time component adds a few intricacies. We use [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) to find the parameters of the model.\n",
    "\n",
    "<b>NOTE:</b> For ease, I disregard matrix notation with the derivatives, but the code will hopefully properly handle matrix multiplication correctly.\n",
    "\n",
    "Define the loss function:\n",
    "\n",
    "$ loss = \\sum_{t=1}^T L(t)$\n",
    "\n",
    "Loss at time $t$ is cumulative from time $t$ to $T$. Thus, let:\n",
    "\n",
    "$$L(t) = \\sum_{s=t}^T l(s)$$\n",
    "\n",
    "where $l(s)$ is $\\frac{1}{2}(h(s) - targ(s))^2$, the difference between $y(s)$ outputted by the model and the true target at $s$, $targ(s)$.\n",
    "\n",
    "Thus, for any $w$ (such as: $\\Theta_{xi}, \\Theta_{hi},\\Theta_{xf} \\Theta_{hf}...$)\n",
    "\n",
    "$$ \\frac{\\partial loss}{\\partial w} = \\sum_{t=1}^T \\frac{\\partial L(t)}{\\partial w} = \\sum_{t=1}^T \\frac{\\partial L(t)}{\\partial h(t)} \\frac{\\partial h(t)}{\\partial w}$$\n",
    "\n",
    "Through recursion, we can rewrite $L(t)= l(t) + L(t+1)$ (though for $t=T, L(t) = l(t)$). Hence, \n",
    "\n",
    "$$\\frac{\\partial L(t)}{\\partial h(t)} = \\frac{\\partial l(t)}{\\partial h(t)} + \\frac{\\partial L(t+1)}{\\partial h(t)}$$\n",
    "\n",
    "First, let's find the update for $\\Theta_h$.\n",
    "\n",
    "\n",
    "$$ \n",
    "\\frac{\\partial loss}{\\partial \\Theta_h} =\\sum_{t=1}^T \\frac{\\partial L(t)}{\\partial h(t)} \\frac{\\partial h(t)}{\\partial \\Theta_{h}}=\\sum_{t=1}^T (\\frac{\\partial l(t)}{\\partial h(t)}+\\frac{\\partial L(t+1)}{\\partial h(t)}) \\frac{\\partial h(t)}{\\partial \\Theta_{h}}=\\sum_{t=1}^T (h(t)-targ(t)+{\\mathbf{next\\_dh}}) \\frac{\\partial h(t)}{\\partial \\Theta_{h}}\n",
    "$$\n",
    ".\n",
    "$$\n",
    "=\\sum_{t=1}^T\\mathbf{dh}*softmax\\_deriv(z_t)*z_t^{\\prime}=\\sum_{t=1}^T\\mathbf{dz}*z_t^{\\prime} \n",
    "$$\n",
    ".\n",
    "Now, let's find the update for $\\Theta_o = \\Theta_{xo}\\Theta_{ho}$.\n",
    "\n",
    "$$ \\frac{\\partial loss}{\\partial \\Theta_o} =\\sum_{t=1}^T \\frac{\\partial L(t)}{\\partial h(t)} \\frac{\\partial h(t)}{\\partial \\Theta_{o}}=\\sum_{t=1}^T (\\frac{\\partial l(t)}{\\partial h(t)}+\\frac{\\partial L(t+1)}{\\partial h(t)}) \\frac{\\partial h(t)}{\\partial \\Theta_{o}}=\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial \\Theta_{o}}$$\n",
    ".\n",
    "$$\n",
    "=\\sum_{t=1}^T\\mathbf{dh}*softmax\\_deriv(z_t)*\\Theta_h*tanh(c_t)*sigmoid\\_deriv(o_t)*[x_t h_{t-1}]^{\\prime}=\\sum_{t=1}^T\\mathbf{do}*[x_t h_{t-1}]^{\\prime} \n",
    "$$\n",
    "\n",
    "\n",
    "The update for $\\Theta_i = \\Theta_{xi}\\Theta_{hi}$, $\\Theta_f = \\Theta_{xf}\\Theta_{hf}$, and $\\Theta_g = \\Theta_{xg}\\Theta_{hg}$ are all quite similar. One main difference is that we will need to pass $\\mathbf{next\\_dc}$, which is the derivative of the lagged cell state with respect to the parameter value. Let's do one of them, $\\Theta_f$.\n",
    "\n",
    "$$ \\frac{\\partial loss}{\\partial \\Theta_f} =\\sum_{t=1}^T \\frac{\\partial L(t)}{\\partial h(t)} \\frac{\\partial h(t)}{\\partial \\Theta_{f}}=\\sum_{t=1}^T (\\frac{\\partial l(t)}{\\partial h(t)}+\\frac{\\partial L(t+1)}{\\partial h(t)}) \\frac{\\partial h(t)}{\\partial \\Theta_{f}}=\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial \\Theta_{f}}=\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)}\\frac{\\partial c(t)}{\\partial \\Theta_{f}}$$\n",
    ".\n",
    "$$\n",
    "\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)}[\\frac{\\partial c(t-1)}{\\partial \\Theta_{f}}*f_t + \\frac{\\partial f(t)}{\\partial \\Theta_f}*c_{t-1}]\n",
    "$$\n",
    ".\n",
    "$$\n",
    "\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)} \\frac{\\partial f(t)}{\\partial \\Theta_f}*c_{t-1}+[(h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)}\\frac{\\partial c(t-1)}{\\partial \\Theta_{f}}*f_t]\n",
    "$$\n",
    ".\n",
    "$$\n",
    "\\sum_{t=1}^T (h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)} \\frac{\\partial f(t)}{\\partial \\Theta_f}*c_{t-1}+[\\frac{\\partial L(t-1)}{\\partial c(t-1)} \\frac{\\partial c(t-1)}{\\partial \\Theta_{f}}*f_t]\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\sum_{t=1}^T [(h(t)-targ(t)+\\mathbf{next\\_dh}) \\frac{\\partial h(t)}{\\partial c(t)} \\frac{\\partial f(t)}{\\partial \\Theta_f}*c_{t-1}]+[\\mathbf{next\\_dc}*\\frac{\\partial f(t)}{\\partial \\Theta_f}*c_{t-1}]\n",
    "$$\n",
    ".\n",
    "$$\n",
    "=\\sum_{t=1}^T[\\mathbf{dh}*softmax\\_deriv(z_t)*\\Theta_h*o_t*tanh\\_deriv(c_t)+\\mathbf{next\\_dc}]*c_{t-1}*sigmoid\\_deriv(f)*[x_t h_{t-1}]^{\\prime} \n",
    "$$\n",
    ".\n",
    "$$\n",
    "=\\sum_{t=1}^T\\mathbf{dc}*c_{t-1}*sigmoid\\_deriv(f)*[x_t h_{t-1}]^{\\prime}=\\sum_{t=1}^T\\mathbf{df}*[x_t h_{t-1}]^{\\prime} \n",
    "$$\n",
    ".\n",
    "Lastly, we need to find a way to pass $\\frac{\\partial L(t)}{\\partial h(t-1)}$ and $\\frac{\\partial L(t)}{\\partial c(t-1)}*f_t$to the previous cell state since $h_{t-1}$ shows up within each $i_t, f_t, o_t$ and $g_t$. This will be easy, since it will be \n",
    "\n",
    "$$\\frac{\\partial L(t)}{\\partial h(t-1)}= \\frac{\\partial L(t)}{\\partial i_t}*\\Theta_{hi}+\\frac{\\partial L(t)}{\\partial f_t}*\\Theta_{hf}+\\frac{\\partial L(t)}{\\partial o_t}*\\Theta_{ho}+\\frac{\\partial L(t)}{\\partial g_t}*\\Theta_{hg}$$\n",
    "$$ =\\mathbf{di*\\Theta_{hi}+df*\\Theta_{hf}+do*\\Theta_{ho}+dg*\\Theta_{hg}}$$\n",
    ".\n",
    "$$\\frac{\\partial L(t)}{\\partial c(t-1)}= \\mathbf{dh}*softmax\\_deriv(z_t)*\\Theta_h*o_t*tanh\\_deriv(c_t)*f_t = \\mathbf{dc}*f_t$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch number 0\n",
      "epoch number 1\n",
      "epoch number 2\n",
      "epoch number 3\n",
      "epoch number 4\n",
      "epoch number 5\n",
      "epoch number 6\n",
      "epoch number 7\n",
      "epoch number 8\n",
      "epoch number 9\n",
      "epoch number 10\n",
      "epoch number 11\n",
      "epoch number 12\n",
      "epoch number 13\n",
      "epoch number 14\n",
      "epoch number 15\n",
      "epoch number 16\n",
      "epoch number 17\n",
      "epoch number 18\n",
      "epoch number 19\n",
      "epoch number 20\n",
      "epoch number 21\n",
      "epoch number 22\n",
      "epoch number 23\n",
      "epoch number 24\n",
      "epoch number 25\n",
      "epoch number 26\n",
      "epoch number 27\n",
      "epoch number 28\n",
      "epoch number 29\n",
      "epoch number 30\n",
      "epoch number 31\n",
      "epoch number 32\n",
      "epoch number 33\n",
      "epoch number 34\n",
      "epoch number 35\n",
      "epoch number 36\n",
      "epoch number 37\n",
      "epoch number 38\n",
      "epoch number 39\n",
      "epoch number 40\n",
      "epoch number 41\n",
      "epoch number 42\n",
      "epoch number 43\n",
      "epoch number 44\n",
      "epoch number 45\n",
      "epoch number 46\n",
      "epoch number 47\n",
      "epoch number 48\n",
      "epoch number 49\n",
      "epoch number 50\n",
      "epoch number 51\n",
      "epoch number 52\n",
      "epoch number 53\n",
      "epoch number 54\n",
      "epoch number 55\n",
      "epoch number 56\n",
      "epoch number 57\n",
      "epoch number 58\n",
      "epoch number 59\n",
      "epoch number 60\n",
      "epoch number 61\n",
      "epoch number 62\n",
      "epoch number 63\n",
      "epoch number 64\n",
      "epoch number 65\n",
      "epoch number 66\n",
      "epoch number 67\n",
      "epoch number 68\n",
      "epoch number 69\n",
      "epoch number 70\n",
      "epoch number 71\n",
      "epoch number 72\n",
      "epoch number 73\n",
      "epoch number 74\n",
      "epoch number 75\n",
      "epoch number 76\n",
      "epoch number 77\n",
      "epoch number 78\n",
      "epoch number 79\n",
      "epoch number 80\n",
      "epoch number 81\n",
      "epoch number 82\n",
      "epoch number 83\n",
      "epoch number 84\n",
      "epoch number 85\n",
      "epoch number 86\n",
      "epoch number 87\n",
      "epoch number 88\n",
      "epoch number 89\n",
      "epoch number 90\n",
      "epoch number 91\n",
      "epoch number 92\n",
      "epoch number 93\n",
      "epoch number 94\n",
      "epoch number 95\n",
      "epoch number 96\n",
      "epoch number 97\n",
      "epoch number 98\n",
      "epoch number 99\n",
      "epoch number 100\n",
      "epoch number 101\n",
      "epoch number 102\n",
      "epoch number 103\n",
      "epoch number 104\n",
      "epoch number 105\n",
      "epoch number 106\n",
      "epoch number 107\n",
      "epoch number 108\n",
      "epoch number 109\n",
      "epoch number 110\n",
      "epoch number 111\n",
      "epoch number 112\n",
      "epoch number 113\n",
      "epoch number 114\n",
      "epoch number 115\n",
      "epoch number 116\n",
      "epoch number 117\n",
      "epoch number 118\n",
      "epoch number 119\n",
      "epoch number 120\n",
      "epoch number 121\n",
      "epoch number 122\n",
      "epoch number 123\n",
      "epoch number 124\n",
      "epoch number 125\n",
      "epoch number 126\n",
      "epoch number 127\n",
      "epoch number 128\n",
      "epoch number 129\n",
      "epoch number 130\n",
      "epoch number 131\n",
      "epoch number 132\n",
      "epoch number 133\n",
      "epoch number 134\n",
      "epoch number 135\n",
      "epoch number 136\n",
      "epoch number 137\n",
      "epoch number 138\n",
      "epoch number 139\n",
      "epoch number 140\n",
      "epoch number 141\n",
      "epoch number 142\n",
      "epoch number 143\n",
      "epoch number 144\n",
      "epoch number 145\n",
      "epoch number 146\n",
      "epoch number 147\n",
      "epoch number 148\n",
      "epoch number 149\n",
      "epoch number 150\n",
      "epoch number 151\n",
      "epoch number 152\n",
      "epoch number 153\n",
      "epoch number 154\n",
      "epoch number 155\n",
      "epoch number 156\n",
      "epoch number 157\n",
      "epoch number 158\n",
      "epoch number 159\n",
      "epoch number 160\n",
      "epoch number 161\n",
      "epoch number 162\n",
      "epoch number 163\n",
      "epoch number 164\n",
      "epoch number 165\n",
      "epoch number 166\n",
      "epoch number 167\n",
      "epoch number 168\n",
      "epoch number 169\n",
      "epoch number 170\n",
      "epoch number 171\n",
      "epoch number 172\n",
      "epoch number 173\n",
      "epoch number 174\n",
      "epoch number 175\n",
      "epoch number 176\n",
      "epoch number 177\n",
      "epoch number 178\n",
      "epoch number 179\n",
      "epoch number 180\n",
      "epoch number 181\n",
      "epoch number 182\n",
      "epoch number 183\n",
      "epoch number 184\n",
      "epoch number 185\n",
      "epoch number 186\n",
      "epoch number 187\n",
      "epoch number 188\n",
      "epoch number 189\n",
      "epoch number 190\n",
      "epoch number 191\n",
      "epoch number 192\n",
      "epoch number 193\n",
      "epoch number 194\n",
      "epoch number 195\n",
      "epoch number 196\n",
      "epoch number 197\n",
      "epoch number 198\n",
      "epoch number 199\n",
      "epoch number 200\n",
      "epoch number 201\n",
      "epoch number 202\n",
      "epoch number 203\n",
      "epoch number 204\n",
      "epoch number 205\n",
      "epoch number 206\n",
      "epoch number 207\n",
      "epoch number 208\n",
      "epoch number 209\n",
      "epoch number 210\n",
      "epoch number 211\n",
      "epoch number 212\n",
      "epoch number 213\n",
      "epoch number 214\n",
      "epoch number 215\n",
      "epoch number 216\n",
      "epoch number 217\n",
      "epoch number 218\n",
      "epoch number 219\n",
      "epoch number 220\n",
      "epoch number 221\n",
      "epoch number 222\n",
      "epoch number 223\n",
      "epoch number 224\n",
      "epoch number 225\n",
      "epoch number 226\n",
      "epoch number 227\n",
      "epoch number 228\n",
      "epoch number 229\n",
      "epoch number 230\n",
      "epoch number 231\n",
      "epoch number 232\n",
      "epoch number 233\n",
      "epoch number 234\n",
      "epoch number 235\n",
      "epoch number 236\n",
      "epoch number 237\n",
      "epoch number 238\n",
      "epoch number 239\n",
      "epoch number 240\n",
      "epoch number 241\n",
      "epoch number 242\n",
      "epoch number 243\n",
      "epoch number 244\n",
      "epoch number 245\n",
      "epoch number 246\n",
      "epoch number 247\n",
      "epoch number 248\n",
      "epoch number 249\n",
      "epoch number 250\n",
      "epoch number 251\n",
      "epoch number 252\n",
      "epoch number 253\n",
      "epoch number 254\n",
      "epoch number 255\n",
      "epoch number 256\n",
      "epoch number 257\n",
      "epoch number 258\n",
      "epoch number 259\n",
      "epoch number 260\n",
      "epoch number 261\n",
      "epoch number 262\n",
      "epoch number 263\n",
      "epoch number 264\n",
      "epoch number 265\n",
      "epoch number 266\n",
      "epoch number 267\n",
      "epoch number 268\n",
      "epoch number 269\n",
      "epoch number 270\n",
      "epoch number 271\n",
      "epoch number 272\n",
      "epoch number 273\n",
      "epoch number 274\n",
      "epoch number 275\n",
      "epoch number 276\n",
      "epoch number 277\n",
      "epoch number 278\n",
      "epoch number 279\n",
      "epoch number 280\n",
      "epoch number 281\n",
      "epoch number 282\n",
      "epoch number 283\n",
      "epoch number 284\n",
      "epoch number 285\n",
      "epoch number 286\n",
      "epoch number 287\n",
      "epoch number 288\n",
      "epoch number 289\n",
      "epoch number 290\n",
      "epoch number 291\n",
      "epoch number 292\n",
      "epoch number 293\n",
      "epoch number 294\n",
      "epoch number 295\n",
      "epoch number 296\n",
      "epoch number 297\n",
      "epoch number 298\n",
      "epoch number 299\n",
      "epoch number 300\n",
      "epoch number 301\n",
      "epoch number 302\n",
      "epoch number 303\n",
      "epoch number 304\n",
      "epoch number 305\n",
      "epoch number 306\n",
      "epoch number 307\n",
      "epoch number 308\n",
      "epoch number 309\n",
      "epoch number 310\n",
      "epoch number 311\n",
      "epoch number 312\n",
      "epoch number 313\n",
      "epoch number 314\n",
      "epoch number 315\n",
      "epoch number 316\n",
      "epoch number 317\n",
      "epoch number 318\n",
      "epoch number 319\n",
      "epoch number 320\n",
      "epoch number 321\n",
      "epoch number 322\n",
      "epoch number 323\n",
      "epoch number 324\n",
      "epoch number 325\n",
      "epoch number 326\n",
      "epoch number 327\n",
      "epoch number 328\n",
      "epoch number 329\n",
      "epoch number 330\n",
      "epoch number 331\n",
      "epoch number 332\n",
      "epoch number 333\n",
      "epoch number 334\n",
      "epoch number 335\n",
      "epoch number 336\n",
      "epoch number 337\n",
      "epoch number 338\n",
      "epoch number 339\n",
      "epoch number 340\n",
      "epoch number 341\n",
      "epoch number 342\n",
      "epoch number 343\n",
      "epoch number 344\n",
      "epoch number 345\n",
      "epoch number 346\n",
      "epoch number 347\n",
      "epoch number 348\n",
      "epoch number 349\n",
      "epoch number 350\n",
      "epoch number 351\n",
      "epoch number 352\n",
      "epoch number 353\n",
      "epoch number 354\n",
      "epoch number 355\n",
      "epoch number 356\n",
      "epoch number 357\n",
      "epoch number 358\n",
      "epoch number 359\n",
      "epoch number 360\n",
      "epoch number 361\n",
      "epoch number 362\n",
      "epoch number 363\n",
      "epoch number 364\n",
      "epoch number 365\n",
      "epoch number 366\n",
      "epoch number 367\n",
      "epoch number 368\n",
      "epoch number 369\n",
      "epoch number 370\n",
      "epoch number 371\n",
      "epoch number 372\n",
      "epoch number 373\n",
      "epoch number 374\n",
      "epoch number 375\n",
      "epoch number 376\n",
      "epoch number 377\n",
      "epoch number 378\n",
      "epoch number 379\n",
      "epoch number 380\n",
      "epoch number 381\n",
      "epoch number 382\n",
      "epoch number 383\n",
      "epoch number 384\n",
      "epoch number 385\n",
      "epoch number 386\n",
      "epoch number 387\n",
      "epoch number 388\n",
      "epoch number 389\n",
      "epoch number 390\n",
      "epoch number 391\n",
      "epoch number 392\n",
      "epoch number 393\n",
      "epoch number 394\n",
      "epoch number 395\n",
      "epoch number 396\n",
      "epoch number 397\n",
      "epoch number 398\n",
      "epoch number 399\n",
      "epoch number 400\n",
      "epoch number 401\n",
      "epoch number 402\n",
      "epoch number 403\n",
      "epoch number 404\n",
      "epoch number 405\n",
      "epoch number 406\n",
      "epoch number 407\n",
      "epoch number 408\n",
      "epoch number 409\n",
      "epoch number 410\n",
      "epoch number 411\n",
      "epoch number 412\n",
      "epoch number 413\n",
      "epoch number 414\n",
      "epoch number 415\n",
      "epoch number 416\n",
      "epoch number 417\n",
      "epoch number 418\n",
      "epoch number 419\n",
      "epoch number 420\n",
      "epoch number 421\n",
      "epoch number 422\n",
      "epoch number 423\n",
      "epoch number 424\n",
      "epoch number 425\n",
      "epoch number 426\n",
      "epoch number 427\n",
      "epoch number 428\n",
      "epoch number 429\n",
      "epoch number 430\n",
      "epoch number 431\n",
      "epoch number 432\n",
      "epoch number 433\n",
      "epoch number 434\n",
      "epoch number 435\n",
      "epoch number 436\n",
      "epoch number 437\n",
      "epoch number 438\n",
      "epoch number 439\n",
      "epoch number 440\n",
      "epoch number 441\n",
      "epoch number 442\n",
      "epoch number 443\n",
      "epoch number 444\n",
      "epoch number 445\n",
      "epoch number 446\n",
      "epoch number 447\n",
      "epoch number 448\n",
      "epoch number 449\n",
      "epoch number 450\n",
      "epoch number 451\n",
      "epoch number 452\n",
      "epoch number 453\n",
      "epoch number 454\n",
      "epoch number 455\n",
      "epoch number 456\n",
      "epoch number 457\n",
      "epoch number 458\n",
      "epoch number 459\n",
      "epoch number 460\n",
      "epoch number 461\n",
      "epoch number 462\n",
      "epoch number 463\n",
      "epoch number 464\n",
      "epoch number 465\n",
      "epoch number 466\n",
      "epoch number 467\n",
      "epoch number 468\n",
      "epoch number 469\n",
      "epoch number 470\n",
      "epoch number 471\n",
      "epoch number 472\n",
      "epoch number 473\n",
      "epoch number 474\n",
      "epoch number 475\n",
      "epoch number 476\n",
      "epoch number 477\n",
      "epoch number 478\n",
      "epoch number 479\n",
      "epoch number 480\n",
      "epoch number 481\n",
      "epoch number 482\n",
      "epoch number 483\n",
      "epoch number 484\n",
      "epoch number 485\n",
      "epoch number 486\n",
      "epoch number 487\n",
      "epoch number 488\n",
      "epoch number 489\n",
      "epoch number 490\n",
      "epoch number 491\n",
      "epoch number 492\n",
      "epoch number 493\n",
      "epoch number 494\n",
      "epoch number 495\n",
      "epoch number 496\n",
      "epoch number 497\n",
      "epoch number 498\n",
      "epoch number 499\n",
      "'we': think\n",
      "'we think': uncertainty\n",
      "'uncertainty and': fears\n",
      "'uncertainty and fears about': inflation\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VNX9x/H3NztJCCEQ9iXIKvsSEXFDFERwQa0rKGqF\nqtVq1br8atXWpbXaat3FHRWx7lZbKwoKyCJhlX3fl7ATQvac3x8ZaFiUJDOZm5n5vJ4nDzN37sz9\nnvTpx5Mz555jzjlERCT0RXldgIiIBIYCXUQkTCjQRUTChAJdRCRMKNBFRMKEAl1EJEwo0CXsmdnD\nZrbdzLZ4XYtIdVKgS1CY2RozO8uD67YA7gA6OucaBegznZm1OcrxODP7m5ltMLN9vjY/5XttX7mf\nUjPLK/d8mJk96PvcWw/7zFt9xx8MRO0S3hToEu5aADucc9mVfaOZxVTyLfcCmUBvoDbQD5gN4JxL\nPvADrAPOK3fsHd/7lwFXH/aZI3zHRY5JgS6eM7ORZrbCzHaa2Wdm1sR33MzsSTPLNrO9ZvajmXX2\nvTbYzBaZWY6ZbTSzO4/yuWcB44Emvp7wG77j55vZQjPbbWbfmtnx5d6zxszuNrP5QG4lQ/0E4GPn\n3CZXZo1zbkwl3j8TSDSzTr5aOgEJvuMix6RAF0+ZWX/gz8ClQGNgLTDO9/JA4DSgHVDHd84O32uv\nAr9yztUGOgMTDv9s59zXwDnAJl9P+Bozawe8C9wGpAP/Bv5lZnHl3noFMARIdc4VV6I504Hbzewm\nM+tiZlaJ9x7wFv/rpY/wPRepEAW6eG0Y8JpzbrZzroCyYYuTzCwDKKJs6KIDYM65xc65zb73FQEd\nzSzFObfLOTe7gte7DPjCOTfeOVcEPAHUAvqWO+dp59x651xeJdvyZ+AxX5uygI1mNqKSn/E2cIWZ\nxQKX+56LVIgCXbzWhLJeOQDOuX2U9cKbOucmAM8CzwHZZjbazFJ8p14MDAbWmtl3ZnZSFa9XCqwH\nmpY7Z31VGuKcK3HOPeecOxlIBR4BXis/pFOBz1gHrAAeBZY756pUi0QmBbp4bRPQ8sATM0sC6gEb\nAZxzTzvnegEdKRt6+Z3v+Ezn3AVAA+AT4J9VvJ4BzQ9cz8fvJUidc3nOueeAXb7aK2MMZTNzKjP+\nLqJAl6CKNbOEcj8xlI1nX2tm3c0snrKe6Qzn3BozO8HMTvQNP+QC+UCpb3rgMDOr4xs22QuUVrCG\nfwJDzOxM3+feARQAUyvZlrjD2hJtZreZWT8zq2VmMb7hltrAnEp+9nuUfX9Q0f9IiQBQ2WlZIv74\n92HPH3HO3WdmfwA+BOpSFqyX+15PAZ4EjqMszP8LPO577SrgWTOLBpZSNm59TM65pWY2HHiGsmGW\nuZRNISysZFsWHvZ8JLAf+BvQhrJe/jLgYufcqsp8sG/s/utK1iOCaYMLEZHwoCEXEZEwoUAXEQkT\nCnQRkTChQBcRCRNBneVSv359l5GREcxLioiEvFmzZm13zqUf67ygBnpGRgZZWVnBvKSISMgzs7XH\nPktDLiIiYUOBLiISJhToIiJhQoEuIhImFOgiImFCgS4iEiYU6CIiYSIkAn3i0mye/3aF12WIiNRo\nIRHoU1ds56mvl1NYXNE9DEREIk9IBHq35qkUFpeyZMter0sREamxQiLQuzdPBWDu+t0eVyIiUnOF\nRKA3Ta1FkzoJTFm+3etSRERqrJAIdDNjQMeGTFq+jf2FxV6XIyJSI4VEoAMM7tKY/KJS/v3jFq9L\nERGpkUIm0Hu3SqN1ehKvTVlNaak2thYROVzIBLqZcXP/NizavJcPZm/wuhwRkRonZAId4PxuTTmx\nVRr3f7qAT+duxDn11EVEDjhmoJvZa2aWbWYLyh1LM7PxZrbc92/d6i2zTHSU8fywnnRolMKt4+Zy\nxcvTWbxZc9NFRKBiPfQ3gEGHHbsH+MY51xb4xvc8KOolx/PhjX15aGhnlmzJYcjTk3n480W6i1RE\nIt4xA905NwnYedjhC4A3fY/fBIYGuK6fFR1lXNWnJd/e2Y8rerfglSmr+cWLU1m3Y38wyxARqVGq\nOobe0Dm32fd4C9Dwp040s1FmlmVmWdu2bavi5Y4uNTGORy7swovDe7Jmey5Dnp7Mt0uzA3oNEZFQ\n4feXoq7sm8mf/HbSOTfaOZfpnMtMT0/393JHNahzY774zak0S0vkujdm8uqU1frCVEQiTlUDfauZ\nNQbw/et5t7h5WiIf3HASAzo25KHPF3HvRz9qXF1EIkpVA/0zYITv8Qjg08CU45+k+BheGNaLm89o\nw7iZ67n6tRnszS/yuiwRkaCoyLTFd4FpQHsz22BmvwT+Agwws+XAWb7nNUJUlHHn2e158rJuZK3Z\nxWUvTSd7b77XZYmIVDsL5lhzZmamy8rKCtr1vlu2jRvfnkW95DjGXHcireonBe3aIiKBYmaznHOZ\nxzovpO4UrazT26UzdmQfcgtK+MULU1mwcY/XJYmIVJuwDnQo2xzjgxtOIj4miuGvzmDhJoW6iISn\nsA90gOPSkxk36iQSY6MZ9soMFm3ScgEiEn4iItABWtRLLBfq07U/qYiEnYgJdCgL9XdH9SE+JpoR\nr/3Axt15XpckIhIwERXoAC3rJfHGdSewv7CEEa/9wO79hV6XJCISEBEX6AAdGqUw+qpM1u3Yz8gx\nWeQXlXhdkoiI3yIy0AFOal2Pv1/Wjay1u7jrg/la+0VEQl7EBjrAuV2bcOfA9nw2bxOvTF7tdTki\nIn6J6EAHuKlfawZ3acSf/7OYKcu3e12OiEiVRXygmxmP/6IbbRvU5uZ3Z7N+pzbJEJHQFPGBDmWr\nNI6+uhelpY5fj52tZXdFJCQp0H1a1kvi8Uu6MX/DHh77conX5YiIVJoCvZyzOzVixEkteXXKar5Z\nvNXrckREKkWBfph7Bx9Px8Yp3PH+PDbv0Z2kIhI6FOiHSYiN5tkre1BYXMqt786luETj6SISGhTo\nR3FcejIPD+3MD2t28syEFV6XIyJSIX4FupndamYLzGyhmd0WqKJqgot6NuOiHk15ZsJypq/a4XU5\nIiLHVOVAN7POwEigN9ANONfM2gSqsJrgT0M70yItkdvGzWVXrhbxEpGazZ8e+vHADOfcfudcMfAd\ncFFgyqoZkuNjeOaKnuzILeCuD7Xei4jUbP4E+gLgVDOrZ2aJwGCg+eEnmdkoM8sys6xt27b5cTlv\ndGlWh7sHdWD8oq28NX2t1+WIiPykKge6c24x8BjwFfAlMBc4Yh1a59xo51ymcy4zPT29yoV66bqT\nW9GvfToPf7FY29eJSI3l15eizrlXnXO9nHOnAbuAZYEpq2aJijKeuKQbdWrFcsu7s9lfWOx1SSIi\nR/B3lksD378tKBs/HxuIomqi+snxPHVZd1Ztz+VP/1rkdTkiIkfwdx76h2a2CPgX8Gvn3O4A1FRj\nndymPjec3ppxM9fzr3mbvC5HROQQMf682Tl3aqAKCRW3D2jH9FU7+L+PfqR781SapyV6XZKICKA7\nRSstNjqKpy/vAQY3vD2LvELtRyoiNYMCvQqapyXyj8u7s2jzXu7W/HQRqSEU6FXUv0PDg/uRvjx5\nldfliIgo0P1xU7/WDOnSmL/8ZwnfLQu9m6ZEJLwo0P1gZjx+SVfaNazNLWNns3LbPq9LEpEIpkD3\nU2JcDC9fnUlcTBTXvP4D23IKvC5JRCKUAj0Amqcl8uqIE9ieU8h1b8wkt0B3kopI8CnQA6Rb81Se\nvbIHCzft4eaxs7XTkYgEnQI9gM48viEPDe3MxKXb+P3HCzSdUUSCyq87ReVIw05syZY9+TwzYQXJ\nCTHcN+R4zMzrskQkAijQq8HtA9qRk1/Mq1NWkxQXze0D23tdkohEAAV6NTAz7j+3I3mFJTw9YQW1\n4mK4sV9rr8sSkTCnQK8mUVHGoxd1Ia+ohMe+XEJiXDQj+mZ4XZaIhDEFejWKjjL+dmk38opKeOCz\nhURHGcP7tPS6LBEJU5rlUs1io6N49soenNmhAfd9skD7kopItVGgB0F8TDTPD+/JWcc34A+fLGDM\ntDVelyQiYcjfLeh+a2YLzWyBmb1rZgmBKizcxMdE8/ywXgzo2JD7P13IG9+v9rokEQkzVQ50M2sK\n/AbIdM51BqKBywNVWDiKi4niuSt7MrBjQx781yJem6JQF5HA8XfIJQaoZWYxQCKgjTaPIS4miueG\n9eTsTg350+eLeEVrqYtIgFQ50J1zG4EngHXAZmCPc+6rQBUWzsq+KO3JOZ0b8fAXixXqIhIQ/gy5\n1AUuAFoBTYAkMxt+lPNGmVmWmWVt26ZNIA6IjY7i6St6MLhLWai/+N1Kr0sSkRDnz5DLWcBq59w2\n51wR8BHQ9/CTnHOjnXOZzrnM9PR0Py4XfmKjo/jH5T04t2vZrkdPjl+mBb1EpMr8ubFoHdDHzBKB\nPOBMICsgVUWQA6GeEBvNP75ZTn5xCfcM6qAFvUSk0qoc6M65GWb2ATAbKAbmAKMDVVgkiY4y/npx\nVxJio3jpu1XkF5bwwHmdiIpSqItIxfl1679z7gHggQDVEtGiooyHLuhMQkw0r0xZTUFxKY9c2IVo\nhbqIVJDWcqlBzIzfDzmeWnHRPDNhBfsKivnLxV1Jjtf/TCJybEqKGsbMuGNge5LiY/jrl0uYs243\nQ7o2Jie/iK17C6ifHMct/dvSPC3R61JFpIaxYM6qyMzMdFlZ+t60omau2cmj/17Mgo17qFMrlvTa\nCazdkUtsdBQvDOtJ3zb1vS5RRILAzGY55zKPeZ4CPbSs2Z7Lr96axdqduYy57kR6t0rzuiQRqWYV\nDXStthhiMuon8c7IE2mSWouRY7JYvT3X65JEpIZQoIeg+snxvHltb6IMRo7JIie/yOuSRKQGUKCH\nqOZpiTw3rCert+dy27i5lJTqDlORSKdAD2F9W9fngfM68s2SbP721VKvyxERj2naYoi7qk9LFm/e\ny/PfrqRTkzoM6drY65JExCPqoYc4M+PB8zvRs0Uqd74/jyVb9npdkoh4RIEeBuJjonlxeC9qJ8Qw\naswsdu8v9LokEfGAAj1MNEhJ4IXhvdi8J4/f6EtSkYikQA8jvVrW5Y/nd2bSsm08M2G51+WISJAp\n0MPMFb2bc1HPpvzjm+V8t0w7RIlEEgV6mDEzHhnahXYNanPbuDls2p3ndUkiEiQK9DBUKy6aF4b3\npKjEcdM7syksLvW6JBEJAgV6mDouPZnHLu7K3PW7efiLRV6XIyJBUOVAN7P2Zja33M9eM7stkMWJ\nf4Z0bcz1p7RizLS1vDdzndfliEg182dP0aVAdwAziwY2Ah8HqC4JkHvO6cDSrTnc98kC2jRIpldL\nLbcrEq4CNeRyJrDSObc2QJ8nARITHcUzV/SgSWotfvXWbDbv0ZekIuEqUIF+OfDu0V4ws1FmlmVm\nWdu2aRqdF1IT43j56kzyCov51VuzyC8q8bokEakGfge6mcUB5wPvH+1159xo51ymcy4zPT3d38tJ\nFbVrWJsnL+vO/A17+L+PfiSYO1WJSHAEood+DjDbObc1AJ8l1Whgp0bcPqAdH83ZyKtTVntdjogE\nWCAC/Qp+YrhFap6bz2jDOZ0b8ei/FzN5uYbARMKJX4FuZknAAOCjwJQj1S0qynjikm60a1ibm8fO\nYY32JBUJG34FunMu1zlXzzm3J1AFSfVLio9h9FWZmG9P0n0FxV6XJCIBoDtFI1SLeok8d2VPVm3P\n5bfvzaVUy+2KhDwFegQ7uU197htyPOMXbeW5iSu8LkdE/KRAj3DX9M3gwh5N+fvXy5i4NNvrckTE\nDwr0CGdmPHphFzo0SuG2cXO13K5ICFOgS9lyu8N6UlxSym/f0/Z1IqFKgS4AZNRP4o8XdGbG6p28\n+N1Kr8sRkSpQoMtBF/dsyrldG/Pk+GXMXb/b63JEpJIU6HKQmfHIhV1omJLArePmaH66SIhRoMsh\n6tSK5anLu7N+534e+HSh1+WISCUo0OUIJ2SkcXP/tnw4ewOfzdvkdTkiUkEKdDmq3/RvQ6+Wdbn3\nw/ks3ZLjdTkiUgEKdDmqmOgonruyJ8kJMVz3xky27ytg4+48bY4hUoNZMDc6yMzMdFlZWUG7nvhv\n/obdXPrSNIpLHMW++emDuzTikaFdqJsU53F1IpHBzGY55zKPdZ566PKzujZL5Z3rTzwY5gD//nEL\nPR4arwW9RGoYBbocU6+WaTx2cZcjjj89Ybm2shOpQRToUiGXndCCt3954iHHnvp6OdNW7fCoIhE5\nnAJdKuyUtvVZ9ehg7j+348FjV748g3U79ntYlYgc4O8WdKlm9oGZLTGzxWZ2UqAKk5opKsq47pRW\n/PD7M8molwjAy5NXUVxS6nFlIuJvD/0fwJfOuQ5AN2Cx/yVJKGhQO4F3R/WhYUo8b01fy+P/Xep1\nSSIRr8qBbmZ1gNOAVwGcc4XOOa3oFEEa16nFQxd0BuClSatYvlU3IIl4yZ8eeitgG/C6mc0xs1fM\nLOnwk8xslJllmVnWtm3b/Lic1EQDOzXid2e3B2DAk5M8rkYksvkT6DFAT+AF51wPIBe45/CTnHOj\nnXOZzrnM9PR0Py4nNdU5nRsdfPzPrPUeViIS2fwJ9A3ABufcDN/zDygLeIkwx6Un896oPgDc9cF8\ntu7N97gikchU5UB3zm0B1ptZe9+hM4FFAalKQk7XZqkHH//2vbla80XEA/7OcrkFeMfM5gPdgUf9\nL0lCUa24aJY8NAiAqSt38C8tuysSdH4FunNurm98vKtzbqhzblegCpPQkxAbzeAuZePpv/tgPvO0\njZ1IUOlOUQmo54f1onGdBACe+Epz00WCSYEuAff0FT0AmLx8O/d98qPH1YhEDgW6BNwJGWm8dk3Z\n0s1vT1/H5/M1ni4SDAp0qRb9OzQ8OJXx5rFz2LJHUxlFqpsCXarNicfV46nLugPQ58/fsGZ7rscV\niYQ3BbpUq6E9mvLw0LL1XkaOydJSuyLVSIEu1W54n5Y8e2UPNu3O41dvz2JnbqHXJYmEJQW6BMW5\nXZvw7LCerNy2j4tfmMrmPXlelyQSdhToEjRntG/A2OtPZFtOAZe+NI1lWm5XJKAU6BJUmRlpvPXL\n3uQXlTL8lRnM0J6kIgGjQJeg69GiLm9e25uk+BiufWMm72vJXZGAUKCLJzo2SWHcqD50bJzC7z6Y\nzz0fzievUCs0ivhDgS6eaZhSti/pdSe34r2s9Qx+ejKz1mp9N5GqUqCLp2Kjo7j/vI68fs0JFJeW\n8osXp3L/pwvYV1DsdWkiIUeBLjVCv/YN+PzmUxlxUgZjpq2l/xPf8uGsDZSWOq9LEwkZCnSpMeok\nxvLg+Z34+Ka+NEmtxR3vz+OSl6bx44Y9XpcmEhL8CnQzW2NmP5rZXDPLClRREtl6tKjLRzf25a+/\n6Mqa7bmc9+wUfj12Nqu27fO6NJEaLSYAn3GGc257AD5H5KCoKOPSzOYM6tyIlyet4pXJq/lywRYu\nzWzGb85sS+M6tbwuUaTG0ZCL1GgpCbHcMbA9k+46g6v6tOSDWRs4/fFveeSLRezSmjAihzDnqv6l\nk5mtBvYAJcBLzrnRP3d+Zmamy8rSyIxU3fqd+3ny62V8PGcjSXExjDz1OH55aiuS4wPxx6ZIzWRm\ns5xzmcc8z89Ab+qc22hmDYDxwC3OuUmHnTMKGAXQokWLXmvXrq3y9UQOWLY1hyf+u5SvFm2lXlIc\nvz6jDcP6tCA+Jtrr0kQCLiiBftgFHwT2Oeee+Klz1EOXQJu9bhePf7mUaat20DS1Fred1ZaLejYj\nOsq8Lk0kYCoa6FUeQzezJDOrfeAxMBBYUNXPE6mKni3qMnbkibz1y96kJcXxuw/mc/ZTk/hywWYC\n1VkRCRX+fCnaEJhiZvOAH4AvnHNfBqYskYozM05tm85nN5/MC8N64pzjhrdnM/S57/l+ReRMwLp1\n3BwufP57r8sQD1X5myTn3CqgWwBrEfGLmXFOl8YM6NiQj2Zv5KmvlzHslRmc3KYed53dgW7NU70u\nsVp9OneT1yXUGLv3F5KaGOd1GUGnaYsSdmKio7j0hOZMuLMf9w05nsWbc7jgue+58e1ZrMjWzUnh\nbumWHLr/aTwfztrgdSlBp0CXsJUQG831px7Hd7/rx61ntmXSsm0MfPI77v5gvrbAC2PrdpZtRP7c\ntys8riT4NHlXwl7thFh+O6AdV53UkucmruCd6ev4eO5GrumbwY2nt6ZuUuT9aR6OFm7aQ3ZOAfsK\nigBYtS3X44qCTz10iRj1k+N54LxOfHPH6ZzXtQkvT17FaX+dyLMTlpOr5XpD3pCnp3Dt6zPZmVt0\n8FhBcWRtmqJAl4jTPC2Rv13ajS9vPY0+revxxFfLOP3xbxkzbQ2FxaVelyd+Wrblf5uPz123m+y9\n+R5WE1wKdIlY7RvV5uWrM/nwxr4cl57E/Z8u5My/f8unczeG9DrsxSWR/R+lqav+N1X1stHT6f3o\nNx5WE1wKdIl4vVrW5b1RfXjj2hOoHR/LrePmcu4zU/h2aXZI3pxUGKGB3jS1bAXO9TvziInQO4UV\n6CKUzWHv174Bn99yCv+4vDs5BUVc8/pMrnh5OnPWhdY+pwVFkRno5Zd7KD7sL6z8osgYS1egi5QT\nFWVc0L0p39zejz+e34kV2fu48Pmp3PBW6MxhL4jQ7wH2FxZzSa9mAPTOSOOhoZ0PvrZh136vygoq\nTVsUOYq4mChG9M3g4l7NeHXyakZPWsn4xVu5pFczbjurHY3qJHhd4iHKj5tH2syOA/YVFJOaGMvs\nPwwgLiaK5PgYOjVJ4aLnp7J2x37aNKjtdYnVTj10kZ+RHB/DrWe15TvfBhsfzt7A6Y9P5C//WcKe\n/UXH/oAgKd8rj8QeenFJKflFpSTFx5CWFHdwffwD4+qb90TGTBcFukgF1E+O58HzOzHhjn4M7tKY\nlyat5NS/TuDF71bWiPHZQwI9AsfQ9/v+Nzh8o5N6SXFEGREzdVGBLlIJzdMSefKy7nxxy6n0bFmX\nv/xnCf0e/5b3Zq7zdLpg+WGWvBrwH5hgO3BjWGLcoYEeEx1F/eR4tu4t8KKsoFOgi1RBxyYpvHFt\nb8aN6kOjOgnc/eGPvnXYt3gy1bF8r/zSl6ZRFEFTFwuKS1i3o+xLz6T4I3esapAST3aOeugicgx9\njqvHxzf15cXhvQC44e1ZXPTCVKav2hHUOg6fe743r+aM71en/YXFnPDw11w2ejpw5JALQMPaCeqh\ni0jFmBmDOjfiv7edxmMXd2Hz7nwuHz2da17/gYWb9gSlhq2HjRHn5Ifn2jTOOZ7+Zjmvf78agGkr\nd7C3XFtrJ8Qe8Z5I6qFr2qJIgMRER3HZCS24oHtT3pi6hucnrmDI01MY1KkRtw1oS4dGKdVy3dyC\nYq569YdDjoVroP934Rb+Pn4ZAF8u2EKLtMRDXu/StM4R72lQO4Ht+wopKC4J+03E/e6hm1m0mc0x\ns88DUZBIqEuIjeaG01sz+e7+/ObMtny/YjuDnprMr9+ZzbKtOcf+gEp6e/raI47tzitk5Jgspq4M\nry34vl6cffDxjNU7eX/WBk5pU//gsVpxRwZ2w5Syewba3/clq7eH95K6gRhyuRVYHIDPEQkrdWrF\ncvuAdky++wxuPqMN3y7N5uynJnHLu3NYkR2YYM8rLGH0pFVHHF+1LZfxi7byqzGzAnKdmmDT7jzG\nL9rKOZ0b8fgvutKuYTIAPVqk8u7IPnx9++lHfV+D2vEHH89bvzsotXrFr0A3s2bAEOCVwJQjEn5S\nE+O48+z2TL67Pzec3ppvFm9lwJOTuG3cHFZt8285gc/mbWRHbiEvDOvJtSdn8OszWgMc/FwLozWq\nXvh2JXvyirj+1OO4JLM5vVqmAdCmQTInta5HmwbJR33fgR46wJ4w/7LY3zH0p4C7gJ+8p9bMRgGj\nAFq0aOHn5URCV1pSHHcP6sD1p7Ri9ORVjJm6ls/mbeKcLo351WnH0bVZ5TaxLil1vDJ5NR0a1WZQ\n50ac06Uxe/YX8dzElaw4GOjhk+hrduTStVkderWsC8Ddg9pTNzGWszs1+tn3NUj5Xw/9wPZ04arK\nPXQzOxfIds797N90zrnRzrlM51xmenp6VS8nEjbqJcdz7znHM/nuMxh52nFMWrqN85/9nitGT2di\nJZbs/XjORpZn7+M3Z7Y9GNzJCWV9tO9XlE2b3JNXxN788OiVbtydR7O6tQ4+T02M465BHUiI/fkv\nOuuV22JwvQL9J50MnG9ma4BxQH8zezsgVYlEgPq+YJ96b39+P/h4Vm/P5drXZ3Lm377jxe9Wsi3n\np+dO78wt5G9fLaVL0zqc0/l/PdToo6wDPvTZ76ul/mByzrFxV97BtVkqIyY6imtPzgDCv4de5SEX\n59y9wL0AZtYPuNM5NzxAdYlEjNoJsYw87ThG9M3g8/mbGPfDev7ynyU88d+l9G1Tn/7t0+neoi6t\n6ifhnOOH1Tt57Msl7Mgt5MXhvY45rLJqey7OuZAefimbdlhKs7qJxz75KB44rxPOwftZ60P+d/Fz\nNA9dpIaIi4niop7NuKhnM1Zk7+P9WesZv3ArD/5r0RHnNq6TwJvX9qZb8yPH3V8Y1pPJK7bz8eyN\nB9d1+XpxNgM6Nqz2NlSX92auA6Bdw6ovgds8LZHcwhKycwqonxx/1L9mQp0Fc92JzMxMl5WVFbTr\niYSDdTv2s3jLXtbuyMUwOjZJITOj7jFvkiktdUxfvYMrX54BwFe/Pc2vQPRSv8cn0jwtkTHX9a5y\n73r8oq2MHFOWPye3qcc71/cJZInVysxmOecyj3WeeugiNVyLeom0qFf5oYaoKKNv6/qMHXkiN749\nm+vemMlLV/WiU5Mj76asyYpKSlm/K48hXRv7NVRS/q7S71fsYPu+sp56ONFaLiJhrm/r+oy5rjf7\nC0sY9sqMarlbtTqt37mfklJHq/pHn2deUU1SD91laubqnX59Xk2kHrpIBOjWPJX7z+3Ibe/NZeCT\nk3h+WE/O6dzoYI83v6iErDW7WLx5L8uzc9hXUEyUGfEx0bRtmMzxjVPIqJdIy3pJQa/94S/KbkRv\nVb9qX4hjaGzVAAAJcklEQVQecPjCXYs27+WcLo39+syaRoEuEiEGd2nMzDU7eWfGOm56ZzZ1asVS\nPzmOKDOWV2ID7BMy6nLdya3o3SqNetU8ZLF1bz4TlpSt3xLoPUEXbdpLv8cn0qNFXZ68rHtAP9sr\nCnSRCBEXE8UjF3bhwh5NeXnyKtbtzMOARnUS6NWyLjkFxaQkxFJa6oiJNqKjjP2FJeQVlbBi6z6W\n+oZqZq7Zxcw1uwDo3DSF4Se2ZGiPpse8waey8otKuGXsHADev+Ek6tQ6cmncqmqdnsSizXvZvCef\nNTv2K9BFJDRlZqSRmZFW6feVlDqWZ+cw7of1vDF1DQALNu7lno9+5J6PfqRb81RuH9COHi1SSTnK\nuuTHUlBcwr78YnLyi5m7fjf3f7rg4FrnR1sW1x8nZKQxbub6gH5mTaBAF5EKiY4yOjRK4cHzO/HA\neR2ZtnIHb89Yy79/3AKUrWQ44rWyddkv6dWM09unk9kyjQa144mKMpxz3PXBfPbkFTGka2NOaVOf\nesnxZK3ZyWvfrz74OYe7qV/rgPX+T25Tj+9X7KBHi9RDAj2/qCTgf2F4QfPQRcQv2Xvz+XLhFu7/\ndOFPnnNmhwbs2l/I7HWVW772zoHtuLl/W39LPGhfQTHrd+5nf2ExF78w7eDxr28//SdXa6wJNA9d\nRIKiQUoCV5+UwaWZzfl4zkYe/WIxOQWH7pj0je+LzfTa8UesUdOpSQpjR/YhJSGGvXnFZK0tm06Y\nmZEW0HFzKNtz9PjGKezYd2gNK7L31ehAryj10EUk4KYs386EJdl8Pn8T2TkFRBncPagDw/q0JDk+\nhglLtjJp2XaapyXyy1NaeVLjt0uz6dK0Dn3/MoErT2zBA+d18qSOiqhoD12BLiIRbdgr0/l+xQ6a\n1a3Fhzf2PWRDjJqiooGuO0VFJKIN8m2QsWFXHk99vczjavyjQBeRiDa0R1NifCsv/jNrg9/bAnpJ\nQy4iEvEKi0vZk1fE6Y9PpKiklIYpCYy9vk+VFkWrDhpyERGpoLiYKNJrx3NL/7YUlTg27MrjoS8W\nUVhcWuHPyM7JJ9+3/rxXFOgiIj439mvN/AcH8n+DOzB+0VYuGz2NxZv3HvN9y7bm0PuRb7j29ZlB\nqPKnKdBFRMpJSYhl1GmtuTSzGXPW7WbI05N5dcrqn9y8e+mWHAY+OQmAaat2BLPUI1Q50M0swcx+\nMLN5ZrbQzP4YyMJERLz0yIVd+OI3pzCgY0Me+nwRd7w/jy178o84b8OumrPxtD899AKgv3OuG9Ad\nGGRmobOnk4jIz4iNjqJTkzq8MKwXt/RvwydzNnLaXyfy1vS1FJX8b2x982EhX1Ds3Th6lW/9d2V/\nfxyY3xPr+wnelBkRkSCIijLuGNieSzObc8f78/jDJwsYM3UNvz6jDYXFpdz3yYJDzl+wcQ/tG6WQ\nHB/8lVX8mrZoZtHALKAN8Jxz7u6jnDMKGAXQokWLXmvXrq3y9UREvFRcUsrYH9YddSGyf1zenfs+\nWUBOfjF1asUy6a4zArYWTVCmLTrnSpxz3YFmQG8z63yUc0Y75zKdc5np6en+XE5ExFMx0VFcfVIG\ns/8wgBMy6h48fmKrNC7o3pTzuzUBYE9eEZ/N3Rj8+gLxIc653WY2ERgELDjW+SIioSwtKY73b+jL\nhCVbadugNk1TawFlG2e8M2MdAONmrqdxnVr0almXuklxQanLn1ku6WaW6ntcCxgALAlUYSIiNV3/\nDg1pnpZIlG/pgF4ty3rtjVISWLhpL9ePyaLHQ+OZunJ7UOrxZ8ilMTDRzOYDM4HxzrnPA1OWiEjo\naZ6WyPf39Oc/t556yPHfvT//J+exB5I/s1zmAz0CWIuISMg7MPxyYY+mLNuaw6lt03nxu5Us2ZLD\n8Y1TqvXa2rFIRKQa/P3SbpgZO/YVsHjz3kPmrlcXBbqISDUwKxtXr5ccz5vX9Q7KNbWWi4hImFCg\ni4iECQW6iEiYUKCLiIQJBbqISJhQoIuIhAkFuohImFCgi4iECb/WQ6/0xcy2AVVdEL0+EJwVbmoO\ntTkyqM2RwZ82t3TOHXP98aAGuj/MLKsiC7yHE7U5MqjNkSEYbdaQi4hImFCgi4iEiVAK9NFeF+AB\ntTkyqM2RodrbHDJj6CIi8vNCqYcuIiI/Q4EuIhImQiLQzWyQmS01sxVmdo/X9QSKmb1mZtlmtqDc\nsTQzG29my33/1i332r2+38FSMzvbm6qrzsyam9lEM1tkZgvN7Fbf8XBuc4KZ/WBm83xt/qPveNi2\n+QAzizazOWb2ue95WLfZzNaY2Y9mNtfMsnzHgttm51yN/gGigZXAcUAcMA/o6HVdAWrbaUBPYEG5\nY38F7vE9vgd4zPe4o6/t8UAr3+8k2us2VLK9jYGevse1gWW+doVzmw1I9j2OBWYAfcK5zeXafjsw\nFvjc9zys2wysAeofdiyobQ6FHnpvYIVzbpVzrhAYB1zgcU0B4ZybBOw87PAFwJu+x28CQ8sdH+ec\nK3DOrQZWUPa7CRnOuc3Oudm+xznAYqAp4d1m55zb53sa6/txhHGbAcysGTAEeKXc4bBu808IaptD\nIdCbAuvLPd/gOxauGjrnNvsebwEa+h6H1e/BzDKAHpT1WMO6zb6hh7lANjDeORf2bQaeAu4Cyu+M\nHO5tdsDXZjbLzEb5jgW1zdokugZzzjkzC7t5pWaWDHwI3Oac23tgM10IzzY750qA7maWCnxsZp0P\nez2s2mxm5wLZzrlZZtbvaOeEW5t9TnHObTSzBsB4M1tS/sVgtDkUeugbgeblnjfzHQtXW82sMYDv\n32zf8bD4PZhZLGVh/o5z7iPf4bBu8wHOud3ARGAQ4d3mk4HzzWwNZUOk/c3sbcK7zTjnNvr+zQY+\npmwIJahtDoVAnwm0NbNWZhYHXA585nFN1ekzYITv8Qjg03LHLzezeDNrBbQFfvCgviqzsq74q8Bi\n59zfy70Uzm1O9/XMMbNawABgCWHcZufcvc65Zs65DMr+/zrBOTecMG6zmSWZWe0Dj4GBwAKC3Wav\nvxmu4LfHgymbEbES+L3X9QSwXe8Cm4EiysbQfgnUA74BlgNfA2nlzv+973ewFDjH6/qr0N5TKBtn\nnA/M9f0MDvM2dwXm+Nq8ALjfdzxs23xY+/vxv1kuYdtmymbhzfP9LDyQU8Fus279FxEJE6Ew5CIi\nIhWgQBcRCRMKdBGRMKFAFxEJEwp0EZEwoUAXEQkTCnQRkTDx//yhIhSh/U6VAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x6e45d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#################################\n",
    "### Author: Paul Soto \t\t  ###\n",
    "### \t\tpaul.soto@upf.edu ###\n",
    "#\t\t\t\t\t\t\t\t#\n",
    "# This file shows how to use a ##\n",
    "# LSTM network with word vectors#\n",
    "# to predict the next word in a #\n",
    "# sequence. It is based off of ##\n",
    "# the code from Nicolas Jimenez #\n",
    "# https://github.com/nicodjimenez\n",
    "#################################\n",
    "\n",
    "import numpy as np \n",
    "import itertools\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def loss(pred, label, deriv=False):\n",
    "\tif deriv:\n",
    "\t\tdiff = (pred - label)\n",
    "\t\treturn diff\n",
    "\treturn 0.5*sum((pred - label) ** 2)\n",
    "\n",
    "def sigmoid(x, deriv=False):\n",
    "\tif deriv:\n",
    "\t\treturn x*(1-x) \n",
    "\treturn 1. / (1 + np.exp(-x))\n",
    "\n",
    "def dtanh(x): \n",
    "\treturn 1. - x ** 2\n",
    "\n",
    "def softmax(x, deriv=False):\n",
    "\tif deriv:\n",
    "\t\tvec = x.reshape((-1,1))\n",
    "\t\tjac = np.diag(x) - np.dot(vec, vec.T)\n",
    "\t\treturn jac\n",
    "\tx = x-x.max()\n",
    "\treturn np.exp(x)/np.exp(x).sum()\n",
    "\n",
    "class LSTM:\n",
    "\t\"\"\"Class initiates an LSTM model with a forget, input and output gate\n",
    "\tand softmax activation function\n",
    "\t\"\"\"\n",
    "\tdef __init__(self, x_size, c_size):\n",
    "\t\t\"\"\"\n",
    "\t\tx_size is size of input\n",
    "\t\tc_size is size of cell states and hidden states (which is output)\n",
    "\t\t\"\"\"\n",
    "\t\tself.x_size = x_size\n",
    "\t\tself.c_size = c_size\n",
    "\t\t# Theta matrices and bias terms for gates and output\n",
    "\t\ttheta_g = np.random.rand(c_size,x_size+c_size)\n",
    "\t\ttheta_i = np.random.rand(c_size,x_size+c_size)\n",
    "\t\ttheta_f = np.random.rand(c_size,x_size+c_size)\n",
    "\t\ttheta_o = np.random.rand(c_size,x_size+c_size)\n",
    "\t\ttheta_h = np.random.rand(c_size,c_size)\n",
    "\t\tb_i = np.random.rand(c_size)\n",
    "\t\tb_f = np.random.rand(c_size)\n",
    "\t\tb_o = np.random.rand(c_size)\n",
    "\t\tb_g = np.random.rand(c_size)\n",
    "\t\tb_h = np.random.rand(c_size)\n",
    "\t\tself.params = {'theta_g':theta_g,'theta_i':theta_i, 'theta_f':theta_f, \n",
    "\t\t\t\t\t\t'theta_o':theta_o,'theta_h':theta_h, 'b_i':b_i, 'b_f':b_f,\n",
    "\t\t\t\t\t\t'b_o':b_o, 'b_g':b_g, 'b_h':b_h}\n",
    "\t\t### Differentials for each parameter \n",
    "\t\tself.deltas = {}\n",
    "\t\tfor param in self.params.keys():\n",
    "\t\t\tself.deltas[param+\"_delta\"] = np.zeros_like(self.params[param])\t\n",
    "\n",
    "\tdef differentiate(self, lr = 1):\n",
    "\t\t### Apply the deltas to each weight \n",
    "\t\tfor param in self.params:\n",
    "\t\t\tself.params[param] -= lr*self.deltas[param+\"_delta\"]\n",
    "\t\t\tself.deltas[param+\"_delta\"] = np.zeros_like(self.params[param])\n",
    "\n",
    "class LSTM_node:\n",
    "\t\"\"\"\n",
    "\tClass for forward propogated and backward propogating cell\n",
    "\t\"\"\"\n",
    "\tdef __init__(self,param):\n",
    "\t\tself.param = param\n",
    "\n",
    "\tdef forward_prop(self,x,h_prev,c_prev):\n",
    "\t\t\"\"\"\n",
    "\t\tx is input at time step t\n",
    "\t\th_prev is hidden state (output) at time t-1\n",
    "\t\tc_prev is cell state at time t-1\n",
    "\t\t\"\"\"\n",
    "\t\txh = np.hstack((x,h_prev))\n",
    "\t\tself.xh = xh\n",
    "\t\tself.i = sigmoid(np.dot(self.param.params[\"theta_i\"],xh)+self.param.params[\"b_i\"])\n",
    "\t\tself.f = sigmoid(np.dot(self.param.params[\"theta_f\"],xh)+self.param.params[\"b_f\"])\n",
    "\t\tself.o = sigmoid(np.dot(self.param.params[\"theta_o\"],xh)+self.param.params[\"b_o\"])\n",
    "\t\tself.g = np.tanh(np.dot(self.param.params[\"theta_g\"],xh)+self.param.params[\"b_g\"])\n",
    "\t\tself.c = self.f*c_prev+self.i*self.g\n",
    "\t\tself.z = self.o*np.tanh(self.c)\n",
    "\t\tself.h = softmax(np.dot(self.param.params['theta_h'],self.z)+self.param.params[\"b_h\"])\n",
    "\t\tself.h_prev = h_prev\n",
    "\t\tself.c_prev = c_prev\n",
    "\n",
    "\tdef back_prop(self,dh, dc_next):\n",
    "\t\t\"\"\"\n",
    "\t\tPerforms gradient descent\n",
    "\t\tdh is the derivate of the loss function wrt to h \n",
    "\t\tdc_next is the derivative of the loss function wrt to c_{t+1}\n",
    "\t\t\"\"\"\n",
    "\t\tdz = np.dot(softmax(self.h,deriv=True),dh)\n",
    "\t\tdc =np.dot(self.param.params['theta_h'].T,dz)*self.o*dtanh(self.c) + dc_next\n",
    "\t\tdo = np.dot(self.param.params['theta_h'].T,dz)*np.tanh(self.c)*sigmoid(self.o,deriv=True)\n",
    "\t\tdi = self.g*sigmoid(self.i,deriv=True)*dc\n",
    "\t\tdf = self.c_prev*sigmoid(self.f, deriv=True)*dc\n",
    "\t\tdg = self.i*dtanh(self.g)*dc\n",
    "\t\tself.param.deltas['theta_o_delta'] += np.outer(do, self.xh)\n",
    "\t\tself.param.deltas['theta_g_delta']  += np.outer(dg, self.xh)\n",
    "\t\tself.param.deltas['theta_i_delta']  += np.outer(di, self.xh)\n",
    "\t\tself.param.deltas['theta_f_delta']  += np.outer(df, self.xh)\n",
    "\t\tself.param.deltas['theta_h_delta']  += np.outer(dz,self.o*np.tanh(self.c))\n",
    "\t\tself.param.deltas['b_i_delta']  += di\n",
    "\t\tself.param.deltas['b_f_delta'] += df      \n",
    "\t\tself.param.deltas['b_o_delta'] += do\n",
    "\t\tself.param.deltas['b_g_delta'] += dg  \n",
    "\t\tself.param.deltas['b_h_delta'] += dz  \n",
    "\n",
    "\t\tdxh = np.zeros_like(self.xh)\n",
    "\t\tdxh += np.dot(self.param.params['theta_i'].T, di)\n",
    "\t\tdxh += np.dot(self.param.params['theta_f'].T, df)\n",
    "\t\tdxh += np.dot(self.param.params['theta_o'].T, do)\n",
    "\t\tdxh += np.dot(self.param.params['theta_g'].T, dg)\n",
    "\t\t# Pass the dL/dc to time step at t-1\n",
    "\t\t# Pass the dL/dh to time step at t-1\n",
    "\t\tself.dc_prev = dc * self.f\n",
    "\t\tself.dh_prev = dxh[self.param.c_size:]\n",
    "\n",
    "def predict_word(sentence, lstm_param):\n",
    "\tx_pred_list = []\n",
    "\ty_pred_list = []\n",
    "\tfor sent in [sentence]:\n",
    "\t\tx_pred_list.append(map(lambda x: vectors[np.where(words==x)[0][0]],\n",
    "\t\t\t\tsent.split()))\n",
    "\tlstm_node_list = []\n",
    "\t# Initiate the LSTM cells and forward propogate\n",
    "\tfor ex_ind in range(len(x_pred_list)):\n",
    "\t\texample = x_pred_list[ex_ind]\n",
    "\t\tfor ind in range(len(example)):\n",
    "\t\t\tif len(lstm_node_list)<len(example):\n",
    "\t\t\t\tlstm_node_list.append(LSTM_node(param=lstm_param))\n",
    "\t\t\t\tif ind==0: \n",
    "\t\t\t\t\tc_prev = np.zeros(c_size)\n",
    "\t\t\t\t\th_prev = np.zeros(c_size)\n",
    "\t\t\t\telse:\n",
    "\t\t\t\t\tc_prev = lstm_node_list[ind-1].c\n",
    "\t\t\t\t\th_prev = lstm_node_list[ind-1].h\n",
    "\t\t\t\tlstm_node_list[ind].forward_prop(example[ind],h_prev, c_prev)\n",
    "\t# Return largest probability word\n",
    "\tprobs= lstm_node_list[-1].h\n",
    "\tprobs_max = np.where(probs==max(probs))\n",
    "\treturn sentence,words[probs_max][0]\n",
    "\n",
    "\n",
    "docs = [\"we think uncertainty about unemployment\",\n",
    "\t\t\"uncertainty and fears about inflation\",\n",
    "\t\t\"we think fears about unemployment\",\n",
    "\t\t\"we think fears and uncertainty about inflation and unemployment\",\n",
    "\t\t\"constant negative press covfefe\"]\n",
    "\n",
    "# Split each document into a list of words\n",
    "docs_split = map(lambda x: x.split(),docs)\n",
    "docs_words = list(itertools.chain(*docs_split))\n",
    "\n",
    "# Find unique words across all documents\n",
    "words = np.unique(docs_words)\n",
    "\n",
    "# Generate a one hot encoded vector for each unique word\n",
    "vectors = np.eye(words.shape[0])\n",
    "x_list = []\n",
    "y_list = []\n",
    "for sent in docs_split:\n",
    "\tif len(sent)<2:\n",
    "\t\tcontinue\n",
    "\tx_list.append(map(lambda x: vectors[np.where(words==x)[0][0]],sent[:-1]))\n",
    "\ty_list.append(map(lambda y: vectors[np.where(words==y)[0][0]],sent[1:]))\n",
    "\n",
    "# Initiate the LSTM parameters\n",
    "x_size  = vectors.shape[0]\n",
    "c_size  = vectors.shape[0]\n",
    "np.random.seed(0)\n",
    "lstm_param = LSTM(x_size, c_size) \n",
    "\n",
    "# Training\n",
    "lstm_node_list = []\n",
    "losses = []\n",
    "for epoch in range(500):\n",
    "\tprint \"epoch number %s\" % epoch\n",
    "\t# Loop through each sentence\n",
    "\tfor ex_ind in range(len(x_list)):\n",
    "\t\texample = x_list[ex_ind]\n",
    "\t\toutput = y_list[ex_ind]\n",
    "\t\t# Each word in the sentence will be a timestep\n",
    "\t\t# Initiate the lstm_nodes\n",
    "\t\tfor ind in range(len(example)):\n",
    "\t\t\tif len(lstm_node_list)<len(example):\n",
    "\t\t\t\tlstm_node_list.append(LSTM_node(param=lstm_param))\n",
    "\t\t\t\tif ind==0: \n",
    "\t\t\t\t\tc_prev = np.zeros(c_size)\n",
    "\t\t\t\t\th_prev = np.zeros(c_size)\n",
    "\t\t\t\telse:\n",
    "\t\t\t\t\tc_prev = lstm_node_list[ind-1].c\n",
    "\t\t\t\t\th_prev = lstm_node_list[ind-1].h\n",
    "\t\t\t\tlstm_node_list[ind].forward_prop(example[ind],h_prev, c_prev)\n",
    "\t\t# Backpropogate and update the parameters in lstm_param\n",
    "\t\tidx = len(example)-1\n",
    "\t\tif ex_ind ==0: curr_loss = 0\n",
    "\t\tcurr_loss += loss(lstm_node_list[idx].h,output[idx])\n",
    "\t\tdh_curr = loss(lstm_node_list[idx].h,output[idx],deriv=True)\n",
    "\t\tlstm_node_list[idx].back_prop(dh_curr, np.zeros(c_size))\n",
    "\t\tidx-=1\n",
    "\t\twhile idx>=0:\n",
    "\t\t\tcurr_loss += loss(lstm_node_list[idx].h,output[idx])\n",
    "\t\t\tdh = loss(lstm_node_list[idx].h,output[idx],deriv=True)\n",
    "\t\t\tdh += lstm_node_list[idx + 1].dh_prev\n",
    "\t\t\tdc = lstm_node_list[idx + 1].dc_prev\n",
    "\t\t\tlstm_node_list[idx].back_prop(dh, dc)\n",
    "\t\t\tidx-=1\n",
    "\t\t# To stop exploding gradient clip the delta values\n",
    "\t\tfor param in lstm_param.deltas:\n",
    "\t\t\tlstm_param.deltas[param] = np.clip(lstm_param.deltas[param],-1,1)\n",
    "\t\tlstm_param.differentiate(lr=0.1)\n",
    "\t\tlstm_node_list = []\n",
    "\tlosses.append(curr_loss)\n",
    "\n",
    "# Let's see how well the model predicts the next word in the following sentences\n",
    "print \"'we': %s\" % predict_word(\"we\", lstm_param)[1]\n",
    "print \"'we think': %s\" % predict_word(\"we think\", lstm_param)[1]\n",
    "print \"'uncertainty and': %s\" % predict_word(\"uncertainty and\", lstm_param)[1]\n",
    "print \"'uncertainty and fears about': %s\" % predict_word(\"uncertainty and fears about\", lstm_param)[1]\n",
    "\n",
    "plt.plot(losses)\n",
    "plt.title(\"Loss for LSTM\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}