Neural_machine_translation_with_attention_v4a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Machine Translation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load all the packages you will need for this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply\n",
    "from keras.layers import RepeatVector, Dense, Activation, Lambda\n",
    "from keras.optimizers import Adam\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import load_model, Model\n",
    "import keras.backend as K\n",
    "import numpy as np\n",
    "\n",
    "from faker import Faker\n",
    "import random\n",
    "from tqdm import tqdm\n",
    "from babel.dates import format_date\n",
    "from nmt_utils import *\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Translating human readable dates into machine readable dates\n",
    "\n",
    "* The model you will build here could be used to translate from one language to another, such as translating from English to Hindi. \n",
    "* However, language translation requires massive datasets and usually takes days of training on GPUs. \n",
    "* To give you a place to experiment with these models without using massive datasets, we will perform a simpler \"date translation\" task. \n",
    "* The network will input a date written in a variety of possible formats (*e.g. \"the 29th of August 1958\", \"03/30/1968\", \"24 JUNE 1987\"*) \n",
    "* The network will translate them into standardized, machine readable dates (*e.g. \"1958-08-29\", \"1968-03-30\", \"1987-06-24\"*). \n",
    "* We will have the network learn to output dates in the common machine-readable format YYYY-MM-DD. \n",
    "\n",
    "<!-- \n",
    "Take a look at [nmt_utils.py](./nmt_utils.py) to see all the formatting. Count and figure out how the formats work, you will need this knowledge later. !--> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 - Dataset\n",
    "\n",
    "We will train the model on a dataset of 10,000 human readable dates and their equivalent, standardized, machine readable dates. Let's run the following cells to load the dataset and print some examples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10000/10000 [00:00<00:00, 18477.61it/s]\n"
     ]
    }
   ],
   "source": [
    "m = 10000\n",
    "dataset, human_vocab, machine_vocab, inv_machine_vocab = load_dataset(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('9 may 1998', '1998-05-09'),\n",
       " ('10.11.19', '2019-11-10'),\n",
       " ('9/10/70', '1970-09-10'),\n",
       " ('saturday april 28 1990', '1990-04-28'),\n",
       " ('thursday january 26 1995', '1995-01-26'),\n",
       " ('monday march 7 1983', '1983-03-07'),\n",
       " ('sunday may 22 1988', '1988-05-22'),\n",
       " ('08 jul 2008', '2008-07-08'),\n",
       " ('8 sep 1999', '1999-09-08'),\n",
       " ('thursday january 1 1981', '1981-01-01')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape: (10000, 30)\n",
      "Y.shape: (10000, 10)\n",
      "Xoh.shape: (10000, 30, 37)\n",
      "Yoh.shape: (10000, 10, 11)\n"
     ]
    }
   ],
   "source": [
    "Tx = 30\n",
    "Ty = 10\n",
    "X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)\n",
    "\n",
    "print(\"X.shape:\", X.shape)\n",
    "print(\"Y.shape:\", Y.shape)\n",
    "print(\"Xoh.shape:\", Xoh.shape)\n",
    "print(\"Yoh.shape:\", Yoh.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Let's also look at some examples of preprocessed training examples. \n",
    "* Feel free to play with `index` in the cell below to navigate the dataset and see how source/target dates are preprocessed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source date: 9 may 1998\n",
      "Target date: 1998-05-09\n",
      "\n",
      "Source after preprocessing (indices): [12  0 24 13 34  0  4 12 12 11 36 36 36 36 36 36 36 36 36 36 36 36 36 36 36\n",
      " 36 36 36 36 36]\n",
      "Target after preprocessing (indices): [ 2 10 10  9  0  1  6  0  1 10]\n",
      "\n",
      "Source after preprocessing (one-hot): [[ 0.  0.  0. ...,  0.  0.  0.]\n",
      " [ 1.  0.  0. ...,  0.  0.  0.]\n",
      " [ 0.  0.  0. ...,  0.  0.  0.]\n",
      " ..., \n",
      " [ 0.  0.  0. ...,  0.  0.  1.]\n",
      " [ 0.  0.  0. ...,  0.  0.  1.]\n",
      " [ 0.  0.  0. ...,  0.  0.  1.]]\n",
      "Target after preprocessing (one-hot): [[ 0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 1.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  1.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.]]\n"
     ]
    }
   ],
   "source": [
    "index = 0\n",
    "print(\"Source date:\", dataset[index][0])\n",
    "print(\"Target date:\", dataset[index][1])\n",
    "print()\n",
    "print(\"Source after preprocessing (indices):\", X[index])\n",
    "print(\"Target after preprocessing (indices):\", Y[index])\n",
    "print()\n",
    "print(\"Source after preprocessing (one-hot):\", Xoh[index])\n",
    "print(\"Target after preprocessing (one-hot):\", Yoh[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Neural machine translation with attention\n",
    "\n",
    "* If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate. \n",
    "* Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down. \n",
    "* The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step. \n",
    "\n",
    "\n",
    "### 2.1 - Attention mechanism\n",
    "\n",
    "In this part, you will implement the attention mechanism presented in the lecture videos. \n",
    "* Here is a figure to remind you how the model works. \n",
    "    * The diagram on the left shows the attention model. \n",
    "    * The diagram on the right shows what one \"attention\" step does to calculate the attention variables $\\alpha^{\\langle t, t' \\rangle}$.\n",
    "    * The attention variables $\\alpha^{\\langle t, t' \\rangle}$ are used to compute the context variable $context^{\\langle t \\rangle}$ for each timestep in the output ($t=1, \\ldots, T_y$). \n",
    "\n",
    "<table>\n",
    "<td> \n",
    "<img src=\"images/attn_model.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "<td> \n",
    "<img src=\"images/attn_mechanism.png\" style=\"width:500;height:500px;\"> <br>\n",
    "</td> \n",
    "</table>\n",
    "<caption><center> **Figure 1**: Neural machine translation with attention</center></caption>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some properties of the model that you may notice: \n",
    "\n",
    "#### Pre-attention and Post-attention LSTMs on both sides of the attention mechanism\n",
    "- There are two separate LSTMs in this model (see diagram on the left): pre-attention and post-attention LSTMs.\n",
    "- *Pre-attention* Bi-LSTM is the one at the bottom of the picture is a Bi-directional LSTM and comes *before* the attention mechanism.\n",
    "    - The attention mechanism is shown in the middle of the left-hand diagram.\n",
    "    - The pre-attention Bi-LSTM goes through $T_x$ time steps\n",
    "- *Post-attention* LSTM: at the top of the diagram comes *after* the attention mechanism. \n",
    "    - The post-attention LSTM goes through $T_y$ time steps. \n",
    "\n",
    "- The post-attention LSTM passes the hidden state $s^{\\langle t \\rangle}$ and cell state $c^{\\langle t \\rangle}$ from one time step to the next. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### An LSTM has both a hidden state and cell state\n",
    "* In the lecture videos, we were using only a basic RNN for the post-attention sequence model\n",
    "    * This means that the state captured by the RNN was outputting only the hidden state $s^{\\langle t\\rangle}$. \n",
    "* In this assignment, we are using an LSTM instead of a basic RNN.\n",
    "    * So the LSTM has both the hidden state $s^{\\langle t\\rangle}$ and the cell state $c^{\\langle t\\rangle}$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Each time step does not use predictions from the previous time step\n",
    "* Unlike previous text generation examples earlier in the course, in this model, the post-attention LSTM at time $t$ does not take the previous time step's prediction $y^{\\langle t-1 \\rangle}$ as input.\n",
    "* The post-attention LSTM at time 't' only takes the hidden state $s^{\\langle t\\rangle}$ and cell state $c^{\\langle t\\rangle}$ as input. \n",
    "* We have designed the model this way because unlike language generation (where adjacent characters are highly correlated) there isn't as strong a dependency between the previous character and the next character in a YYYY-MM-DD date."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Concatenation of hidden states from the forward and backward pre-attention LSTMs\n",
    "- $\\overrightarrow{a}^{\\langle t \\rangle}$: hidden state of the forward-direction, pre-attention LSTM.\n",
    "- $\\overleftarrow{a}^{\\langle t \\rangle}$: hidden state of the backward-direction, pre-attention LSTM.\n",
    "- $a^{\\langle t \\rangle} = [\\overrightarrow{a}^{\\langle t \\rangle}, \\overleftarrow{a}^{\\langle t \\rangle}]$: the concatenation of the activations of both the forward-direction $\\overrightarrow{a}^{\\langle t \\rangle}$ and backward-directions $\\overleftarrow{a}^{\\langle t \\rangle}$ of the pre-attention Bi-LSTM. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing \"energies\" $e^{\\langle t, t' \\rangle}$ as a function of $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t' \\rangle}$\n",
    "- Recall in the lesson videos \"Attention Model\", at time 6:45 to 8:16, the definition of \"e\" as a function of $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$.\n",
    "    - \"e\" is called the \"energies\" variable.\n",
    "    - $s^{\\langle t-1 \\rangle}$ is the hidden state of the post-attention LSTM\n",
    "    - $a^{\\langle t' \\rangle}$ is the hidden state of the pre-attention LSTM.\n",
    "    - $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$ are fed into a simple neural network, which learns the function to output $e^{\\langle t, t' \\rangle}$.\n",
    "    - $e^{\\langle t, t' \\rangle}$ is then used when computing the attention $a^{\\langle t, t' \\rangle}$ that $y^{\\langle t \\rangle}$ should pay to $a^{\\langle t' \\rangle}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The diagram on the right of figure 1 uses a `RepeatVector` node to copy $s^{\\langle t-1 \\rangle}$'s value $T_x$ times.\n",
    "- Then it uses `Concatenation` to concatenate $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$.\n",
    "- The concatenation of $s^{\\langle t-1 \\rangle}$ and $a^{\\langle t \\rangle}$ is fed into a \"Dense\" layer, which computes $e^{\\langle t, t' \\rangle}$. \n",
    "- $e^{\\langle t, t' \\rangle}$ is then passed through a softmax to compute $\\alpha^{\\langle t, t' \\rangle}$.\n",
    "- Note that the diagram doesn't explicitly show variable $e^{\\langle t, t' \\rangle}$, but $e^{\\langle t, t' \\rangle}$ is above the Dense layer and below the Softmax layer in the diagram in the right half of figure 1.\n",
    "- We'll explain how to use `RepeatVector` and `Concatenation` in Keras below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation Details\n",
    "   \n",
    "Let's implement this neural translator. You will start by implementing two functions: `one_step_attention()` and `model()`.\n",
    "\n",
    "#### one_step_attention\n",
    "* The inputs to the one_step_attention at time step $t$ are:\n",
    "    - $[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$: all hidden states of the pre-attention Bi-LSTM.\n",
    "    - $s^{<t-1>}$: the previous hidden state of the post-attention LSTM \n",
    "* one_step_attention computes:\n",
    "    - $[\\alpha^{<t,1>},\\alpha^{<t,2>}, ..., \\alpha^{<t,T_x>}]$: the attention weights\n",
    "    - $context^{ \\langle t \\rangle }$: the context vector:\n",
    "    \n",
    "$$context^{<t>} = \\sum_{t' = 1}^{T_x} \\alpha^{<t,t'>}a^{<t'>}\\tag{1}$$ \n",
    "\n",
    "##### Clarifying 'context' and 'c'\n",
    "- In the lecture videos, the context was denoted $c^{\\langle t \\rangle}$\n",
    "- In the assignment, we are calling the context $context^{\\langle t \\rangle}$.\n",
    "    - This is to avoid confusion with the post-attention LSTM's internal memory cell variable, which is also denoted $c^{\\langle t \\rangle}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Defined shared layers as global variables\n",
    "repeator = RepeatVector(Tx)\n",
    "concatenator = Concatenate(axis=-1)\n",
    "densor1 = Dense(10, activation = \"tanh\")\n",
    "densor2 = Dense(1, activation = \"relu\")\n",
    "activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook\n",
    "dotor = Dot(axes = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def one_step_attention(a, s_prev):\n",
    "    \"\"\"\n",
    "    Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights\n",
    "    \"alphas\" and the hidden states \"a\" of the Bi-LSTM.\n",
    "    \n",
    "    Arguments:\n",
    "    a -- hidden state output of the Bi-LSTM, numpy-array of shape (m, Tx, 2*n_a)\n",
    "    s_prev -- previous hidden state of the (post-attention) LSTM, numpy-array of shape (m, n_s)\n",
    "    \n",
    "    Returns:\n",
    "    context -- context vector, input of the next (post-attention) LSTM cell\n",
    "    \"\"\"\n",
    "    \n",
    "    s_prev = repeator(s_prev)\n",
    "    concat = concatenator([a,s_prev])\n",
    "    e = densor1(concat)\n",
    "    energies = densor2(e)\n",
    "    alphas = activator(energies)\n",
    "    context = dotor([alphas,a])\n",
    "    \n",
    "    return context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_a = 32 # number of units for the pre-attention, bi-directional LSTM's hidden state 'a'\n",
    "n_s = 64 # number of units for the post-attention LSTM's hidden state \"s\"\n",
    "\n",
    "# Please note, this is the post attention LSTM cell.  \n",
    "# For the purposes of passing the automatic grader\n",
    "# please do not modify this global variable.  This will be corrected once the automatic grader is also updated.\n",
    "post_activation_LSTM_cell = LSTM(n_s, return_state = True) # post-attention LSTM \n",
    "output_layer = Dense(len(machine_vocab), activation=softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size):\n",
    "    \"\"\"\n",
    "    Arguments:\n",
    "    Tx -- length of the input sequence\n",
    "    Ty -- length of the output sequence\n",
    "    n_a -- hidden state size of the Bi-LSTM\n",
    "    n_s -- hidden state size of the post-attention LSTM\n",
    "    human_vocab_size -- size of the python dictionary \"human_vocab\"\n",
    "    machine_vocab_size -- size of the python dictionary \"machine_vocab\"\n",
    "\n",
    "    Returns:\n",
    "    model -- Keras model instance\n",
    "    \"\"\"\n",
    "    \n",
    "    X = Input(shape=(Tx, human_vocab_size))\n",
    "    s0 = Input(shape=(n_s,), name='s0')\n",
    "    c0 = Input(shape=(n_s,), name='c0')\n",
    "    s = s0\n",
    "    c = c0\n",
    "    \n",
    "    outputs = []\n",
    "    \n",
    "    \n",
    "    a = Bidirectional(LSTM(units=n_a, return_sequences=True))(X)\n",
    "    \n",
    "    for t in range(Ty):\n",
    "    \n",
    "        context = one_step_attention(a, s)\n",
    "        \n",
    "        s, _, c = post_activation_LSTM_cell(inputs=context, initial_state=[s, c])\n",
    "        \n",
    "        out = output_layer(inputs=s)\n",
    "        \n",
    "        outputs.append(out)\n",
    "    \n",
    "    model = Model(inputs=[X,s0,c0], outputs=outputs)\n",
    "        \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to create your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = model(Tx, Ty, n_a, n_s, len(human_vocab), len(machine_vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_1 (InputLayer)             (None, 30, 37)        0                                            \n",
      "____________________________________________________________________________________________________\n",
      "s0 (InputLayer)                  (None, 64)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional)  (None, 30, 64)        17920       input_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "repeat_vector_1 (RepeatVector)   (None, 30, 64)        0           s0[0][0]                         \n",
      "                                                                   lstm_1[0][0]                     \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[8][0]                     \n",
      "____________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)      (None, 30, 128)       0           bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[0][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[1][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[2][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[3][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[4][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[5][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[6][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[7][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[8][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[9][0]            \n",
      "____________________________________________________________________________________________________\n",
      "dense_1 (Dense)                  (None, 30, 10)        1290        concatenate_1[0][0]              \n",
      "                                                                   concatenate_1[1][0]              \n",
      "                                                                   concatenate_1[2][0]              \n",
      "                                                                   concatenate_1[3][0]              \n",
      "                                                                   concatenate_1[4][0]              \n",
      "                                                                   concatenate_1[5][0]              \n",
      "                                                                   concatenate_1[6][0]              \n",
      "                                                                   concatenate_1[7][0]              \n",
      "                                                                   concatenate_1[8][0]              \n",
      "                                                                   concatenate_1[9][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_2 (Dense)                  (None, 30, 1)         11          dense_1[0][0]                    \n",
      "                                                                   dense_1[1][0]                    \n",
      "                                                                   dense_1[2][0]                    \n",
      "                                                                   dense_1[3][0]                    \n",
      "                                                                   dense_1[4][0]                    \n",
      "                                                                   dense_1[5][0]                    \n",
      "                                                                   dense_1[6][0]                    \n",
      "                                                                   dense_1[7][0]                    \n",
      "                                                                   dense_1[8][0]                    \n",
      "                                                                   dense_1[9][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "attention_weights (Activation)   (None, 30, 1)         0           dense_2[0][0]                    \n",
      "                                                                   dense_2[1][0]                    \n",
      "                                                                   dense_2[2][0]                    \n",
      "                                                                   dense_2[3][0]                    \n",
      "                                                                   dense_2[4][0]                    \n",
      "                                                                   dense_2[5][0]                    \n",
      "                                                                   dense_2[6][0]                    \n",
      "                                                                   dense_2[7][0]                    \n",
      "                                                                   dense_2[8][0]                    \n",
      "                                                                   dense_2[9][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "dot_1 (Dot)                      (None, 1, 64)         0           attention_weights[0][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[1][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[2][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[3][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[4][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[5][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[6][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[7][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[8][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[9][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "c0 (InputLayer)                  (None, 64)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                    [(None, 64), (None, 6 33024       dot_1[0][0]                      \n",
      "                                                                   s0[0][0]                         \n",
      "                                                                   c0[0][0]                         \n",
      "                                                                   dot_1[1][0]                      \n",
      "                                                                   lstm_1[0][0]                     \n",
      "                                                                   lstm_1[0][2]                     \n",
      "                                                                   dot_1[2][0]                      \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[1][2]                     \n",
      "                                                                   dot_1[3][0]                      \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[2][2]                     \n",
      "                                                                   dot_1[4][0]                      \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[3][2]                     \n",
      "                                                                   dot_1[5][0]                      \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[4][2]                     \n",
      "                                                                   dot_1[6][0]                      \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[5][2]                     \n",
      "                                                                   dot_1[7][0]                      \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[6][2]                     \n",
      "                                                                   dot_1[8][0]                      \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[7][2]                     \n",
      "                                                                   dot_1[9][0]                      \n",
      "                                                                   lstm_1[8][0]                     \n",
      "                                                                   lstm_1[8][2]                     \n",
      "____________________________________________________________________________________________________\n",
      "dense_3 (Dense)                  (None, 11)            715         lstm_1[0][0]                     \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[8][0]                     \n",
      "                                                                   lstm_1[9][0]                     \n",
      "====================================================================================================\n",
      "Total params: 52,960\n",
      "Trainable params: 52,960\n",
      "Non-trainable params: 0\n",
      "____________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt = Adam(lr=0.005, beta_1=0.9, beta_2=0.999, decay=0.01)\n",
    "model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s0 = np.zeros((m, n_s))\n",
    "c0 = np.zeros((m, n_s))\n",
    "outputs = list(Yoh.swapaxes(0,1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now fit the model and run it for one epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "10000/10000 [==============================] - 54s - loss: 16.9592 - dense_3_loss_1: 1.2274 - dense_3_loss_2: 1.0533 - dense_3_loss_3: 1.7895 - dense_3_loss_4: 2.6906 - dense_3_loss_5: 0.7594 - dense_3_loss_6: 1.3344 - dense_3_loss_7: 2.8516 - dense_3_loss_8: 0.9749 - dense_3_loss_9: 1.7424 - dense_3_loss_10: 2.5356 - dense_3_acc_1: 0.4776 - dense_3_acc_2: 0.6344 - dense_3_acc_3: 0.2690 - dense_3_acc_4: 0.0850 - dense_3_acc_5: 0.9592 - dense_3_acc_6: 0.2892 - dense_3_acc_7: 0.0411 - dense_3_acc_8: 0.9595 - dense_3_acc_9: 0.2557 - dense_3_acc_10: 0.1180    \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f25c69e0f60>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([Xoh, s0, c0], outputs, epochs=1, batch_size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples: \n",
    "\n",
    "<img src=\"images/table.png\" style=\"width:700;height:200px;\"> <br>\n",
    "<caption><center>Thus, `dense_2_acc_8: 0.89` means that you are predicting the 7th character of the output correctly 89% of the time in the current batch of data. </center></caption>\n",
    "\n",
    "\n",
    "We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.load_weights('models/model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now see the results on new examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "source: 3 May 1979\n",
      "output: 1979-05-03 \n",
      "\n",
      "source: 5 April 09\n",
      "output: 2009-05-05 \n",
      "\n",
      "source: 21th of August 2016\n",
      "output: 2016-08-21 \n",
      "\n",
      "source: Tue 10 Jul 2007\n",
      "output: 2007-07-10 \n",
      "\n",
      "source: Saturday May 9 2018\n",
      "output: 2018-05-09 \n",
      "\n",
      "source: March 3 2001\n",
      "output: 2001-03-03 \n",
      "\n",
      "source: March 3rd 2001\n",
      "output: 2001-03-03 \n",
      "\n",
      "source: 1 March 2001\n",
      "output: 2001-03-01 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']\n",
    "for example in EXAMPLES:\n",
    "    \n",
    "    source = string_to_int(example, Tx, human_vocab)\n",
    "    source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1)\n",
    "    prediction = model.predict([source, s0, c0])\n",
    "    prediction = np.argmax(prediction, axis = -1)\n",
    "    output = [inv_machine_vocab[int(i)] for i in prediction]\n",
    "    \n",
    "    print(\"source:\", example)\n",
    "    print(\"output:\", ''.join(output),\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also change these examples to test with your own examples. The next part will give you a better sense of what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Visualizing Attention\n",
    "\n",
    "Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (such as the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can  visualize what each part of the output is looking at which part of the input.\n",
    "\n",
    "Consider the task of translating \"Saturday 9 May 2018\" to \"2018-05-09\". If we visualize the computed $\\alpha^{\\langle t, t' \\rangle}$ we get this: \n",
    "\n",
    "<img src=\"images/date_attention.png\" style=\"width:600;height:300px;\"> <br>\n",
    "<caption><center> **Figure 8**: Full Attention Map</center></caption>\n",
    "\n",
    "Notice how the output ignores the \"Saturday\" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We also see that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's \"18\" in order to generate \"2018.\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 - Getting the attention weights from the network\n",
    "\n",
    "Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\\alpha^{\\langle t, t' \\rangle}$. \n",
    "\n",
    "To figure out where the attention values are located, let's start by printing a summary of the model ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________________________________________________________________________________________\n",
      "Layer (type)                     Output Shape          Param #     Connected to                     \n",
      "====================================================================================================\n",
      "input_1 (InputLayer)             (None, 30, 37)        0                                            \n",
      "____________________________________________________________________________________________________\n",
      "s0 (InputLayer)                  (None, 64)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional)  (None, 30, 64)        17920       input_1[0][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "repeat_vector_1 (RepeatVector)   (None, 30, 64)        0           s0[0][0]                         \n",
      "                                                                   lstm_1[0][0]                     \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[8][0]                     \n",
      "____________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)      (None, 30, 128)       0           bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[0][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[1][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[2][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[3][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[4][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[5][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[6][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[7][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[8][0]            \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   repeat_vector_1[9][0]            \n",
      "____________________________________________________________________________________________________\n",
      "dense_1 (Dense)                  (None, 30, 10)        1290        concatenate_1[0][0]              \n",
      "                                                                   concatenate_1[1][0]              \n",
      "                                                                   concatenate_1[2][0]              \n",
      "                                                                   concatenate_1[3][0]              \n",
      "                                                                   concatenate_1[4][0]              \n",
      "                                                                   concatenate_1[5][0]              \n",
      "                                                                   concatenate_1[6][0]              \n",
      "                                                                   concatenate_1[7][0]              \n",
      "                                                                   concatenate_1[8][0]              \n",
      "                                                                   concatenate_1[9][0]              \n",
      "____________________________________________________________________________________________________\n",
      "dense_2 (Dense)                  (None, 30, 1)         11          dense_1[0][0]                    \n",
      "                                                                   dense_1[1][0]                    \n",
      "                                                                   dense_1[2][0]                    \n",
      "                                                                   dense_1[3][0]                    \n",
      "                                                                   dense_1[4][0]                    \n",
      "                                                                   dense_1[5][0]                    \n",
      "                                                                   dense_1[6][0]                    \n",
      "                                                                   dense_1[7][0]                    \n",
      "                                                                   dense_1[8][0]                    \n",
      "                                                                   dense_1[9][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "attention_weights (Activation)   (None, 30, 1)         0           dense_2[0][0]                    \n",
      "                                                                   dense_2[1][0]                    \n",
      "                                                                   dense_2[2][0]                    \n",
      "                                                                   dense_2[3][0]                    \n",
      "                                                                   dense_2[4][0]                    \n",
      "                                                                   dense_2[5][0]                    \n",
      "                                                                   dense_2[6][0]                    \n",
      "                                                                   dense_2[7][0]                    \n",
      "                                                                   dense_2[8][0]                    \n",
      "                                                                   dense_2[9][0]                    \n",
      "____________________________________________________________________________________________________\n",
      "dot_1 (Dot)                      (None, 1, 64)         0           attention_weights[0][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[1][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[2][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[3][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[4][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[5][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[6][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[7][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[8][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "                                                                   attention_weights[9][0]          \n",
      "                                                                   bidirectional_1[0][0]            \n",
      "____________________________________________________________________________________________________\n",
      "c0 (InputLayer)                  (None, 64)            0                                            \n",
      "____________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                    [(None, 64), (None, 6 33024       dot_1[0][0]                      \n",
      "                                                                   s0[0][0]                         \n",
      "                                                                   c0[0][0]                         \n",
      "                                                                   dot_1[1][0]                      \n",
      "                                                                   lstm_1[0][0]                     \n",
      "                                                                   lstm_1[0][2]                     \n",
      "                                                                   dot_1[2][0]                      \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[1][2]                     \n",
      "                                                                   dot_1[3][0]                      \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[2][2]                     \n",
      "                                                                   dot_1[4][0]                      \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[3][2]                     \n",
      "                                                                   dot_1[5][0]                      \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[4][2]                     \n",
      "                                                                   dot_1[6][0]                      \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[5][2]                     \n",
      "                                                                   dot_1[7][0]                      \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[6][2]                     \n",
      "                                                                   dot_1[8][0]                      \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[7][2]                     \n",
      "                                                                   dot_1[9][0]                      \n",
      "                                                                   lstm_1[8][0]                     \n",
      "                                                                   lstm_1[8][2]                     \n",
      "____________________________________________________________________________________________________\n",
      "dense_3 (Dense)                  (None, 11)            715         lstm_1[0][0]                     \n",
      "                                                                   lstm_1[1][0]                     \n",
      "                                                                   lstm_1[2][0]                     \n",
      "                                                                   lstm_1[3][0]                     \n",
      "                                                                   lstm_1[4][0]                     \n",
      "                                                                   lstm_1[5][0]                     \n",
      "                                                                   lstm_1[6][0]                     \n",
      "                                                                   lstm_1[7][0]                     \n",
      "                                                                   lstm_1[8][0]                     \n",
      "                                                                   lstm_1[9][0]                     \n",
      "====================================================================================================\n",
      "Total params: 52,960\n",
      "Trainable params: 52,960\n",
      "Non-trainable params: 0\n",
      "____________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate through the output of `model.summary()` above. You can see that the layer named `attention_weights` outputs the `alphas` of shape (m, 30, 1) before `dot_2` computes the context vector for every time step $t = 0, \\ldots, T_y-1$. Let's get the attention weights from this layer.\n",
    "\n",
    "The function `attention_map()` pulls out the attention values from your model and plots them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f25ae723cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAesAAAGsCAYAAAD9ro91AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8HXV5+PHPc7OQEPZVASWoLCIKJmFH3JW2WLFiERUr\nUq1V/Glb27q0Vn8trVbrz1ptFaqlbuBCrUqxihtLJBCCAQKCUgEFqchuQhaS+/z+mLnk5ObOnHOX\nc+83uZ/363WSc853vjPPmZlznzPrE5mJJEkq18BUByBJktqZrCVJKpzJWpKkwpmsJUkqnMlakqTC\nmawlSSqcyVqSpMKZrCVJKpzJWpKkws2c6gA67bbbbrnvvvNHbFu1ahXz5s0b03inU98tLV77lj3N\nXvqu29B8F8R1q1cxe25z3xv+567GtsfsvA3/e//axvbDDtirse3hVSvZdt52je3R2CJNrttvv417\n7rmn6ypZVLLed9/5LL7y6hHbrrj8+xx93LPGNN7p1HdLi9e+ZU+zl753PbCmse0ny69g/8OObmw/\n+KXvb2x72ysO4F2f/3Fj+yXf/MvGtqVXXMrhRx/f2D5rpjsVVYZjj1zU03CusZIkFc5kLUlS4fqW\nrCPiUxFxd0Ss6Nc0JEmaDvq5ZX0ucEIfxy9J0rTQt2SdmZcC9/Vr/JIkTReR2XzZxbhHHjEfuDAz\nD2kZ5vXA6wH23HPPheedf/6Iw61cuZLttmu+FKPNdOq7pcVr37Kn2UvfR9Y3/w1Zs3olc+Y2913x\nP//b2Lb3rttw571tl249trFt1cqVzGuJObx2S4V425+8jWXLri7/0q3MPBs4G2DhwkXZdIlIqZet\nlNZ3S4vXvmVPs5e+47l068V/3Xzp1lldLt26+5unNLZ56Za2Nq6xkiQVzmQtSVLh+nnp1nnAFcCB\nEXFHRJzRr2lJkrQ169sx68w8tV/jliRpOnE3uCRJhTNZS5JUuCm/dEvSlm3nebMa22bMiNZ25rRc\n+z0w0Np+5/2rG9vWrR9sbZ+/+9jKhUpTxS1rSZIKZ7KWJKlwJmtJkgrX12QdEW+JiBURcUNEvLWf\n05IkaWvVz5uiHAK8DjgCOBQ4MSKe1K/pSZK0ternlvWTgSsz8+HMXA9cAvxOH6cnSdJWqW8lMiPi\nycBXgaOB1cB3gKsz883DhrNE5gT23dLitW/Z0+yl72DLn5CHV65k25a+197yy8a2vXeezZ33r2ts\nP3j+7o1t69asYvac5suztpnl6Toqw5SXyMzMH0XE+4FvAauA5cCGEYazROYE9t3S4rVv2dPspe+a\nRzb7Wj9q2ZLLWHjUMxrbT3zfhxvbzjp5H9715Tsa23947omNbbetuJL5hxzZ2O511trS9PXnZWZ+\nMjMXZubxwP1Ac3FaSZI0or7ewSwi9sjMuyPi8VTHq4/q5/QkSdoa9ft2oxdExK7AI8CbMvOBPk9P\nkqStTl+TdWY2H6ySJEk98ZRISZIKZ7KWJKlwlsiUNC6PrB9sbMvM1nYe+N/mtg17trbvst3sxrY7\nZkRru7SlcctakqTCmawlSSpcT8k6IvaNiOfVz+dGxPb9DUuSJA3pmqwj4nXAl4FP1G/tA/xnLyO3\nRKYkSePXy5b1m4BjgYcAMvMnwB7dOlkiU5KkidFLsl6bmY+WvomImUAvpboskSlJ0gToWiIzIv4e\neAB4NfBm4I3AjZn5ri79LJFpGUX7TmDfUuPd0FIjc/Wqlcyd19z3uh/f2di2967bcue9Dze2P+2A\nvcc83RkDXSsSSpOi1xKZvSTrAeAM4AVAAN8E/jV7KIQdEWdQJfdVwA1UW+mNx64XLlyUi6+8esS2\nUssDltZ3S4vXvmVPs5e+v179SGPbtUsv59DDj2tsf/zz3tnYdtbph/Kuf7u2sf32b/9tY9t1Sy/n\naS3T3WHurMY2aTIde+SiCatnPRf4VGaeAxARM+r3mn/y1jLzk8An635/CzQXp5UkSSPq5Zj1d6iS\n85C5wLd7GXlE7FH/P1Qi8/OjDVCSpOmuly3rOZm5cuhFZq6MiG17HL8lMiVJGqdekvWqiFiQmdcA\nRMRCqhPGurJEpiRJ49dLsn4r8KWI+AXVCWaPAU7pa1SSJOlRXZN1Zi6NiIOAA+u3bs7M5tM/JUnS\nhOq1RObhwPx6+AURQWZ+um9RSQVovTox29tbuyYMtlybPJ5+0XQBSJd4266VzoT1G5rLXM6a0Xye\nahCt7dd+9b2NbbeuWMK1X22+j9L80z/T2HbWb+7Ib/9zc/t957+2sU0qUddkHRGfAZ4ILAc21G8n\nYLKWJGkS9LJlvQg4uJeboEiSpInXy3XWK6hOKpMkSVOgly3r3YAbI+IqYO3Qm5n5222dImIOcCmw\nTT2dL2fmX40jVkmSpqVekvV7xjjutcBz6puozAIuj4hvZOaSMY5PkqRpqZdLty6JiH2B/TPz2/Xd\ny2b00C+BoTufzaofHveWJGmUeqm69TqqEpa7ZOYTI2J/4OOZ+dyuI6+KfiwDngR8LDP/fIRhLJE5\ngX23tHiL7tvy1ejWt+1btWrlSuaNIeZe+jVdudXPeNv+hDy8aiXbtpSqXD/YfEnY2tWr2GbuvMb2\nG392f2Pb3jvO4M4HNzS2H/aEXRvbpMnUa4nMXnaDvwk4ArgSIDN/MlSgo5vM3AAcFhE7AV+JiEMy\nc8WwYc4GzoaqRGZTKb5SywOW1ndLi7fkvm0/ZJdcfglHHffMlr7N012y+BKOOra573j6NV1n3S3e\ntuusr/rBpRxxzPGN7es3NPddtuQyFh7VfNfhux9a29h264ol7HfIUY3tL/n4yD/sobrO+l0XPdjY\nft/5L21sk0rUy9ngazNz3dCLiJjJKHdn1wU8vgecMLrwJElSL8n6koh4JzA3Ip4PfAn4erdOEbF7\nvUVNRMwFng/cNJ5gJUmajnpJ1m8HfgVcD/wBcBHwFz30eyzwvYi4DlgKXJyZF441UEmSpqtezgYf\nBM6pHz3LzOuAp48xLkmSVOvl3uC3MsIx6sx8Ql8ikiRJm+j13uBD5gAvA3bpTziSJGm4XnaD3zvs\nrQ9HxDLg3RMdzPrB5MGHRy6VvaGlDWCw5VqZ9RuS+1eta2xvu8xm/YbkvpXNfdt069t2Sv36Dcm9\nDX3bLsjrNs1tZzffz2ZwENasa742dZtZLac49LFkZNt8ymy/7GjV2vWNbRsGk4dWN69TbZckrR9M\n7l/V3Lct5g2Dyf0t6/J4+jWtG93inTHQvFYNZrJqbfN6MbdlnYqAgZZxP363bRvb7pw50NreVuby\nisu/7+VZ2qr0sht8QcfLAaot7V7rYEuSpHHqJen+Q8fz9cBtwO/2JRpJkrSZXnaDP3syApEkSSPr\nZTf4H7e1Z+aHJi4cSZI0XK9ngx8OfK1+/SLgKuAn/QpKkiRt1Euy3gdYkJm/BoiI9wD/lZmv6mdg\nkiSp0kuJzJuBp2Xm2vr1NsB1mXnghAQwrETmZz9/3ojDrV61krktpfbaPsaah1cyZ9uxlVHc0vp2\n6zfQVJaJ7uUMW67AmZKSkb30bbukb0tbp/o5zZbVout8alunui2ftulOVelUaTJNZInMTwNXRcRX\n6tcnAf8+nuA6dZbIPPTpC/Opi44bcbjrr76cpjZo/6O84urFHLLo2JYYmuO7YdlinrKwuW+bbn3b\nEtiNyxZzcEPftqXabZpt11l3K2fYdp11P0tGts2nKxdfwpEtfduus75u6eU87fDmdartOut+Lds2\nvfRrWje6xdt2nXW3717bddZXX3Epi45uLq85e2bzOjVVpVOlEnUt5JGZZwGnA/fXj9Mz8297nUBE\nvCkiltePvcYeqiRJ01OvNzfZFngoM/+tLn25X2be2kvHzPwY8LExRyhJ0jTXdcs6Iv4K+HPgHfVb\ns4DP9jMoSZK0US/1rF8C/DawCiAzfwFs38+gJEnSRr0k63VZnTKeABExr78hSZKkTr0cs/5iRHwC\n2CkiXge8FjinH8HMGIjGs5UHorkNYN36wca2gQhmz2j+XdJ2JnkEzJrRfKbs+paKT0P9m6xc3Xym\n8mAmq9Y0t7f1W9nSr+1Svaq6UnPfmTNmNY+X9upX3bQtg26jbZtuWxWxwexWZay9mlTbGdRtImBm\nS9+mdabbugjNZ913i3feNs2fdSCitX1my3crov2Mb0m96eXe4B+MiOcDDwEHAO/OzIv7HpkkSQJ6\nPBs8My+OiGuA44H7+huSJEnq1Lh/KiIujIhD6uePBVZQ7QL/TES8dZLikyRp2ms7mLRfZq6on58O\nXJyZLwKOpErakiRpErQl60c6nj8XuAigLujRfDZXLSI+FRF3R8SKbsNKkqRmbcn65xHx5oh4CbAA\n+G+AiJhLdWOUbs4FThh3hJIkTXNtyfoM4CnAa4BTMvOB+v2jgH/rNuLMvBRPRpMkady6lsgc18gj\n5gMXZuYhLcNsUiLz8+edP+Jw3UrttX2MbqUfs6VGUj/LKLZdW7x29Sq2mTv6+89069dWzrBbvG3X\n6fazzOV4+rZdB9+9nGjzdLutF23G2ref09zaylxaIlNbiokskdlXnSUyFyxclIc3lNNbesWlNLVB\n+01Rll91OYcdMbbymtcuvZxD28ootiSDbqU5f91yU5RbVyxhv0OOamwfa7+2m1vceM0POHjBMY3t\nO27bfPTjqh9cyhHHNC+fNt36tt0UpVsJxgdWrWtsu2n5FRx02NGN7W03RelWNrJNt75Nya9bSU9o\n/vHYbZpt60W35dN2U5SpKnNpiUxtbby1kCRJheul6tZmm4YjvSdJkvqjly3rf+rxvU1ExHnAFcCB\nEXFHRJwx2uAkSVLLMeuIOBo4Btg9Iv64o2kHoPkAVy0zTx1/eJIkqe0Es9nAdvUwnfWrHwJO7mdQ\nkiRpo8ZknZmXAJdExLmZeftkBJPZfHZ1WxsA3U58b2mf23LW70AEc1tKc7ZdzjRzINip5Qzq7ec0\n/1a6Y8YAe+08Z8S2tvnw8xkD7LHjNq0xNZkxEK1nfEfbNTo9tI+1b5eqkO2lH1vm8YyI1vYHH36k\nsW1wkNZyomseab46Yf2G5N6VzWepN32eDYPJ/auaY4LmcpSDg/BwSznQh1Y3j/eRDYPc9cCaxvbt\nWubh+g3J/S1n5Ld9P8j2kq7jWd+kLU0vl26dGxGbfWMy8zl9iEeSJA3TS7J+W8fzOcBLgeZNCkmS\nNKG6JuvMXDbsrcURcVWf4pEkScP0cp31Lh2P3SLihcCOvYw8Ik6IiJsj4paIePu4o5UkaRrqZTf4\nMiCpTtFaD9xKVeSjVUTMAD4GPB+4A1gaEV/LzBvHHq4kSdNPL7vB9xvjuI8AbsnMnwJExPnAiwGT\ntSRJo9A1WUfEHOCNwHFUW9iXAR/PzOZrOSp7Az/veH0HcOQY45QkadrqWiIzIr4I/Br4bP3WK4Cd\nMvNlXfqdDJyQmb9fvz4NODIzzxw23CYlMj/3+ZFLZHYrc9mmW9+2yzW7lgdsmW63Mn1tc75tum39\nHl65km3HGG+pZS7H03c8pVM3tJXXXL2SOXPHVv503ZpVzJ7TXMa0aX3spWxqNCzhbvG2lYntZ9nV\ntuv+u5a57FNpTmkyTWSJzEMy8+CO19+LiF52Zd8JPK7j9T71e5voLJH59AWLcsFRzxhxZNcsuYym\nNmj/47j8yss57Mjm8oDbNNxIArqXB2y7IceSyy/hqOOe2djelgzaptt2U5RlSy5jYct8avvj2O2z\ntt2E4srFl3Dksc2ftU0/+655pPlGIN3Wi7abotxy7RKedGhzKdK2m6L87IYrefxTmncyNa1Tt61Y\nwvwuZVObboryk+VXsH9LOdD1G5rj/en1S3jCU5un23ZTlG5lYttuitLt+9O2PloiU1ubXgp5XBMR\nj35TI+JI4Ooe+i0F9o+I/SJiNvBy4GtjC1OSpOmrly3rhcAPIuJn9evHAzdHxPVAZubTRuqUmesj\n4kzgm1SFPz6VmTdMRNCSJE0nvSTrE8Y68sy8CLhorP0lSVJvyfpvMvO0zjci4jPD35MkSf3RyzHr\np3S+iIiZVLvGJUnSJGjcso6IdwDvBOZGxENsvFBiHfXZ2xNtIGgsRzkw0NzWdbwDMG+bXnYibC4C\nZs7o5TfNSJ3bz1id2VL7sW26M1tmw0DAnJaSn60lB2k/u/3WXz3c2LZu/SC339Pc/uHFtza2HTtz\nDed/tfl0hgX7NF+Cs+vD6/jMsuYKrq9euG9jW0R7edR5OzevM7fPDPbaeW5je5tf/niAJ+zRfgnW\nSH4xc4B9d9t2TNO8bWbw2J1GLrnazR0zB3jcrmOb7swZwc7zZo+pb7fvjzSdNGahzPy7zNwe+EBm\n7pCZ29ePXTPzHZMYoyRJ01ovm5vfiIjNLr7NzEv7EI8kSRqml2T9px3P51Dd83sZ8Jy+RCRJkjbR\nSyGPF3W+jojHAR/uW0SSJGkTYzlz6g7gyRMdiCRJGlkvVbf+iY21IwaAw4Br+hmUJEnaqJeqW7/X\n8XI9cFtmLp6wAIZV3Trv/JGrbo2nis506tu1X8vi7tZ37frmYg/dKkn9cuW6xrbtYh0rs/nynm1n\nN+8Amrl+DetnNl+StOu2zePtWlGtT1Wd+rZs7TshfaXJNJFVt74APKl+fksPdaxHpbPq1sKFi7Kp\nUs54quhMp77d+rX9OOtW5ajtOutulaT+o/U66ztYvH6fxvYFj2m5zvq+H3PvLgc0tv9Wy3XWSxZf\nwlEtFbsGWq45L3HZ2ndi+kolatxkiYiZEfH3VMeo/x34NPDziPj7iGiua7f5eN4UEcvrx17jD1mS\npOml7QSzDwC7APtl5sLMXAA8EdgJ+GCvE8jMj2XmYfXjF+MLV5Kk6actWZ8IvC4zfz30RmY+BPwh\n8Jv9DkySJFXaknXmCAc4M3MDracpSZKkidSWrG+MiFcPfzMiXgXc1L+QJElSp7azwd8E/EdEvJbq\n9qIAi4C5wEv6HZgkSao0JuvMvBM4MiKew8aa1hdl5ncmJTL1RWvJwS4lCffeufl65rtmDrS2r1q7\nvrFtcEa2tt909+rGtoUMtrZ3+bit7epN670askt764i7lHR14Wka6eXe4N8FvjsJsUiSpBGM5d7g\nkiRpEpmsJUkqXF+TdUScEBE3R8QtEfH2fk5LkqStVd+SdUTMAD4G/AZwMHBqRBzcr+lJkrS16ueW\n9RFUhT9+mpnrgPOBF/dxepIkbZW6lsgc84gjTgZOyMzfr1+fBhyZmWcOG84SmRPYt5/THGxZVR5e\nuZJtW/r+7P7my6t2GHiEhwaba8PMnNF8ic481rGK5jKY++zQfDlZ13llicze+o6j7Oq4ptun5SNN\npokskdlXlsic2L79nObaRzY0tl295DIWHfWMxvZPXXB9Y9vztr2Lbz/82Mb23bffprFtIbezjOYy\nmCcfd1BjW7eSoG3X8W5ty3Y8fcdTdrXNVC0fqUT93A1+J/C4jtf71O9JkqRR6GeyXgrsHxH7RcRs\n4OXA1/o4PUmStkp92w2emesj4kzgm8AM4FOZeUO/pidJ0taqr8esM/Mi4KJ+TkOSpK2ddzCTJKlw\nJmtJkgo35ZduacuxzawZjW0D0d5+zu8e2ti2ZPEDnHNic/uuR765se2s1x/JJ86+sLH9b074SGNb\nAhtaLh5vu767nxovhepjucm+9e1iPCVbx6Nfn0cTw/Knm3PLWpKkwpmsJUkqnMlakqTC9btE5lsi\nYkVE3BARb+3ntCRJ2lr1s0TmIcDrqKpvHQqcGBFP6tf0JEnaWvVzy/rJwJWZ+XBmrgcuAX6nj9OT\nJGmr1M8SmU8GvgocDawGvgNcnZlvHjacJTInsG+p8batZqtWrmReS9/lN/28sW3v3eZx5z2rGtsP\nO+hxjW3dptt29chUlJwsddmOq29h81iFmEZXbk15iczM/FFEvB/4FrAKWA5sVmPREpkT27fUeAdb\nrmdesvgSjjq2uRTib/1x+3XW7zr7ysb2Xy15ZWPbVT+4lCOOOb6xfeaM5h1PU1Fysp/lJqeq71SV\nIfU667J5nfXm+nqCWWZ+MjMXZubxwP3Aj/s5PUmStkZ9vYNZROyRmXdHxOOpjlcf1c/pSZK0Ner3\n7UYviIhdgUeAN2XmA32eniRJW51+l8h8Rj/HL0nSdOAdzCRJKpzJWpKkwvXtOuuxiIhfAbc3NO8G\n3DPGUU+nvltavPYte5rTsa80mfbNzN27DVRUsm4TEVdn5iL7ljdN+05O3y0t3i21r1Qid4NLklQ4\nk7UkSYXbkpL12fYtdpr2nZy+W1q8W2pfqThbzDFrSZKmq+K3rOtblUqSNG0Vnawj4jeB70TE3lMd\niyRJU6XYZB0RLwQ+CJyWmXdGxKTGGlNQoy0i9pyK6Wp0XEaSJluRyToiXgB8GrgRuA8gMwcn+Y/k\nXnUsY7p/ekTsOMrh9wb+Ajh1rJ8zIuaOpV/dd9+ImDPW/mOY3oERcXREzIqIGaPot39ELIqIgdH0\nmwgRsU9dmGafMfZ/8iiGnR0RB9fPnxsRjx3LNMdjrPN3rMtoPMs2Ip4SEc+sl4+01SnuBLOIeC7w\nL8B7gT2BPYALM/Pyuj1yFEFHxHHAwcA5vfaLiDOBFwI3AL8APpGZa0cxzTcC2wP/kpkP9dgngN8D\nngIsAf5jlJ/zTOBAYCXwvsx8cBR99wDeDfxdZt7Za7+xiojfAf4WuLN+XA2c221eRcRJVOvFLcDP\nqeqj/3tmrupvxBARLwbeDvwSeCzwDeBvM3Ndj/3/EPgt4IzM/GUPwz8J+Od6ersAr87Me8cY/qhE\nxAGZ+eP6+YzM3DCKvmNaRuNZthHxG8D7gZ8Cs6jm8f/2GrO0RcjMoh7A4cAx9fMDgb8G/g44tmOY\n6GE8A/X/rwY+CpzWY7+TgEuBnYDvAR8dZfx/AFwJPK5+PbOHPkM/ml4L/BdwVR1H13jrfm8ELgH2\npvrj/mlg/1HEPAB8jSrJ93v5zgK+MLQ8gZcCHwDOAnZo6bcrVYI8uGNeLQX+Eti+zzE/myp5LKzX\niwOoflCdBczoof9vA9dS3VZwNNP9IPAQcGb9ekav68Q4PuuJwMPA5zve6/oZx7OMxrNsgWfVy+aI\n+vVXgOf1ez324WOyH8XtBs/MpZn5g4gYyMybqRLPI8CJEXFMPUwvW5xPrP//LHAZ8HTg1T3sYt4R\n+DBVsnwE+GOotja6TbDeDf0bVFupD9dbUx+tt7QbZWZGxCuBNwPvBH5AlSBe2i3eiNgBWAC8nCrx\n/bBu+khE7N+l794RcWBmDgJnAntGxEHdPucE2AEYiu0rwIVUSfwVLZ93PbAd8BiAzPwUcBvVPaBP\n7GewwDHARzJzGbAmq63OU4ATgHf00H8v4AuZeXtEzBrFdD9O9UPstRHxyszcUK8r2432A/QiIuZR\nrQdvBdZFxGcBMnNDj7ulx7qMxrNsfwn8QWZeFRGPAY4EzoyIT0TEyZ5foK1Fccl6SJ1AyMyfAJ8B\n1gAvj4gju/WtL/e6OCJOq8dzAVUSeyVwepcv8G1UW3pnZOYLMnNdRPwf4Pe7/aHNzNXARcD7gH8D\nHg9cBzwlImZ3CftAqq2Za4E/o9odeCbwsrZ4s9p1/CaqwwUvycwTqHanHw6c1jTd+g/z24B/iYjX\nU+22X0u1dd63k6gy8xHgQ8DvRMQz6uVzObAcOK6l34PA56gS12kRcVYd743A8/oRa8c82IcqcQCs\nrXcN3w6cDjwvIvboMr9uB46vfxg9Uo/7tHrXb6PMvCUzPwv8FfBnEfFb9fkcfzbWcym6TG8V1Vbt\n56nWjTmdCbuH/mNaRuNZtpn5o8z8Xv3yDOCfM/Mk4ArgZDYuN2nLNtWb9r0+gIOotmJ273H4FwHX\nAKd2vPcNql2LO7b0244qmXyQahfbq4FlwCE9TncOVaLcpX79cqrd6dt26XcS8J/AUzreW0J1LK7r\nbl6qLdXLgKdSbY18AXh8D7EuqId9F9VWylJg7z4vyzlUP0TOBo7veP+7wGEt/Xak+sH1KeBDHe9f\nSMsu9AmI97nAxcDC+vUA1Z6Avah+CM7r0n8HNh7OORE4tZ7PTxpFDCdQ/fC7mnp3cb8fVLunLwA+\nW79eABzUpc+YllE/li3VD+cFkzGvfPjo92PCf533S2beFBEfzHrLpIfhvx4RG4D31bunH6A65veh\nbDn5KjNXRsQHqI4z/ilwL/CazFzR43TXAEvrM1rPoNqleGpmPtyl6/epkvwrIuK7wFyqk8U+kpm/\n7mHSP6P6w/YhqiTyssz8WQ+xXlNvWW9DlYQOo9ojcOdoT+brVWauiYjPAQm8o971vpbqhMK7Wvo9\nCHwuIs7Les9LRLya6gSsnk+CGoMlwGLglHqeXA0M1icv7kKVuBtl5kMR8c/Ai6l2az9Itefmll4D\nyMz/johl9fNfjfFzjEpm3hsRfwB8ICJuovr+PLtLnzEto/Eu2+HrakS8lGp9+kW3vtKWoLizwSda\nRDyT6izTh4F3ZLWbude+s+DRXbejne62VMc1l2Tmj3rssxfwO/VjPfC2zLxulPE+BhjMMZ7VHRHv\nojoR6vVj6T/Kac0GjqU6KW8N8I+Z+cP2Xpv0fy3V7tpTMvP6/kT56LT2Bn4feA7VLtZ1VLtZTx3l\nOjUbIHs8i7wEEfFHwJ8Dzx/tfB7rMhpHv22AV1Gda3JKrz+ypdJt9ckaHk2cmdUx5cmc7pi2TOvj\nyZGZK/sQVtM0IzMzIl5OdSz2pMmaX/XJSzm0RTWKfvsCs0azhToe9R6aRVSX9d0DfCOrkyC3WhGx\nM/BF4E9G88Oxo/+YltE4+s0Cng/8z9a+bDS9TItkrd7UJ0mdCNzqFomGRMSc+pCJpClispYkqXDF\nXrolSZIqJmtJkgpnspYkqXAma0mSCmeyliZRREz45XgRMT8iXtHQNhARH4mIFRFxfUQsjYj9JjoG\nSf21xdzBTFKj+cArqO7pPdwpVHe0e1pWNeH3AfpeUlTSxHLLWpoCEfGsiPh+RHw5Im6KiM8NFQOJ\niNsi4u/rLeGroqptTUScGxEnd4xjaCv9fcAzImJ5fbexTo8F7sqNhXHuyMz76/4viIgrIuKaiPhS\n1NW8IuLJRqa7AAAU80lEQVSEOqZr6q3yC+v33xMRb+uY/oqImF8/f1Ud6/KoKl7NGIoxIs6KiGsj\nYklE7Fm/v2dEfKV+/9qoK+o1jUea7kzW0tR5OtW94w8GnkB169UhD2bmU6lqsX+4y3jeDlyWmYdl\n5v8b1vZF4EV18vuHiHg6QETsBvwFVe3nBVQFQv44IuYA51AVwllIXbayTUQ8mWoL/tjMPIzqXt6v\nrJvnUd1y91CqOvGvq9//CHBJ/f4C4IYu45GmNXeDS1Pnqsy8AyAillPtzr68bjuv4//hCbhnmXlH\nRBxIdU/z5wDfiYiXURWKORhYXG/Qz6a65/lBVHew+0kd12eBbveJfy5VYl9aj2sucHfdto6qwAxU\n1eueXz9/DlVFO7Iqv/lgRJzWMh5pWjNZS1NnbcfzDWz6fcwRnq+n3hsWEQNUCbarzFxLVR72GxHx\nS6pyrN8CLs7MUzuHjYjDWkb16PRrc4a6Af+eme8Yoc8jHffHH/4Zh2sbjzStuRtcKtMpHf9fUT+/\njWrLE6oSrkOlOX8NbD/SSCJiQV3NbSjBPw24nars57Edx8PnRcQBwE3A/Ih4Yj2KzmR+G9UuayJi\nATB0Vvl3gJMjYo+6bZe6EEeb7wB/WA8/IyJ2HON4pGnBZC2VaeeIuA54CzB00tg5wDMj4lrgaDae\n1X0dsKE+UWv4CWZ7AF+PiBX1cOuBj9Y1sV8DnFdP5wrgoLpgx+uB/4qIa9h0N/QFwC4RcQNwJvBj\ngMy8ker497fqcV1MdWJbm7cAz46I66l2jx88xvFI04KFPKTCRMRtwKLMvKeAWJ5FVVf9xKmORZrO\n3LKWJKlwbllLklQ4t6wlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZ\nrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawl\nSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkq\nnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJ\nWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqS\npMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTC\nmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpms\nJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJ\nKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqc\nyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMla\nkqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKk\nwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZ\nrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawl\nSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkq\nnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJ\nWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqS\npMKZrCVJKpzJWpKkwpmsJUkqnMlakqTCmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkqnMlakqTC\nmawlSSqcyVqSpMKZrCVJKpzJWpKkwpmsJUkq3MypDmBL9YIXnpD33HNP1+Hy0X8a2poagWxu2rxn\n6zQaBsrWrgVNKxv7bfZ+Nscx0jhGWj5NPYbHNXx8I7c3jK2H/iNHAZmtc3qz9WbkeTTyHO3ed+Se\nrf2yyzJoXJ9GmEmd4xjhg3X9vo00MxraRjv8JkO1fXkf/S60z+xN2kc5jzq/cCMtw7bhGye4Wb+R\nvtTDYx6hT9sfk47p5+pffTMzTxgh2GnJZD1G995zD4uXXL3JlyWp1ucc9kXJji9n5/reOWzmpuv2\n0LCd353O/hvHu2n/zml1fi+6xTXisKP4XBM5rcGOhDDUPrjZfKneGBw+DxMGN5knG+fZ4LB5mpkM\nsvEPa3a8N9TeOfymcQ317WjL6v9H4xoWy2BH+9Dr7Bh+cPjn6hj38NfVuIdPuyO24a87P2du7NP5\nOTs/Y27yOTYdtjPuZORxdX7OoT6dy2/EcTXElcPGtfnr9uF7G3bzvoODvcfCZuPavK2zfSKGH8u4\nqsAHO76QgxvfG/H1CM+b+g4Otfc4fFN7/XzN8o/thh7lbnBJkgpnspYkqXAma0mSCmeyliSpcCZr\nSZIKZ7KWJKlwJmtJkgpnspYkqXAma0mSCmeyliSpcCZrSZIKZ7KWJKlwJmtJkgpnspYkqXAma0mS\nCmeyliSpcCZrSZIKF5k51TFskSLiv4HdpjqOEewG3DPVQTQoNTbjGr1SYys1Lig3tlLjuiczT5jq\nIEphst7KRMTVmbloquMYSamxGdfolRpbqXFBubGVGpc25W5wSZIKZ7KWJKlwJuutz9lTHUCLUmMz\nrtErNbZS44JyYys1LnXwmLUkSYVzy1qSpMKZrCVJKpzJegsVESdExM0RcUtEvH2E9oMi4oqIWBsR\nbysorldGxHURcX1E/CAiDi0othfXsS2PiKsj4rgS4uoY7vCIWB8RJ5cQV0Q8KyIerOfX8oh492TE\n1UtsHfEtj4gbIuKSEuKKiD/tmF8rImJDROxSSGw7RsTXI+Laep6dPhlxqUeZ6WMLewAzgP8BngDM\nBq4FDh42zB7A4cBZwNsKiusYYOf6+W8AVxYU23ZsPI/jacBNJcTVMdx3gYuAk0uIC3gWcOFkLL8x\nxLYTcCPw+Pr1HiXENWz4FwHfLWievRN4f/18d+A+YPZkL18fIz/cst4yHQHckpk/zcx1wPnAizsH\nyMy7M3Mp8Ehhcf0gM++vXy4B9ikotpVZ/6UC5gGTcfZl17hqbwYuAO6ehJhGE9dU6CW2VwD/kZk/\ng+r7UEhcnU4FzpuEuKC32BLYPiKC6ofrfcD6SYpPXZist0x7Az/veH1H/d5UG21cZwDf6GtEG/UU\nW0S8JCJuAv4LeG0JcUXE3sBLgH+ZhHh6jqt2TH3o4BsR8ZTJCa2n2A4Ado6I70fEsoh4dSFxARAR\n2wInUP0Amwy9xPZR4MnAL4Drgbdk5uDkhKduZk51AJqeIuLZVMl6Uo4L9yozvwJ8JSKOB/4aeN4U\nhwTwYeDPM3Ow2ugpxjVUu5lXRsRvAv8J7D/FMQ2ZCSwEngvMBa6IiCWZ+eOpDetRLwIWZ+Z9Ux1I\nhxcCy4HnAE8ELo6IyzLzoakNS+CW9ZbqTuBxHa/3qd+baj3FFRFPA/4VeHFm3ltSbEMy81LgCRHR\n72ItvcS1CDg/Im4DTgb+OSJOmuq4MvOhzFxZP78ImDUJ86un2Ki2HL+Zmasy8x7gUqDfJzOOZh17\nOZO3Cxx6i+10qkMHmZm3ALcCB01SfOrCZL1lWgrsHxH7RcRsqi/+16Y4Jughroh4PPAfwGmTvJXT\nS2xPqo/XERELgG2Afv+Y6BpXZu6XmfMzcz7wZeCNmfmfUx1XRDymY34dQfX3ZDJ+fPWy/n8VOC4i\nZta7nI8EflRAXETEjsAz6xgnSy+x/YxqTwQRsSdwIPDTSYxRLdwNvgXKzPURcSbwTaqzPD+VmTdE\nxBvq9o9HxGOAq4EdgMGIeCvV2Z9926XVS1zAu4FdqbYOAdbnJFT86TG2lwKvjohHgNXAKR0nnE1l\nXJOux7hOBv4wItZTza+X93t+9RpbZv4oqjK21wGDwL9m5oqpjqse9CXAtzJzVT/jGUNsfw2cGxHX\nA0F16KXE0pnTkrcblSSpcO4GlySpcCZrSZIKZ7KWJKlwJms9KiJOioiMiIM63psfEa0n5vQyzESK\niNdExEcnaFwREd+NiB3q1xs67tv8pfpM4tGMb+Uohz83RrjXd0QsioiP1M8f/bwR8YahG3zU7+81\nmumNVlT31z5mnON45xj6vCwifhQR3xv2/vyIeEXH63GtC/X8f1Z985T5Y+h/UL2+/DAiFkbEG8ca\nyyim+Z76c58bEc+q3zs/Ikq5xl19YLJWp1OBy+v/p4vfBK7tOEt+dWYelpmHAOuAN3QOXCf3vn9v\nMvPqzPw/I7z/8cz8dP3yNUBfkzXV/b/Hlayp7jk9WmcAr8vMZw97fz7VrURLcRLw5cx8OtVla31P\n1g3+BfizKZq2JoHJWgBExHZUdxM7g+oazJGGeU1EfLXeCvlJRPxVR/OMiDgnqmo934qIuXWf10XE\n0qgq+VwwfEs1IgYi4raI2KnjvZ9ExJ4R8aKIuLLeavl2fe3n8Jg22TLt3LKNqsLR0qhuh/neho/+\nSpqvd70MeFK9NXdzRHwaWAE8LiJOjapy2IqIeP+wmP5fPR++ExG79zAfnhdVla8fR8SJ9fDPiogL\nR/i874mIt9WfeRHwuXrL7rci4j87hnt+RHxlhP7Prefn9RHxqYjYpn7/tqhvaFJv1Q9tab4B+KN6\nGs+o5/fHR4h3ky3ciLiw/gzvA+bW/T83QjybzceoqncdB3wyIj4wrMv7gGfU4/uj+r29IuK/6/Xm\n7zvG/YKoKs9dE9Veku2GTx94kOpH2X3AhoiYUX/GFXVcf1SP67CIWFKvS1+JiJ2jumvbW6kuX/te\nHdsT69g+UH/+S+rvzE8j4n1RVZ27qh73E+txj7ieR8Q/1vOCiHhhRFwa1Q/FlVSXyg3FDtW6+ryI\n8HLcrdVUVxLxUcaDKml9sn7+A2Bh/Xw+sKJ+/hrgLqrrpOdSJa5F9TDrgcPq4b4IvKp+vmvHNP4G\nePMI0/5H4PT6+ZHAt+vnO7Px8sLfB/6hI46P1s/PpaMKFbCy/v8FwNlU14sOABcCx48w7duB7Ufo\nP5Mqif9h/fkGgaPqtr2obiCxez3cd4GT6rYEXlk/f3dHnCPOhzr+/65j3J/qzltz6KhoNezzvoe6\nihrwfWBR/TyAm4Dd69efB1407LPOobo/9AH1608Db62f3wbsVj9fBHx/+PS6xPtojPVwFwLP6pyn\nI8z7tvn46Gcb1ufR+dIxb34K7FjHcTvVnbp2o7pr2bx6uD8H3t3D92AhcHHH653q/68Dnlk//7/A\nh0dYHvOpvysdsT4APJbqBjt3Au+t297SMY6m9Xxb4Abg2cDNwBO7xH4x9ffWx9b3cMtaQ06lqsRD\n/X/TrvCLM/PezFxNdSeyoXt735qZy+vny6j+cAEcEhGXRXWjhVcCIxV7+AJwSv385fVrqG6J+M26\n75829G3ygvrxQ6p7WB/EyPet3iUzf93xem5ELKe6oczPgE/W79+emUvq54dTJbNfZeZ64HPA8XXb\nYEf8n2Xj/GmbD1/MzMHM/AlV4hn1LR4zM4HPAK+q91IczeZFUg6kWk5Dd4779464R2Pc8dba5uNo\nfCczH8zMNVRlMfcFjgIOBhbXy/P36ve7+SnVbWb/KSJOAB6K6o5jO2XmUE3s0cy3pZl5V2aupSpR\n+a36/evZ+B0ZcT3PzIeB11El4Y9m5v90mdbd9P+wiKaIu0xEROxCdfP+p0ZEUt3hKCPiT0cYfPhd\ndIZer+14bwPVljdUW2InZea1EfEaqq2N4a6g2t28O9UxwL+p3/8n4EOZ+bWoTqR5zwh911Mfzql3\nEc4e+ljA32XmJ0bos0n/iBjIjdWFVmfmYZ0DRHWntbHebWpo/pxL83xomqej9W/A14E1wJfqBNir\nR+cj1RZqm5Hi7ezfyzgm0vB1bybV8r84M0d1/kVm3h8Rh1IVtXgD8LvAH7X36jm2wY7Xg2z8+9u2\nnj+V6lh4L0l4DtXucW2F3LIWVLeN/Exm7pvVPagfR3UT/2eMMOzzI2KXqI5JnwQs7jLu7YG7ImIW\n1RblZuqtwq8AHwJ+lBuLe+zIxmIDv9cw/tuodl0C/DYwq37+TeC1Q8cpI2LviNhjhP43A0/o8hmG\nuwp4ZkTsFhEzqPZCDG11DVDNT6hOhLq8ft42H14W1bH7J9ax3NxjHL+uxwtAZv6CqrzhX1Al7uFu\nBuZHxJPq16d1xH0bG+fjS5um0RLvbcBh9fuPo6qfPOSR+nMP1zYfm4wUz0iWAMcOfdaImBcRB3Tr\nVB+3H8jMC6jm44LMfBC4PyKGvg+d820ssQ034noeEfsCfwI8HfiNiDiyy3gOoDo0pa2QyVpQ/ZEc\nfjLSBYy8K/yquu064ILMvLrLuP8SuJIqqd/UMtwXgFexcRcyVFsYX4qIZUDTPYrPofqDfy3Vrt9V\nAJn5LarjtlfUuxe/zMh/SP+Lkbf2G2XmXcDbge8B1wLLMnPoJLVVwBFRXcr2HKrjm9A+H35GNV+/\nAbyh3p3bi3OBj9cnNA3tyfgc8PPM3KxoRT3e06nm6fVUW3dD96t+L/CPEXE11dbpkK8DLxk6wawl\n3sVUP/BuBD5CdehhyNnAdcNPMOsyH5tcR3Ui2LUdJ5htJjN/RXU8+7yIuI5q700vu+v3Br5f7zr/\nLPCO+v3fAz5Qj+swNi7XzmneS7XbfcUIJ8a1eQ/D1vOodud8kup4+C+oTvz814gYcY9FfVLa6sz8\n31FMV1sQ7w2untW7bxdl5plTHctEiYjHAp/OzOdPdSwTIaozsn+YmZ/sOvDYxn8u1QleX+7H+DU2\n9Q+Xh/q13DX13LLWtFZv3Z0T9U1RtmT1ltnTqLYINb08QHXim7ZSbllLklQ4t6wlSSqcyVqSpMKZ\nrCVJKpzJWpKkwpmsJUkq3P8HlGnNmOZD6BMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2617a06f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, \"Tuesday 09 Oct 1993\", num = 7, n_s = 64);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that the places where the network is paying attention makes sense to you.\n",
    "\n",
    "In the date translation application, you will observe that most of the time attention helps predict the year, and doesn't have much impact on predicting the day or month."
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "n16CQ",
   "launcher_item_id": "npjGi"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}